The Psychophysics Psyber Lab




Brian Scurfield's Thesis

Discrimination Among Events by Neural Networks

October, 1994

A thesis submitted to the Victoria University of Wellington in fulfillment of the requirements for the degree of Doctor of Philosophy in Psychology
Victoria University of Wellington, New Zealand


Usually, the performance of neural networks in event discrimination tasks is measured by the proportion of correct decision. Although the deficiencies of proportion correct warrant the use of receiver operating characteristic (ROC) analysis, the detectability measures associated with ROC analysis also have deficiencies. Information theory is used to develop a new detectability measure that is free of the deficiencies of the existing detectability measures. The new measure, denoted D2, is based on the area below and the area above the ROC curve. It was used to evaluate how well two neural networks---the Hopfield network and the back--propagation network---distinguish between stored and other patterns. It is shown that the probability distributions of the Liapunov function associated with the Hopfield network can be used to construct ROC curves. For the back--propagation network, ROC curves can be constructed using the posterior uncertainty of the events. The Hopfield network and the back--propagation network were compared, also, to a benchmark nearest--neighbour network.

ROC analysis is generalized in order to define appropriate indices for neural networks involved in tasks with three or more events. It is shown that the performance of an observer (such as a neural network) in an identification task with n independent events can be represented in n! ROC spaces of dimension n. Each ROC space is associated with a unique pairing of the events and decisions. A hypersurface can be generated in each ROC space by manipulating the observer's decision criteria. It is shown that the hypervolume of each hypersurface is a probability and that the hypervolumes sum to one. Using these facts, the detectability D2 is generalized. The generalized measure, denoted Dn , is nonparametric and independent of the criteria. The value of Dn is shown to increase monotonically with n and to be equal to the channel capacity of an observer in a n--interval forced--choice task. Examples are given of the application of generalized ROC analysis to neural networks. In particular, Dn was used to evaluate how well the back--propagation network can distinguish among sets of up to seven stored patterns. It is concluded that measuring the performance of neural networks is a more difficult problem than is generally supported, and that sound detectability measures such as Dn are required.

Last updated 08 Nov 2009 04:37 PM

This psyber lab webpage is maintained by Judi Lapsley Miller
Comments, suggestions, and enquires always welcome.
Generic Copyright Statement, 2000-2010. All rights reserved.