|
| |
Brian Scurfield's Thesis
Discrimination Among Events by Neural Networks
October, 1994
A thesis submitted to the Victoria University of Wellington in fulfillment
of the requirements for the degree of Doctor of Philosophy in Psychology
Victoria University of Wellington, New Zealand
Abstract
Usually, the performance of neural networks in event discrimination tasks is
measured by the proportion of correct decision. Although the deficiencies of
proportion correct warrant the use of receiver operating characteristic (ROC)
analysis, the detectability measures associated with ROC analysis also have
deficiencies. Information theory is used to develop a new detectability measure
that is free of the deficiencies of the existing detectability measures. The new
measure, denoted D2, is based on the area below and the area
above the ROC curve. It was used to evaluate how well two neural networks---the
Hopfield network and the back--propagation network---distinguish between stored
and other patterns. It is shown that the probability distributions of the
Liapunov function associated with the Hopfield network can be used to construct
ROC curves. For the back--propagation network, ROC curves can be constructed
using the posterior uncertainty of the events. The Hopfield network and the
back--propagation network were compared, also, to a benchmark nearest--neighbour
network.
ROC analysis is generalized in order to define appropriate indices for neural
networks involved in tasks with three or more events. It is shown that the
performance of an observer (such as a neural network) in an identification task
with n independent events can be represented in n! ROC spaces of
dimension n. Each ROC space is associated with a unique pairing of the
events and decisions. A hypersurface can be generated in each ROC space by
manipulating the observer's decision criteria. It is shown that the hypervolume
of each hypersurface is a probability and that the hypervolumes sum to one.
Using these facts, the detectability D2 is generalized. The
generalized measure, denoted Dn , is nonparametric and
independent of the criteria. The value of Dn is shown to
increase monotonically with n and to be equal to the channel capacity of
an observer in a n--interval forced--choice task. Examples are given of
the application of generalized ROC analysis to neural networks. In particular,
Dn was used to evaluate how well the back--propagation network
can distinguish among sets of up to seven stored patterns. It is concluded that
measuring the performance of neural networks is a more difficult problem than is
generally supported, and that sound detectability measures such as Dn
are required.
Last updated
08 Nov 2009 04:37 PM
|
