Browse Books

Reviewer: John A. Fulcher

Researchers and students meeting neural networks for the first time are often referred to two classic references in the field. The first of these is Minsky and Paperts Perceptrons [1], and the other is the so-called “PDP” books of Rumelhart and McClelland. Moreover, a collection of classic neural network papers is to be found in Anderson and Rosenfelds Neurocomputing: foundations of research [2]. Of the four PDP books in this review, two comprise the original 1986 two-volume set, whose ninth printing was in 1989. The authors of these books are David Rumelhart, James McClelland, and 14 other members of the PD Research Group at the University of California at San Diegos Institute for Cognitive Science. The first volume is subtitled Foundations , while the second is subtitled Psychological and biological models . The subtitle of this second volume, taken together with the name of the parent institute, reveals a cognitive science rather than an artificial neural network (ANN) perspective. For readers interested in the latter rather than the former, Volume 1 of the PDP books would most likely be the only volume of interest. This volume is broken down into the following three sections: “The PDP Perspective” (4 chapters), “Basic Mechanisms” (4 chapters), and “Formal Analyses” (5 chapters). Volume 2 comprises the sections “Psychological Processes” (6 chapters), “Biological Mechanisms” (6 chapters), and “Conclusion” (1 chapter). Perhaps the most important chapter in the entire two-volume set, from an ANN perspective, is chapter 8, “Learning Internal Representations by Error Propagation,” written by Rumelhart, Hinton, and Williams. Minsky and Paperts earlier analysis of perceptrons is the starting point for this chapter, with the exclusive-OR problem used to illustrate first the limitations of perceptrons, then later the capabilities of multilayer perceptrons (MLPs). Widrow and Hoffs least mean squares or delta learning rule is expanded into the generalized delta rule, and the error backpropagation algorithm developed (specifically the error between desired and actual outputs upon presentation of an input-output training exemplar) is used to modify the interconnecting weights within the feedforward network. The convergence and stability of this algorithm are then discussed in terms of gradient descent in weight space. The capabilities of the backpropagation algorithm are then demonstrated using the following selection of classically hard pattern classification problems: XOR parity encoderdecoder symmetry addition negation TC character discrimination The chapter concludes with a discussion of recurrent networks, a modification of feedforward MLPs, which can handle time-varying sequences. This chapter on backpropagation is often cited in the ANN literature as the classic reference on the subject. Another important chapter is chapter 7, “Learning and Relearning in Boltzmann Machines,” by Hinton and Sejnowski. This is one of the classic references on Boltzmann machines, and explains the use of the simulated annealing algorithm and statistical weight update in order to escape from local minima in the energy landscape during the training process. Boltzmann machines are often discussed in the ANN literature in the light of improvements to Hopfield networks; little is made of this aspect in this chapter, however. The remaining chapters vary in quality and relevance, which is to be expected of a collection of papers from various permutations of members of the UCSD PDP research group. The orientation of the authors is toward cognitive science rather than ANNs; more relevant and more accessible books on ANNs have been published during the last two or three years, and these will be the subject of a forthcoming comparative review in Computing Reviews . The term “parallel distributed processing” (PDP) is not widely used in the ANN field, for instance. The two remaining PDP books, both titled Explorations in parallel distributed processing: a handbook of models, programs, and exercises , are manuals for the software that accompanies the original two-volume PDP set—one for DOS (two 5.25-inch disks) and one for the Macintosh (two 3.5-inch disks). I shall restrict my comments to the DOS volume here, since this software was written first (1988); the Macintosh software (1989) is a simple port of the DOS software, and as such suffers from the lack of a graphical user interface of a standard usually associated with Macintosh software (indeed, all programs use an 80×24 character text window only). Moreover, the Macintosh manual is identical to the DOS manual (344 pages), with 11 more pages added following the index, under the heading “Notes for Using the PDP Software on Macintosh Computers.” The EPDP book (manual) contains seven chapters and seven appendices, as follows: Introduction Interactive Activation and Competition Constraint Satisfaction in PDP Systems Learning in PDP Models: the Pattern Associator Training Hidden Units: the Generalized Delta Rule Other Learning Models: Autoassociators and Competitive Learning Modeling Cognitive Processes: the Interactive Activation Model Setting up the PDP Software on the PC Command and Variable Summary File Formats for Network, Weight, Template, Look, and Pattern Files Plot and Colex: Utility Programs for Making Graphs Answers to Questions in the Exercises An Overview of the PDP Software Instructions for Recompiling the PDP Programs The manuals are well written and presented, and explain the underlying theory of the various ANN models supported in the software, but could have been improved by the inclusion of more diagrams. The software is robust, bug-free, reliable, and of a commercial standard (unlike some ANN software that accompanies some more recent ANN books). It is rather inaccessible for those encountering ANNs for the first time, however. One has to wade through a considerable amount of background material—on competitive learning, pattern association, and the like—before meeting backpropagation in chapter 5, and even then we are forced to use terminology peculiar to the PDP group, rather than the more usual and accepted terminology found in more recently published ANN books. For instance, global and local “maxima of the goodness function,” appear rather than “minima of the energy function,” as does “pattern associators” rather than “perceptrons.” Moreover, the book makes no mention of MLPs in relation to backpropagation. The EPDP software is locked into the two-volume PDP book set, and cannot be considered as a general-purpose ANN software tool. It has no support for Hopfield networks, Grossbergs ART, Kohonens Self-Organizing Feature Map, or many other fundamental ANN models, for example. Commercial ANN software packages like HNCs ExploreNet or NeuralWorks Professional-II+ are much more accessible, use standard terminology, and support a much wider range of the more significant ANN models. Indeed, several introductory-level ANN textbooks published during the last two to three years come with ANN software simulators that are preferable to EPDP (and this will be the subject of another forthcoming comparative review in CR ). In summary, the two-volume PDP set and the accompanying DOS or Macintosh EPDP software are important works for students and researchers of either biological or artificial neural networks. They are best suited as introductions to cognitive science rather than to ANNs, however.