INFORMATION TO USERS This manuscript has been reproduced from the microfilm master. UMI films the text directly from the original or copy submitted. Thus, some thesis and dissertation copies are in typewriter face, while others may be from any type of computer printer. The quality of this reproduction is dependent upon the quality of the copy submitted. Broken or indistinct print, colored or poor quality illustrations and photographs, print bleedthrough, substandard margins, and improper alignment can adversely affect reproduction. In the unlikely event that the author did not send UMI a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note will indicate the deletion. Oversize materials (e.g., maps, drawings, charts) are reproduced by sectioning the original, beginning at the upper left-hand comer and continuing from left to right in equal sections with small overlaps. Each original is also photographed in one exposure and is included in reduced form at the back of the book. Photographs included in the original manuscript have been reproduced xerographically in this copy. Higher quality 6" x 9" black and white photographic prints are available for any photographs or illustrations appearing in this copy for an additional charge. Contact UMI directly to order. University Microfilms International A Bell & Howell Information Company 300 North Zeeb Road. Ann Arbor, Ml 48106-1346 USA 313/761-4700 800/521-0600 Order Number 9427736 Exploring the computational capabilities of recurrent neural networks Kolen, John Frederick, Ph.D. The Ohio State University, 1994 Copyright ©1994 by Kolen, John Frederick. All rights reserved. UMI 300 N. Zeeb Rd. Ann Arbor, MI 48106 EXPLORING THE COMPUTATIONAL CAPABILITIES OF RECURRENT NEURAL NETWORKS DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University By John F. Kolen, B.A., M.S. The Ohio State University 1994 Dissertation Committee: rove J. B. Pollack B. Chandrasekaran Adviser department of Computer and D. Wang V J Information Sciences Copyright by John F. Kolen 1994 Very few beings really seek knowledge in this world. Mortal or immortal, few really ask. On the contrary, they try to wring from the unknown the answers they have already shaped in their own minds—justification, confirmation, forms of consolation without which they can’t go on. To really ask is to open the door to a whirlwind. The answer may annihilate the question and the questioner. Anne Rice, The Vampire Lestat (p. 332) To Mary Jo ii ACKNOWLEDGEMENTS While a dissertation has a single author by definition, many people are responsible for its existence and should be recognized for their efforts and contributions. Dr. Jordan Pollack, my advisor for six years, has provided unique environments, both physical and intellectual, for myself and his other students. His dogged insistence that we solve the really big problems directed me to bountiful orchards where my own eye for feasible projects led me to the low hanging fruit. Dr. B. Chandrasekaran helped at a crucial time by sending me to his graduate students seven years ago as my interests swayed from software testing to neural networks. His and Dr. DeLiang Wang’s suggestions and comments strengthened the final version of this document. As Dr. Pollack’s first student at OSU, I watched his research team grow over the years. The other post-connectionist pre-docs, Dr. Peter Angeline, Edward Large, Viet-Anh Nguyen, Gregory Saunders, and David Stucki, have been an incredible group of people to work with. They helped me separate good ideas from bad hallucinations, sharpen my incomprehensible babble into salient arguments, and develop advising skills. I must also thank other members of the Laboratory for Artificial Intelligence Research, both past and present, who have helped me over the years: Drs. Ashok Goel and Dean Allemang for advising me during my first years at OSU, Barbara Becker for reading my papers although they had nothing to do with systemic grammars, Arun Welch and Bryan Dunlap for keeping nervous running, and finally Dr. Dale Moberg for his enlightening discussions on just about everything. I should not forget to mention the emotional support of the Dead Researchers Society (they know who they are). Our many meetings have helped soothe the anguish inflicted by troublesome advisors, collapsing research fields, and upheaving job markets. I officially express gratitude to my two sources of financial support during the last seven years. The Department of Computer and Information Science supported my first five quarters at OSU through a graduate teaching assistantship. The remainder of my financial support as a graduate research assistant came from the Office of Naval Research by way of grant numbers N00014-89-J-1200 and N00014-92-J1195. Thanks go out to my typists and editors, Mocha and Smeagol. Chapter VI should be dedicated to them and the makers of the Squiggle Ball™. I could not be in the position of writing these acknowledgments would not be possible without the support of my parents throughout the years. Their love, hard work, and sacrifice gave me the opportunity to choose a path that they could not follow themselves. Finally, there is one person that deserves more gratitude than I can express on this page. Her words have encouraged me during the darkest hours. When I became too absorbed in my work, she would remind me of what is truly important. She even read the document cover to cover during the final revisions. Mary Jo, thanks for all your love, understanding, and support. VITA December 17, 1965 ..................................... Bom - San Diego, California 1987 .............................................................. B.A., University of California at San Diego, La Jolla, California 1987-1988...................................................... Graduate Teaching Assistant, The Ohio State University, Columbus, Ohio 1988 ........................... ................................... M.S., The Ohio State University, Columbus, Ohio 1989-Present.................................................. Graduate Research Assistant, The Ohio State University, Columbus, Ohio PUBLICATIONS Kolen, J. F. (1994) Fool’s Gold: Extracting Finite State Machines From Recurrent Network Dynamics. In J. D. Cowan, G. Tesauro, and J. Alspector (eds.), Advances in Neural Information Processing Systems 6, 501-508. San Mateo, CA:Morgan Kaufmann. Kolen, J. F. (1994) Recurrent Networks: State Machines or Iterated Function Systems?. In M. C. Mozer, P. Smolensky, D. S. Touretzky, J. L. Elman, & A. S. Weigend (Eds.), Proceedings of the 1993 Connectionist Models Summer School, 203-210. Hillsdale, NJ: Erlbaum Associates. Kolen, J. F., and J. B. Pollack. (1993) The apparent computational complexity of physical systems. In The Proceedings of The Fifteenth Annual Conference of the Cognitive Science Society, 617-622. .Hillsdale, NJ:Erlbaum. Large, E. W., and J. F. Kolen. (1993) A dynamical model of the perception of metrical structure. In Proceedings of the Society for Music Perception and Cognition. Philadelphia, PA. May 1-3,1993. Saunders, G., J. F. Kolen, P. J. Angeline, and J. B. Pollack. (1992) Additive modular learning in preemptrons. In The Proceedings o f The Fourteenth Annual Conference o f the Cognitive Science Society, 1098-1103. Hillsdale, NJ:Erlbaum. Kolen, J. F. and A. K. Goel. (1991) Learning in Parallel Distributed Processing networks: Computational complexity and information content. IEEE Transactions on Systems, v Man, and Cybernetics, 21, 359-367. Kolen, J. F. and J. B. Pollack. (1991) Multiassociative Memory. In The Proceedings of The Thirteenth Annual Conference of the Cognitive Science Society, 785-790. Hillsdale, NJ:Erlbaum. Kolen, J. F. and J. B. Pollack. (1991) Back Propagation is Sensitive to Initial Conditions. In R. P. Lippman, J. E. Moody, and D. S. Touretzky (eds.), Advances in Neural Information Processing Systems 3, 860-867. San Mateo, CA:Morgan Kaufmann. Kolen, J. F. and J. B. Pollack. (1990) Back Propagation is Sensitive to Initial Conditions. Complex Systems, 4, 269-280. Kolen, J. F. and J. B. Pollack. (1990) Scenes from Exclusive-Or: Back Propagation is Sensitive to Initial Conditions. In The Proceedings o f The Twelfth Annual Conference o f the Cognitive Science Society, 868-874. Hillsdale, NJ:Erlbaum. Kolen, J. F. (1989) Review of “Fast learning in artificial neural systems: Multilayer perceptron training using optimal estimation”. Neural Network Review, 3. Goel, A. K., J. F. Kolen, and D. Allemang. (1988) Learning in connectionist networks: Has the credit assignment problem been solved?, In The Proceedings o f the SIGART Aerospace Applications o f Artificial Intelligence Conference, 11:74-80. Kolen, J. F. (1988) Faster learning through a probabilistic approximation algorithm. In The Proceedings o f the Second IEEE International Conference on Neural Networks, 1:449-454. Uht, A. K., C. D. Polychronopoulos, and J. F. Kolen. (1987) On the combination of hardware and software concurrency extraction methods. In The Proceedings of the Twentieth Annual Workshop on Microprogramming, 133 - 141. FIELDS OF STUDY Major Field: Computer and Information Sciences, Prof. J. B. Pollack Specialization: Artificial Intelligence Minor Field: Theory of Computation, Prof. T. Long Minor Field: Computational Geometry, Prof. K. Supowit TABLE OF CONTENTS DEDICATION......................................................................................................................