United States Patent [19] [11] Patent Number: 5,946,673 Francone Et Al
Total Page:16
File Type:pdf, Size:1020Kb
US005946673A United States Patent [19] [11] Patent Number: 5,946,673 Francone et al. [45] Date of Patent: Aug. 31, 1999 [54] COMPUTER IMPLEMENTED MACHINE Chris Gatercole & Peter Ross, “Dynamic Training Subset LEARNING AND CONTROL SYSTEM Selection for Supervised Learning in Genetic Program ming”, Feb. 25, 1994, pp. 1—10, Department of Arti?cial [76] Inventors: Frank D. Francone, 4806 Fountain Intelligence University of Edinburgh, Edinburgh EH1 1HN Ave., Suite 77, Los Angeles, Calif. 90029; Peter Nordin, Dacapo Ab UK. Torggatan 8, S-411 05 Gothenburg, Sweden; Wolfgang Banzhaf, Ackerstr, (List continued on next page.) 16 44575, Castrop-Rauxel, Germany Primary Examiner—Robert W. Downs [21] Appl. No.: 08/679,555 Attorney, Agent, or Firm—Arter & Hadden LLP; David G. [22] Filed: Jul. 12, 1996 Alexander [51] Int. Cl.6 .................................................... .. G06F 15/18 [57] ABSTRACT [52] US. Cl. .............................. .. 706/13; 395/81; 701/301 [58] Field of Search ................................ .. 395/13, 22, 81; In a computer implemented learning and/or process control 701/210, 301; 706/13 system, a computer model is constituted by the most cur rently ?t entity in a population of computer program entities. [56] References Cited The computer model de?nes ?tness as a function of inputs and outputs. Acomputing unit accesses the model with a set U.S. PATENT DOCUMENTS of inputs, and determines a set of outputs for which the 4,697,242 9/1987 Holland et al. ......................... .. 706/13 ?tness is highest. This associates a sensory-motor (input 4,821,333 4/1989 Gillies ............. .. 382/308 output) state with a ?tness in a manner that might be termed 4,881,178 11/1989 Holland et al. 706/12 “feeling”. The learning and/or control system preferably 4,935,877 6/1990 KOZa ....................................... .. 706/13 utilizes a compiling Genetic Programming system (CGPS) 4,961,152 10/1990 Davis ...................................... .. 706/13 5,048,095 9/1991 Bhanu et al. 382/173 in which one or more machine code entities such as func 5,111,531 5/1992 Grayson et al. .. 706/23 tions are created which represent solutions to a problem and 5,113,482 5/1992 Lynne . .. 706/23 are directly executable by a computer. The programs are 5,136,686 8/1992 KOZa ....................................... .. 706/13 created and altered by a program in a higher level language 5,140,530 8/1992 Guha et a1. ............................. .. 706/13 such as “C” which is not directly executable, but requires 5,148,513 9/1992 KoZa et al. 706/13 translation into executable machine code through 5,150,289 9/1992 Badavas 364/154 5,159,660 10/1992 Lu 618.1. 706/23 compilation, interpretation, translation, etc. The entities are initially created as an integer array that can be altered by the 5,172,253 12/1992 Lynne . .. 706/19 5,182,793 1/1993 Alexander et al. 706/13 program as data, and are executed by the program by 5,222,192 6/1993 Shaefer ................................... .. 706/13 recasting a pointer to the array as a function type. The (List continued on next page.) entities are evaluated by executing them with training data as inputs, and calculating ?tnesses based on a predetermined FOREIGN PATENT DOCUMENTS criterion. The entities are then altered based on their ?t 0696000 2/1996 European Pat. Off. ...... .. G06F 15/18 nesses using a genetic machine learning algorithm by recast ing the pointer to the array as a data (eg integer) type. This OTHER PUBLICATIONS process is iteratively repeated until an end criterion is Andrew Schulman, David Maxey, Matt Pietrek, “Undocu reached. mented Windows”, 1992, pp. 189—398, Addison—Wesley Publishing Company. 68 Claims, 49 Drawing Sheets Initialization Main CGPS lr_oop Decompile Chosen Solution End 5,946,673 Page 2 US. PATENT DOCUMENTS Q. Wang and A.M.S. ZalZala, “Genetic Control of Near Time—optimal Motion for an Industrial Robot Arm,” Proc. 5,224,176 6/1993 Crain ..................................... .. 382/136 5,245,696 9/1993 Stork e161. 706/13 Int’l. Conf. on Robotics and Automation, vol. 3, pp. 5,249,259 9/1993 Halvey .. .... .. 706/13 2592—2597, Apr. 1996. 5,255,345 10/1993 Shaefer . .. 706/13 W.L. Baker and J.A. Farrell, “An Introduction to Connec 5,257,343 10/1993 Kyuma et al. 706/18 tionist Learning Control Systems”, Handbook of Intelligent 5,274,744 12/1993 Yu et al. .. 706/20 Control: Neural, FuZZy, and Adaptive Approaches; Ch. 2, pp. 5,301,256 4/1994 Nakamura ............................... .. 706/62 35—63, Van Nostrand Reinhold, NeW York, 1992. 5,343,554 8/1994 KoZa et al. ............................. .. 706/13 P]. Werbos, “Neurocontrol and Supervised Learning: An 5,367,612 11/1994 BoZich et al. 706/23 OvervieW and Evaluation”, Handbook of Intelligent Con 5,390,282 2/1995 KoZa et al. .. 706/13 5,390,283 2/1995 Eshelman et al. 706/13 trol: Neural, FuZZy, and Adaptive Approaches; Ch. 3, pp. 5,394,509 2/1995 Winston .................................. .. 706/13 65—89, Van Nostrand Reinhold, NeW York, 1992. 5,400,436 3/1995 Nara et al. .............................. .. 706/13 R. Langari and HR. Berenji, “FuZZy Logic in Control 5,410,634 4/1995 Li . .. 706/62 Engineering”, Handbook of Intelligent Control: Neural, 5,414,865 5/1995 Beran .... .. 706/14 FuZZy, and Adaptive Approaches; Ch. 4, pp. 93—137, Van 5,428,709 6/1995 Banzhaf .... .. 706/13 Nostrand Reinhold, NeW York, 1992. 5,434,796 7/1995 Weininger 364/578 5,440,723 8/1995 Arnold et al. 714/33 J .A. Franklin and DA. White, “Arti?cial Neural NetWorks in Manufacturing and Process Control”, Handbook of Intel 5,448,681 9/1995 Khan . .. 706/23 5,455,938 10/1995 Ahmed et al. 364/488 ligent Control: Neural, FuZZy, and Adaptive Approaches; 5,471,593 11/1995 Branigan ..... .. 395/582 Ch. 8, pp. 235—258, Van Nostrand Reinhold, NeW York, 5,493,631 2/1996 Huang et al. 706/23 1992. 5,504,841 4/1996 Tani . .. 395/81 DA. Sofge and TC. White, “Applied Learning: Optimal 5,566,275 10/1996 Kano .. 395/81 Control of Manufacturing”, Handbook of Intelligent Con 5,608,843 3/1997 Baird, III ................................ .. 706/25 trol: Neural, FuZZy, and Adaptive Approaches; Ch. 9, pp. OTHER PUBLICATIONS 259—Van Nostrand Reinhold, NeW York, 1992. P]. Werbos, T. McAvoy and T. Su, “Neural NetWorks, R. Huang and TC. Fogarty, “Adaptive Classi?cation and System Identi?cation, and Control in the Chemical Process Control—Rule Optimisation via a Learning Algorithm for Industries”, Handbook of Intelligent Control: Neural, FuZZy, Controlling a Dynamic System,” Proc. 30th Conf. on Deci and Adaptive Approachesl Ch. 10, pp. 283—356, Van Nos sion and Control, pp. 867—868, Dec. 1991. trand Reinhold, NeW York, 1992. LE. Lansberry and L. Wozniak, “Adaptive Hydrogenerator AG. Barto, “Reinforcement Learning and Adaptvie Critic Governor Tuning With a Genetic Algorithm,” IEEE Trans. Methods”, Handbook of Intelligent Control: Neural, FuZZy, on Energy Conversion, vol. 9, No. 1, pp. 179—185, Mar. and Adapative Approaches; Ch. 12, pp. 469—491, Van Nos 1994. trand Reinhold, NeW York, 1992. L. Moyne, et al., “Genetic Model Reference Adaptive Con P]. Werbos, “Approximate Dynamic Programming for trol,” 1994 IEEE Int’l Symp. on Intelligent Control, pp. Real—Time Control and Neural Modeling”, Handbook of 219—224, Aug. 1994. Intelligent Control: Neural, FuZZy, and Adaptive K.—T. Lee, et al., “Genetic—Based Reinforcement Learning Approaches; Ch. 13, pp. 493—525, Van Nostrand Reinhold for FuZZy Logic Control Systems,” 1995 IEEE Int’l Conf. on NeW York, 1992. Systems, Man and Cybernetics, vol. 2, pp. 1057—1060, Oct. SB. Thrun, “The Role of Exploration in Learning Control”, 1995. Handbook of Intelligent Control: Neural, FuZZy, and Adap W.—P. Lee, et al., “AHybrid GP/GAApproach for Co—evolv tive Approaches; CH. 14, pp. 527—559, Van Nostrand Rein ing Controllers and Robot Bodies to Achieve Fitness—Speci hold, NeW York, 1992. ?ed Tasks,” Proc. 1996 IEEE Int’l. Conf. on Evolutionary SP. Singh and RS. Sutton,“Reinforcement Learning With Computation, pp. 384—389, May 1996. Replacing Eligibility Traces”, Machine Learning, 1, pp. Q. Wang and A.M.S. ZalZala, “Transputer Based GA Motion 1036, Jan.—Mar. 1996, pp. 123—158. Control for PUMA Robot,” Mechatronics, vol. 6(3), pp. T. Jaakkola, S.P. Singh and MI. Jordan, “Reinforcement 349—365, Apr. 1996. Learning Algorithm for Partially Observable Markov Deci Ray, T.S., “Is It Alive Or Is It GA?”, School of Life & Health sion Problems,” Dept. of Brain & Cognitive Sciences, Sciences, University of DelaWare, Jul., 1991, XP Cambridge, Ma, Dec. 1994. 002047764, pp. 527—534. SP. Singh, “An (almost) Tutorial on Reinforcement Learn R. Huang and TC. Fogarty, “Adaptive Classi?cation and ing”, Learning to SOlve Markovian Decision Tasks, Chap Control—Rule Optimisation via a Learning Algorithm for ters 2,3 and 4, PhD. Thesis, pp. 1—75, 1993. Controlling a Dynamic System,” Proc. 30th Conf. on Deci P]. Werbos, “Optimization Methods for Brain—Like Intelli sion and Control, vol. 1, pp. 867—868, Dec. 1991. gent Control”, National Science Foundation, Arlington, VA, E.—G. Talbi and T. Muntean, “Designing Embedded Parallel Dec. 1995, IEEE. Systems With Parallel Genetic Algorithms,” IEE Colloquium P.J. Werbos, “Optimal Neurocontrol: Practical Bene?ts, on ‘Genetic Algorithms for Control Systems Engineering’, NeW Results and Biological Evidence”, Nat’l Science Foun pp. 7/1—2, May 1993. dation, Arlington, VA, Nov. 1995, IEEE. I. Ashiru and C. CZarnecki, “Optimal Motion Planning for M.O. Duff, “Q—Learning for Bandit Problems”, Machine Mobile Robots using Genetic Algorithms,” IEEEE/IAS Learning: Proceedings of the TWelfth International Confer Int’l. Conf. on Industrial Automation and Control, pp. ence on Machine Learning, pp.