Neuro-Fuzzy Dynamic Programming for Decision-Making and Resource Allocation during Wildland Fires A thesis submitted to the Graduate School of the University of Cincinnati in partial fulfillment of the requirements for the degree of Master of Science in the School of Aerospace Systems of the College of Engineering and Applied Science by Nicholas P. Hanlon B.S., Georgetown College, 2001 M.B.A., Northern Kentucky University, 2006 Committee Chair: Kelly Cohen, Ph.D. Abstract Fire is a natural agent of change for our planet’s survival and has the capability to cause devastating effects (economical, societal, environmental, etc) when it encroaches into our daily lives. In the midst of a wildland fire, incident commanders are bombarded with massive amounts of data, accurate or not, and must make real-time decisions on how to allocate available resources to extinguish the fire with minimal damage. The scenario is modeled as an attacker-defender style game, such that the defender (resources with fire retardants) is protecting its assets (homes, businesses, power plants, etc) while the attacker (wildland fires) is attempting to deliver maximum destruction to those assets. The problem can be formulated in terms of optimal control theory, utilizing the gold standard of optimization, Dynamic Programming (DP), to exhaustively search the solution space for the minimized cost. However, its drawback is directly related to its method of finding the optimal solution: the exhaustive search. The amount of processing time to compute the minimum cost exponentially increases with the complexity of the system. For this reason, the DP approach is generally executed offline for real-world applications. Due to the large solution space of a wildland fire scenario, execution of DP offline is problematic as resource allocation decisions must be made in real-time. The current research effort seeks to show a new and unique control algorithm, based on Neuro- Fuzzy Dynamic Programming (NFDP), that can nearly replicate the DP algorithm results but can execute in real-time and remain robust to uncertainties. An artificial neural network provides the approximate cost-to-go function for the DP, fulfilling the need for real-time execution. The neural network is trained by approximate policy iteration using Monte Carlo simulations. Since our sensors may provide inaccurate or incomplete data of the environment, a fuzzy logic component is integrated to provide robustness in the system. The problem is also extended to include multiple layers of defense as I opposed to a one layer attempt to eliminate the incoming threat. The multi-layered defense requires a unique approach in the NFDP algorithm that calculates future expected costs since a fire must successfully elude three layers of defense to constitute an attack on an asset. Four control methodologies are examined in the research: a greedy-based heuristic, DP, NDP (Neuro-Dynamic Programming), and NFDP. DP and the heuristic are used as benchmark cases; the premise of the heuristic approach is to protect the highest valued assets at all costs. The control methodologies are compared based on three parameters: processing time, remaining asset health, and scalability. The processing time quantifies the requirement of real-time decisions. The asset health is a measure of how well the defender protected its assets from the attacker. Scalability is how well the algorithm scales with increased complexity. With proper adjustments to the architecture and training techniques of the artificial neural network and fine-tuning of the fuzzy controller parameters, NFDP illustrates its ability to perform real-time decision-making, obtaining near optimal results in the presence of uncertainty in the sensor data, and scales well with increased complexity. II Blank Page or Copyright Notice III Acknowledgements First and foremost I would like to thank my family, especially my wife Dr. Jaime Dann Hanlon for her never-ending love, support, and great patience at all times. She gave me the confidence to pursue this degree and the journey would never have been possible without her encouragement. I would like to express my sincere gratitude to my advisor Dr. Kelly Cohen for his support, guidance, and wealth of knowledge he provided for this thesis, especially in the area of fuzzy logic. I thoroughly enjoyed our meetings, whether academic or personal in nature, and I look forward to our future work. I would also like to thank Dr. Manish Kumar for his invaluable assistance and guidance for the thesis. Dr. Kumar was always kindly available for questions on the thesis and helped provide direction in the areas of Dynamic Programming and Neural Networks. I am also grateful for the committee members, Dr. Bruce Walker and Dr. Grant Schaffner, for their time to review my work and provide feedback. Finally, I would like to thank Dr. Benjamin Tyler and Dr. Praveen Chawla of Edaptive Computing, Inc. Dr. Tyler and Dr. Chawla provided the means for this project to exist and were an immense help in the formulation of the control algorithms. IV Table of Contents Acknowledgements ...................................................................................................................................... IV Table of Contents .......................................................................................................................................... V List of Figures ............................................................................................................................................... VI Chapter 1: Introduction ................................................................................................................................ 1 Chapter 2: Literature Survey ......................................................................................................................... 5 Dynamic Programming............................................................................................................................ 11 Artificial Neural Networks ....................................................................................................................... 14 Fuzzy Logic .............................................................................................................................................. 17 Previous Research ................................................................................................................................... 19 Reasoning Behind NFDP .......................................................................................................................... 22 Chapter 3: Problem Formulation and Scenario Description ....................................................................... 24 Vector Assignment .................................................................................................................................. 25 Uncertainty Analysis ............................................................................................................................... 32 Figures of Merit ....................................................................................................................................... 33 Software and Platform ............................................................................................................................ 33 Syscape .................................................................................................................................................... 34 Chapter 4: Benchmark Methods ................................................................................................................. 36 Greedy-Based Heuristic .......................................................................................................................... 36 Dynamic Programming............................................................................................................................ 36 Chapter 5: NDP Formulation ....................................................................................................................... 40 Neural Network Architecture.................................................................................................................. 41 Challenges ............................................................................................................................................... 45 Chapter 6: NFDP Formulation ..................................................................................................................... 46 Fuzzy Inference System ........................................................................................................................... 47 Challenges ............................................................................................................................................... 49 Chapter 7: Simulation Results ..................................................................................................................... 51 Sensitivity to Engagement Probability .................................................................................................... 51 Uncertainty Analysis ............................................................................................................................... 53 Scalability ................................................................................................................................................ 58 Chapter 8: Conclusions & Recommendations for Future Work ................................................................