Applications of Artificial Intelligence to Space Shuttle Mission Control

Home , Flight controller

John F. Muratore TroyA. Heindel Terri B. Murphy National Aeronautics and Space Administration Lyndon B. Johnson Space Center Houston Texas

Arthur N. Rasmussen MITRE Corporation

Robert Z. McFarland Unisys Corporation

ABSTRACT Real time expert systems have been developed In the past, the MCChas relied on mainframe- by the National Aeronautics and Space based processing and display techniques which Administration (NASA) to monitor telemetry emphasizedthe use of highly skilled data from the Space Shuttle. These systems personnel knownas flight controllers to have been used during SpaceShuttle flights. monitor data, detect failures, analyze system This represents the first use of expert performance, and makechanges to flight systemsto makeflight decisions in a real plans. Figure 1 shows the working environment space mission. This paper describes the of the MCCand Figure 2 shows an example of requirements that led to the developmentof the displays provided by this system. This the expert system, the architecture used to worked well for programsof short - duration, obtain real - time performance, and the payoffs but as NASAlooks to operating long duration from the system. programs such Space Station Freedom, techniques are being explored to automate mission monitoring, display and analysis INTRODUCTION functions. During the recovery from the Challenger The successful launch of Discovery in accident, two real time expert systems were September 1988 represented a bright new implementedand certified for use in making beginning for NASA.It also represented a flight- critical decisions. This work was bright newbeginning in the Mission Control erformed by a team of flight controllers and Center (MCC)at the Lyndon B. Johnson Space ~nowledge engineers from a combined NASA- Center. For the first time knowledgebased. industry team using commercial hardware and systemswere used at this critical facility software. These two expert systems which which is the focal point for SpaceShuttle monitor Space Shuttle communications and the flight operations. Knowledge- based systems were used to monitor the Space Shuttle, Space Shuttle Main Engines were successfully detect faults and advise flight operations used in the STS- 26 Discoveryflight. personnel. This is the first use of knowledge THE PROBLEM based systems technology in a NASA spaceflight operational environment. Flight The MCCinformation systems are vital to the managementdecisions were madedirectly based safety and success of mannedspace flights on the results of knowledgebased systems. conducted by the United States. The This is a milestone in the application of centralized system currently employed artificial intelligence. presents only raw data to flight controllers with very little interpretation. All processing is contained in a single large Figure 1 - Space Shuttle Mission Control

EAST / COM SYNC PL PNL FM PhIL RCVR LCK 0 5OURCE KU PM.. PH" ERR O H U/L RATE DLPU COHERENT ISEARCH D CODE HF D~SP PI~R ANT 5EL ER ’DETECT D D/L RATE 1 GNC @~1~ ON MODE MAN ITR~CK D CODE 2 GNC ~11 ON ELEC KU 0PER D ENCR U/L CLR CLR 3 GNC ~11 ON BEAM Zi6 5YNC D D/L CLR CLR W S OSOFF XPDR 5EL ’DRTA GD D RCDR NSA NSR ¯ FM MODE R/L HDR OFF RCDR Y ~S -113~ PRE AMP LDR OFF ERROR R O.Z O.Z MODE OP’F H~TER ¢ Z B-MODE D --CCTV ANT SEL ER PA OFF OFF ~NGLE D D/L 5EL MODE OPC TEMP -84) °5¢ , , UHF ¯ TEMP 5V5 5EL--’ RFL 2 ¯ 1 O.? MODE ¥CU MN ~SECURE TV~ ~$PC - - TEC-- Z96-6ZZ Z59--979 D~NLIN~ ENA ~DU PNR ? 5M O D ZTS-gB7 Z~3-Z~T GRM SEL NORM TVPE ? BF5 D ~LC OUTPUT ? RECORDER5 DDH 0PS MODE TK ~TP DIR 5PMTN TEHR DDH I ZTI:Z2:¢O:2~ FR.,5 100 I 5TBV I O REV I 5T0P 98 DDH 2 271:18:53:33 FR.S 0 Z 5TBY I 0 REv I 5T0P ~8 DDH 3 Z71:ZZ:lO:20 FR/5 100 P/L STBY i 0 REV’i 5T0P 91 DDH ~ :71:ZZ:iO:ZO FR/5 i~0 FAULT BCE STRG 3 5TKR GPC 1Z3~ TIME 271:OS::S:lO.07

Figure 2 - Typical MCC Mainframe Display

16 mainframe computer with software which is Thefirst task of this project wasto provide difficult to changeand verify. This system an "i ntelligentassociate" to the flight presents data to the flight controllers, but controller monitoring the SpaceShuttle’s not information. It is the job of the flight communications and data systems. The controllers to convert this raw data to associate expert system is namedafter the information that can be used to managethe first flight controller position selected for mission. Teachingflight controllers to automation and is called the Integrated perform this task is a major training problem CommunicationsOfficer (INCO) Expert System. which typically takes two to three years to Development was started in August 1987 and complete. The nature of the mainframe - based the system was placed in the MCCin April MCCsystem, which mostly provides monochrome 1988. Approximately eight person years worth text displays, causes even.the most highly of effort and $400,000 of hardware were trained and motivated flight controllers to required to field this system. This system makeoccasional monitoring errors. In the was placed next to the INCOconsole which problem domain of the MCC,a flight allows operators to compareresults of the controller error can lead to grave conventional console to those of the expert consequences. ~iYuStemnng the(Figure STS- 3).26 flightThis systemof Discoverywas used in The problem of ensuring high -quality September 1988. decisions by the flight control team using this minimal information processing The INCO Expert System was implemented on a capability is further complicated by NASA’s conventional color graphics engineering unique bi - modal age distribution. Dueto the workstation. The Masscomp5600 with the Unix hiring freezes between the Apollo and Space operating system was chosen to be compatible Shuttle programs, the majority of NASA with other workstations in the MCC. personnel fall into two d!stinct groups: the younger "Shuttle- only group under 35 years Automated monitoring was performed by both of age and the more experienced "Apollo - era algorithmic and heuristic techniques. veterans" of greater than 45 years of age. Knowledge was represented both procedurally Theseveterans represent a dwi ndli ng supply in C languagecode and in rules. The of corporate knowledge and experience as they representation for specific knowledge was are promoted, retire or moveon to other driven by the complexity of the knowledge and activities. required rate of execution. In the Masscomp environment, conventional C programming As NASAmoves into the SpaceStation era, it lenerally executes faster than the is further confronted by the requirement to ~nterpreter in a rule - basedsystem. continuously operate the station for its twenty year lifetime. It is not reasonableto The procedural representations were built in expect that we will be able to maintain a a structured natural language and translated large work force of highly skilled flight into C. This was done by a tool created by controllers to perform the demanding the project called Computation Development workloads that are imposed by our current Environment (CODE). CODEallows flight style of systems monitoring over that amount controllers to specify monitoring algorithms ot time. in a high - level language and then generates the C code necessary to perform the NASA’S SOLUTION algorithm. The rule - basedrepresentation for both algorithms and heuristics were built Artificial Intelligence is a natural source using the CLIPSexpert system tool developed of techniques for converting data into by the Mission Planning and Analysis Division information, capturing corporate knowledge, at the Lyndon B. JohnsonSpace Center. and lowering operator training and response time. In 1987, Mission Operations Directorate One of the requirements placed on the INCO at the Lyndon B. Johnson SpaceCenter started Expert Systemis to advise flight controllers a project to apply the techniques and in real time. This typically meansfailure methodologies of AI such as expert systems detection within five secondsof an event. and natural languageinterfaces to real space This requires the expert system to have mission operations problems. direct electronic accessto the real time telemetry from the SpaceShuttle. A major requirement on the INCO Expert System was also that it be isolated from all of the

17 Figure 3 - Expert SystemWorkstation Installed Next To Conventional Console in MCCDuring STS- 26 existing MCCsystems so that problemsin the The third layer uses procedural techniques to stand - alone expert system could not affect implement domain - specific knowledge. This flight critical mainframeprocessing. The knowledgeis entirely algorithmic in nature. combination of these two requirements forced For example, in this layer the system may us to build a completely independentreal monitor a voltage and signal an alarm if the time telemetry processing capability into the voltage is below a required level. These INCOExpert System. algorithms were built using CODE. The stringent demandsof executing processes The fourth layer uses rule - basedtechniques supporting real time telemetry processing to represent both algorithmic and heuristic while operating under an unmodified Unix led knowledge. It is often easier to implement to the developmentof an innovative complexalgorithms, such as those that architecture tot meeting real time knowledge - perform overall systemsanalysis, in the rule based system needs. The architecture was base rather than in the layer three based on a four layer model (Figure 4). Raw procedures. This must be balanced with speed data enters the first layer; as it movesup of execution concerns. Items that had to through the layers, it is converted to higher execute once every second were implemented quality information. using the layer three procedural techniques. Thefirst layer performs basic data Rules execute as an embeddedcomponent of the acquisition tasks such as telemetry entire system. Rule based componentsare only decommutation.This layer is performed by a called into operation whenthe failure commercial telemetry hardware device which detection algorithms at the third layer transfers data into the Masscompvia a Direct detect a significant changein the system MemoryAccess (DMA) interface. status. In this way we improve overall system performance. The rules are implementedin The secondlayer contains generic data CLIPS and communicatewith layer three conversion algorithms that do not require procedures via shared memory domain - specific knowledge. For example, algorithms that convert telemetry data between different floating point formats are containedin this layer. This layer is performed in C on the Masscompworkstation. Layer 1 and 2 - Data Acquisition and NonDiscipline Specific Algorithms location: ADS-IO0 Hardware input: 192Kbps output: 2000 parameters/second

Layer 3 - Discipline Specific Algorithms location: MasscompSoftware input : 2000 parameters/second output: 100 facts/second

Layer 4 - KnowledgeBased Expert Systems location: MasscornpSoftware input: 100 facts/second output : 4 assessments~second

USER INTERFACE I FLIGHT CONTROLLER

Figure 4 - Layered Architecture of INCOExpert System

An interesting characteristic of this layered On several occasions during ground approachis that as data movesup the layers, simulations and shuttle flights, the system the total amountof data decreases, but the has detected failures that were undetected by information value of the data increases. For the flight control team. Sufficient example 192,000 bits of information enter confidence in the system has been gained so layer one, but the rule - basedexpert system that conventional equipment is beginning to only operates on 350 facts generated by the be replaced by expert systems and some layer three algorithms. These350 facts manpowerreductions will occur. contain important verified information about the system, where the raw telemetry bits by EXPERIENCEAND PAYOFF themselves do not uniquely identify conditions on the spacecraft. Whenwork was started on this expert system there was considerable debate on the expected Failures are detected by the system and payoff. Viewpoints centered around three flagged to the operator via a color graphic potential payoff areas. Each of these three interface in less than five seconds.Figure 5 views required different emphasis during the showsa typical display. This contains all of expert system development the information contained in the display from the mainframe system shownin Figure 2.

19 Figure 5 - INCOExpert System Schematic Display

The first view was to increase the quality The third view was that the use of expert and productivity of our current experienced systems should allow true manpowerreductions personnel. Proponentsof this viewpoint by automating part of the systems monitoring believed that the expert system should be job. This view requires deep knowledgeabout developed to allow an experienced operator to specific tasks as well as concentration on makebetter flight decisions. This could be fault annunciation software and reliability done by developing a system capable of issues. continuously analyzingall incoming telemetry to a depth that wouldbe impractical even for In Mission Control we have taken the approach an expert operator. that our first priority is quality of decisions. The INCOexpert system was Thepotential dollar value of this payoff is designedinitially to allow an expert INCOto difficult to quantify. A single gooddecision function more productively by relieving him which allowed completion of a several hundred of the mechanicaltasks associated with million dollar mission wouldclearly pay for scanning data. Our secondpriority was in a large amountof expert system work, but it presenting data to operators so that less is difficult to say that the gooddecision experienced operators could operate at high was completely the result of the expert proficiency levels. Our lowest priority was system. Developing an expert system to meet in reducing manpower.The INCOexpert system this goal requires the incorporation of deep does meet all of these goals toa varying knowledge about a given spacecraft monitoring extent. task. The INCO expert system captured deep The secondview was to use expert systems to knowledge about monitoring the Space Shuttle allow less experienced personnel to perform and in fact does allow operators to perform a at the level of moresenior personnel. This more thorough job of monitoring telemetry. It has a measurablecost benefit in that it contains knowledge about the monitoring task allows shorter training times. This viewpoint that is probablyonly sharedby five or six requires an emphasis on breadth of knowledge experts, allowing even experienced INCO’s to as well as improved human- computer operate with the benefit of this expertise. interfaces.

20 The INCO expert system hardware investment Basedon the experience with INCO, we feel was $400,000 and approximately 8 man- years of that it is possible to reducethe training effort ($480,000) over two years. With the time for a "first - time" flight controller reduction of the INCOteam scheduled for late from 2 years to approximately 1.5 years. Four 1989, we will achieve a cost - paybackin operators are in training at any time in the approximately two years. The cost - payback INCOarea resulting in a potential payoff of howeveris not nearly as important as the approximately 1 person - year per year fact that the the systemwill allow us to (approximately 80,000 per year). better use our precious and scarce resource Unfortunately we will not be able to achieve of experienced personnel. this benefit as long as it is required to train the flight controllers in both the OPERATORACCEPTANCE conventional mainframe system and the expert system. Weare trying to increase the The real measureof successfor this reliability and operator confidence in this enterprise is the acceptanceof real time system to allow for total committmentto the expert system technology by the mission expert system approach. In support of this operations community. Because the system was Oal we have removed two mainframe monitors developedprimari!y by flight controllers, ~r acceptance in mission operations occurred om the conventional INCO consoles and have almost immediately. A rapid prototyping replaced them with display units from expert methodology which allowed us to react rapidly system workstations. These expert system to changessuggested by the flight control workstation displays will be used as primary team also increased acceptance. On several tools bythe INCOsfor the STS- 29 mission occasions when the mainframe complex has scheduled in March of 1989. After STS- 29 we failed during simulations, flight controllers will remove two more mainframe display units. did not hesitate to use the expert systemas Weplan to start using the workstations as their only basis for flight decisions. the prime flight controller tool in late 1989 and to start to see sometraining benefits in The degree of acceptance was dramatically early 1990. demonstratedby the chain of events that led to our secondexpert system. In May 1988 the In late 1989,the INCOexpert system will shuttle flight controllers responsi ble for allow us to reduce the size of the INCO monitoring the main engines determined that monitoring team from 4 operators per shift to there were several failure modesof the main three operators per shift. Becausewe engines that required automated fault currently have five teamsof operators this detection. Flight controllers could not will allow us to save approximately 5 person - manually perform calculations and assessments years per year (approx $400,000per year). fast enough to meet the demandsof monitoring this high performance system in dynamic It is important to realize that this payoff will not result in lower operations costs or flight. lower levels of staffing. The Shuttle program The necessary fault detection routines were is under constant pressure to fly more often designedand built using the first three and to exact more performance from any layers of the IN COExpert System.All of the flight. This requires more manpower.The knowledge in this main engine system was personnel reduction from the expert system algorithmic in nature and of low complexity. will be re- invested to form a sixth INCO The nature of the knowledge combined with the flight control team. This will allow us to high - speed requirements for decisions during meet the demandsof the 1990 Shuttle flight ascent led to a decision to place all of our schedule. Use of the expert system has madea efforts in the first three layers. newsou rce of trai ned flight controllers available, which we can apply to new problems Development started in May 1988 and the and higher flight rates. The systemhas systemwas certified for use in August1988. allowed us to Do more for the same money" Three full time personnel were assigned to rather than "Do the samefor less money. this task, termed the Booster Expert System. The Booster Expert System provided a new and significant capability to Mission Control: Booster was certified for use in making flight- critical decisions and used during the STS- 26 flight. This is a major payoff to NASA becauseit is a large improvementin the quality of flight decisions during the TheINCO Expert Systemproject is the first dynamic ascent phase. significant operational use of knowledgebased systems technology in a NASA operational environment. It has shownthat FUTUREEFFORTS expert systemscan play an integral role in mannedspace flight operations. As NASAmoves Basedon the STS- 26 experience, this effort forward in future spaceactivities, expert is being expandedinto multiple new systemswill be there. disciplines such as mechanicalsystems, electrical power and guidance,navigation and control. In at least two areas, successful implementation of the expert system will result in small manpowerreductions. Each of these new systems represents new monitoring challenges, but the basic layered ACKNOWLEDGEMENTS architecture of the INCOexpert system will be used. The INCOExpert System project was co- funded Thegeneral applicability of this by three NASAorganizations. Funding was architecture was proved in 1988 when the provided by the Office of Aeronautics and system was used by NASAand the United States SpaceTechnology, the Office of SpaceStation Air Force for monitoring telemetry from and the Office of SpaceFlight. The effort experimental aircraft at the NASADryden also received funding from the joint Flight ResearchFacility and the Air Force USAF/NASAAdvanced Launch System Project Flight Test Center at EdwardsAir Force Base. Office. The effort was developedby a joint In each of these cases telemetry data from an NASA/industry team including membersfrom experimental aircraft was incorporated into Rockwell Space Operations, The Mitre the INCOstructure. In both cases the system was modified to the aeronautics applications Corporation, Unisys, and Dual and Associates. in less than 48 hours. Useof this structure The authors wish to thank each of these will allow operations personnel at these organizations for their support. The authors facilities to concentrate on expert system wish to acknowledge the programming support knowledgebase issues, rather than forcing provided to INCOby ThomasKalvelage, Daryl them to develop another real time Brown, Glenn Binkley, Erick Kindred, Cheryl environment. Whittaker and Debbie Horton. The Booster ExpertSystem was designed and programmedby Michael Dingier. The ADI emulat on was Thedifferent expert systemswill be designed and programmed by Mark Gnabasik. connectedby an Ethernet in late 1989. This Installation support was provided by James will allow us to experiment with cooperation Gentry of the Bendix Field Engineering betweenmultiple expert systems. Just as the Corporation. Dr John Bull and Dr Peter various flight controllers communicateand Friedland from AmesResearch Center, and Dr work together in a control room, the local Melvin Montemerlo, Gregg Sweitek, Lee Holcomb area network will provide us with the and Charles Holliman from NASAHeadquarters opportunity to allow the expert systems to provided valuable commentsand insights work together. throughout the development of the project. Part of the human- computerinterface for the Guidance, Navigation and Control Expert Systemwill include graphic displays of the astronaut’s flight instruments on ground workstations by reconstructing the displays from telemetry. The Attitude Direction Indicator (ADI) or astronaut’s "artificial horizon" instrument emulation is already complete and will be used during STS- 29 in March 1989.