Proceeding of the 6th International Symposium on Artificial Intelligence and Robotics & Automation in Space: i-SAIRAS 2001, Canadian Space Agency, St-Hubert, Quebec, Canada, June 18-22, 2001.

FULLY AUTOMATIC AND OPERATOR-LESS ANOMALY DETECTING GROUND SUPPORT SYSTEM FOR MARS PROBE "NOZOMI"

NAOMI NISHIGORI Fujitsu Limited 9-3, Nakase 1-chome, Mihama-ku, Chiba-shi, Chiba 261-8588, Japan Phone: 81-43-299-3248 Email: [email protected]

MASASHI HASHIMOTO Institute of Space and Astronautical Science (ISAS) 3-1-1 Yoshinodai, Sagamihara, Kanagawa 229-8510, Japan Phone: 81-42-759-8352 Email: [email protected]

AKINARI CHOKI Institute of Space and Astronautical Science (ISAS) 3-1-1 Yoshinodai, Sagamihara, Kanagawa 229-8510, Japan Phone: 81-42-759-8386 Fax: 81-42-759-8387 Email: [email protected]

MITSUE MIZUTANI Fujitsu Advanced Solutions Limited 1-2-4, Shinkoyasu, Kanagawa-ku, Yokohama-shi, Kanagawa 221-0013, Japan Phone: 81-45-438-2082 Email: [email protected]

Keywords: expert system, GEOTAIL, Mars probe, ISACS-DOC has caught many abnormalities including monitoring and diagnostic expert system, NOZOMI, serious problems which actually happened to satellite abnormalities, scientific satellite, spacecraft “NOZOMI” and warned to “NOZOMI” operators abnormalities, rule-based reasoning, heuristic about them immediately at SSOC. ISACS-DOC for classification problem. “NOZOMI” will be continuously used until the end of the “NOZOMI” mission to keep the operation safe Abstract under strict limitation of the operation budget at ISAS. In this paper, the process of ISACS-DOC development, This paper describes a fully automatic and operator- the improvement points that were performed, and less anomaly detecting ground support system named some actual examples of abnormalities that were ISACS-DOC for Japan’s first Mars probe “NOZOMI”. detected by ISACS-DOC will be reported as well as This system is the second application of expert the outline of ISACS-DOC itself. technique to daily operation of scientific spacecraft at the Institute of Space and Astronautical Science 1. Introduction (ISAS) in Japan. Many improvements are achieved based on the experiences of the first expert application The study of diagnostic expert system has been to monitoring and diagnosis system for geomagnetic becoming active since the second half of 1980s and observation satellite “GEOTAIL”. ISACS-DOC is also been applied to the space field. The first stage now used to keep the safer operation of “NOZOMI” as of diagnostic expert system treated problems as well as the first system for “GEOTAIL” at ISAS’ ‘heuristic classification problem’, namely, the Sagamihara Spacecraft Operation Center (SSOC). diagnosis was conducted by forward or backward

inference engines referring the knowledge database 2.2 Four Steps on GEOTAIL System Development that was previously defined using specialists’ knowledge or experiences. ISAS also decided to It was the first trial for ISAS to develop diagnostic apply rule-based reasoning diagnostic expert system to system for the actual scientific spacecraft, and we perform the safe operation of the spacecraft control on developed the system in following four steps: the ground with small number of operators. This (1) Prototype system for the limited subsystem : We system was named Intelligent SAtellite Control developed a prototype system to evaluate the Software -DOCtor (ISACS-DOC) and its first version effectiveness of the rule-based reasoning was developed and has been used to geomagnetic diagnostic system targeting to the spacecraft observation satellite “GEOTAIL” for almost nine commutation system only. We decided to use a years. Although model-based reasoning diagnostic commercially available application software to expert system has been very active and many reports reduce the development cost. This diagnostic have been able to find nowadays, ISAS decided to expert tools package running on the OS/2 apply the same technique to the second version of environment had three standard functions of the ISACS-DOC and successfully developed the fully knowledge database editor, the forward inference automatic and operator-less anomaly detecting ground engine, and the rule-based knowledge database. support system for Japan’s first Mars probe (2) Diagnostic system operated in manual mode : We “NOZOMI”. Two ISACS-DOCs for “GEOTAIL” developed the second-step prototype system and “NOZOMI” are both backing up daily spacecraft targeting the whole subsystems of the spacecraft. operations powerfully in ISAS. This paper reports The necessary data for diagnosis was manually on many trials and ideas on developing ISACS-DOC input to the system as we did not have much and also the latest status of ISACS-DOC for confidence that the system was useful to the “NOZOMI”. spacecraft operation. (3) Diagnostic system operated in on-line data 2. Process of ISACS-DOC Development feeding mode : It took 30-40 minutes to input necessary data to the system in manual mode, 2.1 Basic Concept of ISACS-DOC though it was recognized that the system could diagnose the spacecraft status almost correctly. Following is the basic concept in designing ISACS- The system was revised to the on-line data feeding DOC. Although these ideas were discussed at the system as almost all necessary data was input on a beginning of the first version of ISACS-DOC for real time basis from telemetry data. “GEOTAIL” launched in 1992, they were not changed (4) Diagnostic system cycled automatically : It was throughout the second version of system development strongly required to watch whole spacecraft status for “NOZOMI” launched in 1998 . all the time automatically during the real time (1) Operators can easily recognize abnormalities of operation and to warn operators of the spacecraft satellites/spacecraft by being provided appropriate abnormality when some trouble happened. The monitoring and abnormality detection capability system upgraded to repeat the diagnosis every even if they do not have enough knowledge. three minutes during the real time operation. (2) Workload on operators to use ISACS-DOC should be minimized. In a spacecraft operation, a significant amount of data (3) Knowledge database can be easily modified to is produced and several pages of quick look screens correspond to an unexpected situation after launch. are necessary to display them. The operator would (4) A fatal damage of the satellites/spacecraft can be be flustered without the appropriate knowledge on avoid even if the operator fails to find a complete how to deal with the identified trouble. We noticed solution to the trouble by taking first aid action. through the actual spacecraft operation that ISACS- (5) The necessary information for diagnosing DOC was very useful to avoid overlooking the satellites/spacecraft can be obtained not only from spacecraft contingencies as the system automatically the telemetry data but also from the ground checking 570 items of spacecraft health condition tracking stations. every few minutes. ISACS-DOC for “GEOTAIL” found several spacecraft contingencies in the

Page 2

Figure 1 Monitor Screen of Solar Particle Monitor (SPM) Including Reproduced Data communication subsystem, control subsystem and following reasons, though the model-based reasoning scientific instruments and helped the spacecraft technique becoming active: operators to cope with them. l It would have been very hard and difficult to verify the diagnoses model of “NOZOMI” before 2.3 New Ideas on NOZOMI System Development spacecraft launch with low cost, if the model-based Learned from GEOTAIL experiences reasoning technique was applied to the system. l Many experiences and knowledge obtained from It is possible to accumulate statistical data of the the system development of ISACS-DOC for diagnostic knowledge for cars, electric appliances for “GEOTAIL” seemed to be useful and applicable to home use because there are plenty of exactly the same other future satellite operations by relatively little or almost the same products on the market. It is very efforts. easy to obtain the information about what kind of troubles happens at what rate. Using this information, ISACS-DOC for “NOZOMI” is developed according it is possible to analyze the troubles statistically. to the following ideas. Some of them are introduced However, there is no exactly the same scientific based on lessons learned from ISACS-COD for spacecraft. So the knowledge database for a “GEOTAIL". scientific spacecraft mainly depends on the design knowledge. For this reason, a wide variety of design (1) Wide range of information can be gathered knowledge for developing spacecraft has to be through an on-line network to realize extensive and accumulated in the diagnostic expert system. The reliable diagnosis , and all data used for diagnosis knowledge database of the ISACS-DOC for are saved in the system and can be used for “GEOTAIL” was made using a lot of design checking a long-term trend of the data. knowledge obtained by interviewing 30 or more (2) Only reliable results should be shown in order to experts. We also decided to apply the rule-based avoid giving operators unnecessary concerns. reasoning diagnostic system for “NOZOMI” from the Only related facts should be shown if the

Page 3

coordinate the differences between actual status of “NOZOMI” after launch and the preset-knowledge database before launch, to add insufficient knowledge and tune the data collection function. It took another 5 months to finely adjust the knowledge database through daily operations of ISACS-DOC. After that, operation of ISACS-DOC was placed under the control of the “NOZOMI” operation team on June of 1999. Furthermore, following new functions were added to ISACS-DOC for “NOZOMI” according to the actual spacecraft operation until now.

(1) Diagnosis for Data from Onboard Data Recorder “NOZOMI” is operated using only one tracking antenna at Usuda Deep Space Center (UDSC) of ISAS usually. SSOC cannot make contact with “NOZOMI” during the invisible time for long hours. In addition to that, “NOZOMI” lost its S-band telemetry capability after launch because of the TMS malfunction that happened on July 5, 1999. SSOC can not see the real-time data while the remaining X- band transmitter is sending reproduced data from the Figure 2 Window of Long-Term Power Supply onboard data recorder. Therefore, it is very Monitor important for ISACS-DOC to be able to diagnose the data sent from the onboard data recorder as well as diagnostic result is uncertain. To perform this the real-time data. The “NOZOMI” operation team idea, parallel formed decision tree is more can quickly see what happened during the invisible appropriate for the spacecraft diagnosis than time by this capability. Figure 1 is an example of the serially formed decision tree. Solar Particle Monitor (SPM) including reproduced (3) Since “NOZOMI” is the deep space mission, data from the data recorder onboard. communication links should be carefully watched by comparing the real receiving levels of down- links and up-links with estimated valued which are calculated from antennas patterns, attitude of “NOZOMI”, distance between “NOZOMI” and ground tracking antenna, performance of ground station, etc. (4) Transmission time between “NOZOMI” and ground station should be taken into account in designing ISACS-DOC since it reaches up to 30 minutes of round-trip time. This is very important when “NOZOMI” data and ground station data are used together. (5) Important items for diagnosing should be reflected on spacecraft designing.

2.4 Many Improvements after NOZOMI Launch

Five months of test running were allocated to Figure 3 Screen Display of Range-Rate Measurement

Page 4

Figure 4 System Configuration of ISACS-DOC for NOZOMI

3. Running Status of Latest ISACS-DOC (2) Trend-Graph Drawing of Long Period Data A long-term variation of some data is essential to According to the above mentioned implements, the investigate the cause of degradation, malfunction and fully automatic and operator-less anomaly detecting so on. For examples, solar-cell efficiency, battery ground support system are powerfully backing up capacity, A/E of thermal control material, etc. change daily “NOZOMI” operation The outline of ISACS- slowly with time. Figure 2 is an example screen DOC for “NOZOMI” including system configuration, window of the long-term power supply monitor shown diagnosis flow, and also some actual examples of by ISACS-DOC for “NOZOMI”. diagnosing results will be introduced in this section.

(3) Diagnosis of Range-Rate Measurement System 3.1 System Configuration The trajectory of “NOZOMI” is determined from the data obtained by range and range-rate measurement ISACS-DOC consists of two computers as shown in systems. ISACS-DOC is watching the health of the figure 4. They have the following functions. range-rate measurement system by monitoring the difference between the measured value and expected (1) Data collecting function value that is calculated from the latest trajectory This function collects the data to be monitored and elements. This function is also helpful to monitor the diagnosed from each data server of various ground performance of trajectory change operation roughly operation systems through ISAS’ ground network in though its major role is to watch the health condition real-time, edits the files of these data, and transfers of the range-rate measurement system. Figure 3 these files to another computer that has the diagnosis shows a screen display of the range-rate measurement function. Below are the data to be collected. monitor. l Telemetry data sent from the spacecraft. l Status data of ground tracking station. l Orbit data.

Page 5

Figure 5 Diagnosis Screen of TMS Temperature High

l Aspect data of ”NOZOMI”. DOC. About 610 of questions are automatically This function is repeatedly practiced while the issued and approximately 460 of results are prepared computer is working. in the present ISACS-DOC.

(2) Diagnosis function 3.2 Diagnosis Flow All actual diagnoses are practiced on a personal computer (PC). This function is realized with the ISACS-DOC for “NOZOMI” is always waiting for commercially available application software named the diagnosis before the start of “NOZOMI” “Manadeshi-kun” running on Microsoft® Windows operation at SSOC. This system automatically NT® environment. The diagnostic information files starts diagnosing when the necessary data are made by the data collecting function are transferred to prepared in the WS in Figure 4 during real-time the PC automatically and used for diagnosis. The operation. The diagnosis is cycled automatically rule-base knowledge database of “NOZOMI” is stored about every 5 minutes. The diagnosis pauses on the PC. Diagnosis is executed on the basis of this when the data is not prepared for some reason, for database. The database can be easily modified example, in case that the ground-tracking antenna is according to the actual-status change of “NOZOMI”. not ready. The diagnosis is executed in all of the The data flow in the data collecting function are also areas of “NOZOMI”. ISACS-DOC warns the shown in Figure 4. operators when it finds something abnormal. The operators can get the information about the details The scale of knowledge database in ISACS-DOC is of the abnormality including first aid action to often changed according to status change of rescue “NOZOMI” from fatal damage that may be “NOZOMI” and newly obtained information that is caused by the abnormality. Five standard windows useful to improve the diagnostic capability by ISACS- on the screen of ISACS-DOC for “NOZOMI” are

Page 6

Figure 6 Diagnosis Screen of Helium Gas-Tank Pressure Low prepared to display the diagnosed information of major malfunction of helium gas leakage in the abnormal items, explanation of each abnormality, propulsion system of “NOZOMI” suddenly occurred normal and actual status/values causing the at beginning of May in 1999. ISACS-DOC showed abnormality, related figures/graphic data, urgent levels, the primary pressure of the helium gas-tank at a contact information such as telephone numbers of dangerously low level and still decreasing. The senior engineers/scientists who can supervise the leakage was stopped by sending the proper commands. further contingency operation for the abnormality, Fortunately, there was no serious impact on the Mars urgent commands to save the probe from catastrophe observation plan by “NOZOMI”. (first-aid commands), and some common data like distance between the probe and the earth, etc. as Monitoring and diagnostic items related to the serious shown in Figure 1. problems are carefully refined and reinforced. Communication-link monitor is one of the examples. 3.3 Operation Results The difference between actual and estimated receiving levels of S-band up-link and X-band down-ink is ISACS-DOC has issued a lot of warnings including reduced by reevaluating the performance of the some serious unexpected problems. The screen in communication system at UDSC, antenna patterns and Figure 5 warns the temperature of the S-band temperature dependence of AGC calibration data of transmitter (TMS) onboard “NOZOMI” is rather high, the receivers onboard “NOZOMI”. Figure 7 is a though it is still within the allowable range. window screen of the communication links monitor. However, the temperature change should be carefully monitored after this caution. The related data like the 4. Conclusion temperature history of TMS and the first aid action, if necessary, automatically appears on the screen. ISAS has been studying to develop the rule-base Figure 6 is another and more hazardous example. A reasoning diagnostic expert system to perform the safe

Page 7

Figure 7 Screen Display of Communication Links Monitor of ISACS-DOC spacecraft operation under strong demand of cost the knowledge database of ISACS-DOC. The reduction since 1990. After many years’ trials and authors also would like to acknowledge Professors efforts, ISAS successfully developed a fully automatic Ichiro Nakatani, Toshifumi Mukai, and Hajime and operator-less anomaly detecting ground support Hayakawa of ISAS for their great supports of this system called ISACS-DOC for Mars probe development. “NOZOMI”. ISACS-DOC for “NOZOMI” has been used to keep the safer operation of “NOZOMI” on References dairy basis at ISAS from February, 1999. The effectiveness of this system has been already shown [1] Hashimoto, M. Nishigori, N., and Mizutani, M.: by finding some abnormalities of “NOZOMI”. Running Status of Monitoring and Diagnostic “NOZOMI” will be injected into Mars’ orbit on Expert System for Mars Observer “NOZOMI”, January 2004 and collect scientific data to study the Proceedings of the 22nd International Symposium structure and dynamics of the Martian atmosphere on Space Technology and Science, ISTS 2000-f- on its interaction with solar wind approximately for 08, 2000. 2 years after the injection. ISACS-DOC for [2] Yamada, T.: Cost Effective Development of “NOZOMI” will also support this operation Communication Systems for Space Operations, powerfully. Proceedings of the 2nd International Symposium on ‘Reducing the Cost of Spacecraft Ground The discussion of next ISACS-DOC for sample-return Systems and Operation’, RAL.GS2.12, pp.12.1- mission of ISAS named “MUSES-C” has already 12.7, 1997. stated. A lot of lessons learned through the [3] Hashimoto, M. Nishigori, N., and Mizutani, M.: development of ISACS-DOC for “NOZOMI” will be Operating Status of Monitoring and Diagnostic introduced into the next ISACS-DOC. Expert System for Geomagnetic Satellite GEOTAIL, Proceedings of the 2nd International The authors would like to thank all of the people for Symposium on ‘Reducing the Cost of Spacecraft giving us precious information of each onboard Ground Systems and Operation’, instruments of “NOZOMI” and ground support RAL.GS2.73,pp.73.1-73.8,1997. systems. These information are essential to construct

Page 8