Trans. JSASS Space Tech. Vol. 7, No. ists26, pp. Tf_1-Tf_6, 2009

ISACS-DOC: Monitoring and Diagnostic System for and

By Mitsue MIZUTANI1), Toshinori HIROSE2), Ryoji TAKAKI3) and Hideyuki HONDA3)

1)Solution Planning Dept., Fujitsu Advanced Solutions Limited, Yokohama Japan; 2)Science Solutions Div., Fujitsu Limited, Chiba Japan; 3)Institute of Space and Astronautical Science, Japan Aerospace Exploration Agency, Sagamihara Japan

(Received May 8th, 2008)

ISACS-DOC (Intelligent Satellite Control Software-DOCtor), which is an automatic monitoring and diagnostic system for scientific satellites or spacecraft, aims to rapidly and accurately capture important changes and sign of anomaly during daily satellite operations. After three systems for deep space missions, the new generation of ISACS-DOC with a higher speed processing performance had been developed for the satellites in earth orbit, AKARI and HINODE. This paper reports on the newest ISACS-DOC about enhanced functions, operating status, and an approach to create standards to build and keep up the knowledge data base. Continuous enhancements through the actual operations are the advantage of ISACS-DOC.

Key Words: monitoring-and-diagnosing, knowledge-base, satellite, safe-operation, cost-reduction

1. Introduction sample-return probe (launched in 2003). ISACS-DOC is independent from any other existing In order to satisfy the high-grade observation satellite operation systems and has high flexibility to add requirements of a scientific satellite, the onboard and new functions and monitoring methods, so it acts as a ground operation systems are becoming more complicated pilot system to improve satellite operation systems. The and elaborate. For this reason there is a real need to monitoring and diagnostic techniques verified by this ensure accurate and safe satellite operations. Since 1992 system will contribute to the continuous improvement of Institute of Space and Astronautical Science of Japan satellite operational technologies through their Aerospace Exploration Agency (ISAS/JAXA) has been incorporation into future satellite control systems. conducting research and development on a ground system The next-generation of ISACS-DOC with a higher for monitoring and diagnosing the status of satellites. At processing performance has been developed (4) and put the beginning of the research and development of the into operation for AKARI (-F) since it launched in system, no similar systems which diagnose a whole 2006. Infrared Imaging Surveyor AKARI is in a satellite in practical use could be found in Japan and any low-altitude earth orbit and different in various aspects other countries. This system is called Intelligent Satellite from the deep space probes targeted by the past Control Software-DOCtor (ISACS-DOC) (1, 2, 3). ISACS-DOC systems. The greatest difference in terms of The purposes of ISACS-DOC are to accurately grasp ISACS-DOC for AKARI is the requirement for signs of serious change and abnormality of any onboard high-speed data processing. The short visible time (deep system during regular operation without the need of space spacecraft HAYABUSA: 8 hours, AKARI: 10 experts in the satellite control room, to issue an alert on a min/path and 4 paths/day) and high-rate downlink data display or send out an e-mail, to enhance the safety of transfer (HAYABUSA: 16 Kbps, AKARI: 4 Mbps) means satellite operations, and consequently to reduce that a large volume of data needs to be processed in a operational costs. ISACS-DOC can be positioned as a short period of time. Moreover, it is preferable to process system which supports monitoring by an operator in both real and reproduced telemetry data at the same time. regular operation and diagnosis by an expert in abnormal For high-speed data processing, the processing operation. If the cause of an incident or irregularity can be performance of ISACS-DOC for AKARI is enhanced by identified with a high degree of reliability, the system conducting optimization of data processing procedures. should serve up diagnostic results and solutions. However, The newest ISACS-DOC has been developed for if the cause cannot be identified from detected signs, it is HINODE (SOLAR-B) based on the system for AKARI. preferable to consult an expert, presenting the related HINODE is also in a low-altitude earth information. ISACS-DOC should help the expert to orbit and its requirements for monitoring and diagnostic diagnose the incident or abnormality. To do so, the are similar to AKARI. information from both the satellite and ground facilities Both of ISACS-DOCs for AKARI and HINODE needs to be integrated and presented. continue to operate successfully. So far, ISACS-DOC system has been applied to the This paper reports on the enhanced functions, operating following deep space missions - the Geomagnetic status, and an approach to create standards to build and Observation Satellite (launched in 1992), Mars keep up the knowledge data base of the monitoring and Explorer (launched in 1998), and a diagnostic system ISACS-DOC.

Copyright© 2009 by the Japan Society for Aeronautical and Space Sciences and ISTS. All rights reserved. Tf_1 Trans. JSASS Space Tech. Japan Vol. 7, No. ists26 (2009)

Figure 1: Operation image of ISACS-DOC for satellites in earth orbit 2. Overview of ISACS-DOC Figure 2 shows the main window of ISACS-DOC. 2.1. Outline of ISACS-DOC Detected anomalies are listed in the main part of the Figure 1 shows an operation image of ISACS-DOCs for window, which is the important region for AKARI and HINODE which are satellites in earth orbit. ISACS-DOC users. Each detected anomaly is linked to its ISACS-DOC system receives both real time and monitor window (Fig. 3), so that the users can grasp the reproduced telemetry data from the data distribution current status of the anomaly. server, and monitors and diagnoses them. The diagnosed results are sent by e-mail and referable though web browser display. ISACS-DOC consists of following 6 main functions.

1. Import: the system imports knowledge data base and telemetry definition data base, and extracts telemetry entry, which are required in monitoring and diagnosing. 2. Telemetry reception: the system receives telemetry data in real time, decomposes them to each telemetry entry, extracts status values, converts to physical Figure 2: Main window of ISACS-DOC values and finally creates primary data, according to the telemetry definition data base. 3. Monitoring data creation: the system creates monitoring data which are necessary for monitoring and diagnosing, based on the knowledge data base. A filtering procedure and limit check are applied to primary data, creating secondary data. Compound data are created from multiple data: secondary data and compound data, as needed. 4. Monitoring and diagnosing: the system monitors and diagnoses according to the definition of knowledge data base. 5. Display: monitored and diagnosed results are shown through a HTTP server. 6. Notification by e-mail: monitored and diagnosed results are sent by e-mail after the spacecraft operation or monitoring and diagnosing of reproduced telemetry Figure 3: Monitor window with detailed information data are completed.

Tf_2 M. MIZUTANI et al.: ISACS-DOC: Monitoring and Diagnostic System for AKARI and HINODE

The monitor window (Fig. 3) shows detailed 3.1. System improvement information: 1) current values of the anomaly data, 2) The following functions were enhanced in the body of conditions of anomaly detection, 3) setting values of limit ISACS-DOC. checks, 4) time series plots of not only the anomaly data a) E-mail sortation but also the related data, which provides valuable clues The purpose of the e-mail sortation is to serve and for an understanding of the circumstances, and 5) direct the alert e-mails to the appropriate persons. messages of anomaly descriptions and the way to ISACS-DOC notifies monitored and diagnosed results response. by e-mail. As the e-mail is made after the series of On the left side of the main window (Fig. 2), there is a telemetry data form the spacecraft real time operation or list menu for all monitoring items, which show current the data reproduced and downloaded operation are status of monitoring data regardless of whether an completed, the monitored and diagnosed results contain anomaly is detected. Each listed monitoring item is linked some monitoring alert items in a variety of subsystems (if to its monitor window (Fig. 3). any). The function of e-mail sortation filters the results, edits the alert e-mail by the kind of subsystems, then 2.2. Knowledge data base serves and directs the e-mails to the appropriate persons. Diagnostic requirements are defined in the knowledge b) Watchdog of ISACS-DOC itself data base. The data used for diagnosis and detecting the The purpose of the watchdog of ISACS-DOC is to early signs or anomaly are described on mainly these three monitor ISACS-DOC system itself and notify the kinds of spreadsheets, as follows. administrator of ISACS-DOC by e-mail of operation status of ISACS- DOC system. 1. Diagnostic input data (status and analog data): the ISACS-DOC system works with those three processes, input data which are necessary for monitoring and the monitoring process (real time), the monitoring process diagnosis are defined in these status and analog data (reproduced), and the monitoring control process. The sheets. For example, the telemetry name, the limit function of the watchdog of ISACS-DOC checks itself values for limit check, the collecting cycle and period every 30 minutes whether those processes are alive or not, of the data, the filtering type and valid/invalid and if an ISACS-DOC system error should be found, conditions. sends DOC-error e-mail every 30 minutes. If no 2. Compound data: the new data which are created with ISACS-DOC system error is found, it sends DOC-alive input data or other compound data in ISACS-DOC are e-mail once a day. The watchdog e-mails are also sent to defined in this compound data sheet. Each defined the manager of the satellite project team. compound data has the method of how to create itself. c) Intelligible display of compound data ISACS-DOC has not only a number of standard The purpose of the intelligible display of compound methods which are increasing, but also the interface of data is to explain to users how the compound data were user plug-in to solve new requirements. The compound created with other telemetry or compound data on the data and its flexibility are a key to realize higher level online monitor display of ISACS-DOC. of diagnosis and to advance ISACS-DOC itself. Figure 4 shows the intelligible explanation display 3. Monitoring and diagnosis requirements: monitoring window of compound data. ISACS-DOC creates and diagnosis requirements are defined in this sheet. compound data according to the definition of the Each entry has conditions of anomaly detection, the knowledge data base, such as to configure the calculation diagnostic cycle, the importance level, the time limit result with telemetry data, to configure start time when for response. It has also the specification of the data certain conditions are met, and so on. The intelligible which are used to draw a graph, output messages as display shows the definitions of the knowledge data base. anomaly descriptions and the way to response. They Each monitor window (Fig. 3) has the entrance to the are displayed in the monitor window with detailed explanation window of a monitoring item, and if the information. monitoring item uses compound data, the name of the compound data is the entrance to the intelligible The support functions to build the knowledge data base explanation window of the compound data. easily and correctly are realized by using macro features of the spreadsheet software in the knowledge data base file. They decrease input loading of users and check consistency of the knowledge data base.

3. Enhancement through the Operation

After ISACS-DOC for AKARI was put into the real operation, some lack of features and many improvements became clear. This chapter describes the recent enhancements of ISACS-DOC.

Figure 4: Explanation window of compound data

Tf_3 Trans. JSASS Space Tech. Japan Vol. 7, No. ists26 (2009)

d) Useful graph functions  Understanding of trend The purpose of the useful graph functions is to grasp the For some physical values, the tendency and the rate of trends or details of the data. ISACS-DOC has an offline increasing or decreasing are important. ISACS-DOC monitor, and the users can get out any period of specified understands the tendency and the rate, then detects the data to the offline trend graph. And the graph is alert. expandable to grasp the detailed changes and the values of 3.2. GUI-based editor for the knowledge data base the data. Diagnostic requirements are defined in the knowledge Figure 5 shows an offline trend graph. The part of the data base. The data used for diagnosis and how to monitor graph ranged specification in a framed rectangle is and detect the early signs or anomaly are described on the expanded (Fig. 6). spreadsheets. The diagnostic knowledge data of AKARI and HINODE were collected on these spreadsheets in the developing phases. Now in operating phases, it is expected to keep up more timely and efficiently the knowledge data base following the status of the satellite operation. To facilitate this, a GUI-based editor has been developed to make and maintain the knowledge data base easily and efficiently, as only a few people familiar with ISACS-DOC system were able to input or change the knowledge data base in some of the previous ISACS-DOCs. This was one of the main issues which should be resolved in ISACS-DOC system. The editor is designed to allow experts who are unfamiliar with the ISACS-DOC system to maintain the knowledge data base running on ISACS-DOC in a timely Figure 5: Offline trend graph fashion. The editor has several functions to reduce input loading of users and to check consistency of the knowledge data base. It displays a window for each knowledge object (Fig. 7, 8), has wizards to input knowledge data (Fig. 9), pull-down menus by which users can select a suitable input value, and automatically imports the telemetry definition data base. Several values which are defined in the telemetry definition data base, are used as default values or are displayed as reference values for data input. Moreover, the following functions are realized in order to ensure consistency of the knowledge data base, format check (required item, fixed word, data type and number of characters)

Figure 6: Expanded graph display

e) Newly added monitoring methods The purpose of the new monitoring methods is to actualize more complicated and useful monitoring and diagnosis. The new methods are enhanced with greater definition of the knowledge data base.  Detection of continuing status There is the type of alert that some conditions not only are met but also continue for a while. ISACS-DOC recognizes the continuing status and period, then detects the alert.  Reference to previous values Some counter or status items have to be watched the changes of the values. ISACS-DOC refers to previous values and compares the current and previous values, then detects the alert. Figure 7: Compound data definition window

Tf_4 M. MIZUTANI et al.: ISACS-DOC: Monitoring and Diagnostic System for AKARI and HINODE

and consistency check (duplication of entry, HINODE is operated as visible real-time paths about relation of limit values, relation between collecting four times a day. And the reproduced telemetry data sets cycle and collecting period of data, conditional from HINODE are received from seventeen to twenty-two expression for anomaly detection, and lack of times a day. Therefore ISACS-DOC for HINODE required data for diagnosing). monitors and diagnoses HINODE and sends OK or alert (usually OK) e-mails more than twenty times a day. AKARI and HINODE project team members comment that detection of alerts is of course important but daily OK e-mails make them happy, and the monitor graphs are useful both in alert and fine operation time. 4.2. Detected the first sign of the onboard coolant boil-off for AKARI The far- and mid-infrared all-sky survey, an AKARI main subject, had been successfully performed in the expected time frame of about 500 days, and finished at 08:33 UT on August 26, 2007, due to the boil-off of the onboard coolant liquid helium. Since then AKARI moved to observations with the near-infrared camera operating at a warmer temperature maintained only by the mechanical coolers. ISACS-DOC for AKARI detected the first sign of the rapid temperature increase of the cryogenics which would indicate the boil-off of the onboard liquid helium. The Figure 8: Result definition window sign was obtained at a down-link station of the Ground Station, where data could not be monitored in real time. ISACS-DOC immediately notified experts in AKARI project team by email of the alert. The early alert helped them to make possible the early decisions for the next operation within 2 hours after the notification, and the immediate countermeasure at the next command-operation contact. Figure 9: Condition definition wizard

The security functions are also added to the GUI-based editor. It has the log-in window, and the only authenticated users can edit the knowledge data base. The knowledge data base makes a backup copy file itself automatically before the overwrite-save-command is executed. In addition, the knowledge data base understands itself the changes of the knowledge data. It records automatically by whom and when changes are made in any one knowledge-item when the overwrite- save-command is executed. The GUI-based editor helps to build and keep up the knowledge data base more certainly, efficiently and safely. The editor has already been in use by AKARI and HINODE project team members to maintain the knowledge data base.

4. Operation Results of ISACS-DOC

4.1. Daily operating status Now, AKARI is operated as visible real-time paths about three times a day. And the reproduced telemetry data sets from AKARI are received two or three times a day. Therefore ISACS-DOC for AKARI monitors and diagnoses the AKARI and sends OK or alert (usually OK) e-mails five or six times a day. Figure 10: Graphs at the time of the coolant boil-off

Tf_5 Trans. JSASS Space Tech. Japan Vol. 7, No. ists26 (2009)

Instruction Administrator Person in charge of each manual of Definition of subsystem Knowledge ・ Users Telemetry ・ Automatically import templates ・ Operating parameters definition ・ Subsystems data base of Knowledge data base

Knowledge data base Person in charge of each file Sample subsystem knowledge Definition of data base file ・ Diagnostic input data ・ Compound data ・ Diagnosis requirements Knowledge templates

Figure 11: Process image of Knowledge data base with Knowledge templates

Figure 10 shows the monitor graphs of ISACS-DOC for AKARI at the time of the onboard coolant liquid helium 6. Conclusions boil-off. ISACS-DOCs for AKARI and HINODE are used in 5. Knowledge Templates daily operation of the satellites. ISACS-DOCs for AKARI and HINODE will continue to assist safe operation of It is distinctly important for ISACS-DOC to collect AKARI and HINODE until the end of the missions. diagnostics knowledge, to produce and maintain a knowledge data base at a useful and high level. Many Acknowledgements knowledge data bases were previously collected, largely, The authors are pleased to acknowledge AKARI and manually by experts and those with experience, and then HINODE project team members for their helpful incorporated into ISACS-DOCs. However greater comments and suggestions. They are using each efficiencies are expected. ISACS-DOC in the real operation of the satellites. Now, we are considering the preparation of knowledge The authors are also deeply grateful for the longtime templates about scientific satellites by utilizing those research and development by Dr. Hashimoto M. for experiences so far. We are analyzing and arranging the ISACS-DOCs for spacecraft in deep-space missions. knowledge data base already collected, to summarize ISACS-DOCs for satellites in earth orbit are based on common knowledge into templates allowing the those fruitful experiences. incorporation of many diagnostic rules into systems. Figure 11 shows a process image of the knowledge data References base with the knowledge templates. The knowledge 1) Hashimoto M., Nishigori N., Mizutani M.: Operating Status of templates are comprised of an instruction manual and a Monitoring and Diagnostic Expert System for Geomagnetic Observation Satellite GEOTAIL, Proceeding of the 2nd sample knowledge data base file. At first, the International Symposium on Reducing the Cost of Spacecraft administrator of the knowledge data base defines the users Ground System and Operations, RALGS2.73, pp.73.1 – 73.8, who edit the diagnostic knowledge data, the operating July 1997 parameters of the GUI-based editor, and subsystems in 2) Hashimoto M., Choki A., Nishigori N., Mizutani M.: Operating Status of Monitoring and Diagnostic Expert System for Mars knowledge data such as Power control subsystem, Data Probe NOZOMI, Proceeding of the 4th International handling subsystem, Communication subsystem and so on. Symposium on Reducing the Cost of Spacecraft Ground System Usually the person in charge of each subsystem of a and Operations, 2001 3) Hashimoto M., Honda H., Nishigori N., Mizutani M.: satellite is the user of the knowledge data base. The Monitoring and Diagnostic Expert System for Sample-Return sample common knowledge data in each subsystem will Probe HAYABUSA, Proceeding of the 5th International be arranged into the template file, so that the user can Symposium on Reducing the Cost of Spacecraft Ground System and Operations, 2003 define and build a basic knowledge data base efficiently. 4) Takaki R, Hashimoto M., Honda H., Choki A., Mizutani M., The instruction manual will be described the features of Hirose T.: Automatic Monitoring and Diagnostic System for diagnosis also in each subsystem. Infrared Imaging Surveyor ASTRO-F, Proceeding of the In order to create standards to build and keep up the International Symposium on Space Technology and Science knowledge data base, the approach of the knowledge (ISTS), 2006 templates is the key of the next or future ISACS-DOC.

Tf_6