62nd International Astronautical Congress, Cape Town, SA. 2011

IAC-11-B3.4 – B6.6.4

THE COLUMBUS GROUND SEGMENT – A PRECURSOR FOR FUTURE MANNED MISSIONS

Thomas Mueller Deutsches Zentrum f. Luft- u. Raumfahrt, GSOC/Col-CC, Germany, [email protected]

The European decentralized operations concept enables all participating countries to establish a transnational centre of competence that actively cooperates in European participation to the International Space Station (ISS). To this end eighteen European facilities and three International Partner centres are interconnected by a Ground Segment providing data, voice and video services via the central hub at the Columbus-Control Centre.

Operating this Ground Segment is a significant challenge for the Ground Operations Team at Col-CC, not only due to the vast number of facilities and the related world-wide distribution, but also because of the number of different users (Columbus and ATV flight control, payload facilities, engineering support, PR) with their specific operational needs and constraints. In contrast to previous short duration missions with sequential mission phases, the continuous ISS operations support requires consideration to the current increment execution in parallel with preparation and training of following increment(s) and post increment evaluation. The long lifetime duration of 12+x years requires continuous maintenance and sustaining engineering of the ground segment infrastructure with focus on the life-span of individual components as well as life-cycles of entire technologies. Replacement of equipment or systems must be performed with minimal impact on real-time operations, and in coordination with increment execution/preparation activities. An important component of this structure is the application of human resources. An experienced team of qualified operators and engineers is to be trained to maintain a level of proficiency that is applied over this long period.

The above is reproduced in a system architecture is required supporting parallel real-time operations, simulation (preparation) and test (sustaining engineering) activities with varying instances of system configurations. For some systems however, like voice and video, different instances are not feasible, therefore rigid configurations are applied. The organisational approach of the Col-CC was established by running three main control rooms with individual ground operations performed in dedicated rooms. This concept required the ground operations team to be organised in a new approach. New processes and a suite of tools for anomaly resolution and configuration control were developed with everything is tied together by a powerful ground operations and resource planning.

This Columbus Ground Segment required a new approach, which shall be discussed here. Its distribution capability and requirement implementation is unique in the human space exploitation activities but can indeed be seen and applied as a precursor for future manned space missions, which require multi-national collaboration.

I. INTRODUCTION The world-wide distribution of a ground segment is not any longer limited to a network of ground stations In the beginning the space programs were self with the aim to provide a good coverage of the space standing national activities, often in competition to other craft. The coverage is sometimes – like for the ISS- nations. Today space flight becomes more and more an anyway ensured by using a relay satellite system international task. Complex space mission and deep instead. In addition to the enhanced down- and uplink space explorations are not longer to be stemmed by one methods a ground segment is aimed to connect the agency or nation alone but are joint activities of several different centres of competence of all participating nations. The best example for such a joint (ad-) venture agencies/nations. at the moment is the International Space Station ISS. From the space craft operations point of view such Such international activities define complete new transnational ground segments are required to support requirements for the supporting ground segments. distributed and shared operations in a predefined decision/commanding hierarchy. This has to be taken

IAC-11- B3.4 – B6.6.4 Page 1 of 11 Copyright © 2011 by Mr. Thomas Mueller. Published by the IAF, with permission and released to the IAF to publish in all forms. 62nd International Astronautical Congress, Cape Town, SA. 2011 into account in the technical topology as well as for the o Toulouse, France operational set-up and teaming. o Noordwijk, Netherlands o Neaples, Italy Last not least increases the duration of missions, o Cologne, Germany which requires a certain flexibility of the ground o Zuerich, Switzerland segment and long-term maintenance strategies for the o Odense, Denmark ground segment with a special emphasis on non- o Trondheim, Norway intrusive replacements. The Russian space station MIR o Madrid, Spain has been in the orbit for about 15 years, the ISS is o Bruxelles, Belgium currently targeted for 2020, to be for over 20 years in  Operations Management Team (OMT) in space. Noordwijk, Netherlands  European Astronaut Training Centre (EAC) in Cologne, Germany  Artemis Ground Station for ATV support in II. THE ESA GROUND SEGMENT – A UNIQUE Redu, Belgium NETWORK The European facilities are connected to and interact with the International Partner Control Centres:

 The ISS main Control Centre (MCC-H) in Houston, USA  The Payload Operations Centre (HOSC/POIC) in Huntsville, USA  The Russian Control Centre (MCC-M) in Moscow, Russia

Interconnection All these facilities comprise the ESA Ground Segment. They are interconnected by a terrestrial based Wide Area Network (WAN), the so-called Fig. 1: The ESA Ground Segment Interconnection Ground-Subnetwork (IGS). This strange name has some historical reasons. ESA’s contributions to the ISS are the Columbus module and the ATV support vehicles. Both missions The network is a private network running over a are operated and supported by using one ESA ground public WAN provided by a commercial service segment. provider. It is somehow remarkable that due to security reasons it is a pure terrestrial network. Distribution In order to respect the special ESA set-up allowing As shown in Figure 1 the IGS has a star-like each member state to actively participate in a mission, a topology with Col-CC as the central hub. The Col-CC decentralized operations concept was implemented with acts as central service provider to all European facilities transnational active centres of competence in each and as the hub to/for the International Partner sites. contributing country: Services provided by Col-CC are:  The main Columbus Control Centre (Col-CC)  Data in Oberpfaffenhofen, Germany Telemetrie is downlinked from the ISS to  The ATV Control Centre (ATV-CC) in MCC-H, HOSC and MCC-M, routed to the Toulouse, France Col-CC and from there distributed to the  Three Engineering Support Centres (ESC) for European facilities either raw or processed, technical support for the Columbus module and depending on the nature of the data the ATV space crafts  Commanding for Columbus o Bremen, Germany User Control Centres command their o Turin, Italy payloads via a central command system at o Les Mureaux, France Col-CC which routes the commands to the  9 User Operations Control Centres (USOC) for command system at MCC-H for uplink to Payload Operations the station

IAC-11- B3.4 – B6.6.4 Page 2 of 11 Copyright © 2011 by Mr. Thomas Mueller. Published by the IAF, with permission and released to the IAF to publish in all forms. 62nd International Astronautical Congress, Cape Town, SA. 2011

 Voice On top of real-time operations Training and A central voice system at Col-CC is Simulation activities, Public Relations and service connected to the voice systems of MCC-H, provision to International Partners have to be supported. HOSC and MCC-M. Remote keysets are provided to all European sites directly One consequence of this multi usage profile was that connected to the voice matrix at Col-CC. A the ground segment operations and engineering was self standing voice system at ATV-CC is decoupled from the flight operations. Therefore Col-CC connected to Col-CC system. provides two separate functions, Columbus operations  Video and ESA ground segment operations. The ESA ground A central video matrix at Col-CC provides segment operations running from a dedicated control access to different video sources like on- room within the Col-CC. This is a significant difference board video from MCC-H and MCC-M. to other centres like MCC-H where the Ground  Ops Support Tools Controller (GC) is part of the flight control team and is A suite of smaller, web-based applications sitting in the main control room. for planning, anomaly handling, logs, information exchange, etc is available at This separation of the ground segment requires Col-CC. NASA tools are replicated at Col- ground segment requires a separate and decoupled CC to be accessible from European users resource planning function. via the IGS.  Connectivity Point-to-point connections are configured between some sites for discrete III. CONTINOUS OPERATIONS applications. Parallelism of Activities Multiple User Profile A standard space mission (e.g. satellite positioning) It is remarkable that this ground segment supports or even the previous manned short duration missions two totally different missions: often have more sequential mission phases.

 Columbus, the European laboratory and Design -> Implementation -> Training and European payload execution on-board the Validation -> Execution (launch) -> and post activities. ISS in general and  ATV, a support and transport vehicle flying The continuous operations of the ISS lead to a to and docking at the station. parallelism of these activities for different mission subsets named increments. During the running Although two different missions operated from two increment is executed, the last increment is post- different control centres, it made perfectly sense to use a evaluated and the next increment with future payloads is common infrastructure as the majority of the com- already in preparation. Consequently the ground munication needs are the same. Video, Voice, Ops segment has to support different activities in parallel Support Tools (at least the NASA and Ground Segment often with the need that the ground segment is adapted tools) are commonly used services. The fact that ATV- or re-configured for new payload requirements. This CC has a separate voice system is simply related to the leads to execution of simulations and validations chosen system and to ease configuration as both systems activities in parallel to on-going operations with two are separated. The Telemetry and Telecommand system different configurations of the systems. of ATV is of course separated as well, whereby station data and ATV data while ATV is docked to the station To support this, the main ground systems at Col-CC is available via the common data routing system. are available three times in different instances:

Even the payload operations differ depending where  Real-Time the payload is located. Payloads located in the Russian for Operations or US segments are using MCC-M or NASA command  Simulation systems and by-passing the Col-CC systems, payloads for Preparation and Training located in Columbus are operated via a commanding  Test interface connected to the command system of Col-CC. for Sustaining Engineering and Validation

IAC-11- B3.4 – B6.6.4 Page 3 of 11 Copyright © 2011 by Mr. Thomas Mueller. Published by the IAF, with permission and released to the IAF to publish in all forms. 62nd International Astronautical Congress, Cape Town, SA. 2011

All three instances are totally decoupled and can be - the main Ground operations Control Room configured separately to support: (GOCR) for real-time operations, - the back up control room (BUGOR) in the back-  Different configurations for operations and up centre. preparation  Testing & Validation of new The back-up room is used for supporting simulations configurations, software/hardware releases and validation activities by an additional separate and ops products Ground Control Team in parallel to real-time  Different data basis (rt data vs. sim data) operations.  Training with Simulator(s) and failure injections The ground segment is capable to support two different activities in parallel without any interference For some systems/services different instances are not feasible. The video system can anyway route Real-time Execution and Simulation or Test different video channels, the voice system is also only one system due to the singularity of the interfaces to the The ATV mission can be supported in parallel to on- international partners and the keyset distribution in going Columbus operations. Whereby only one Europe. A separation is done via dedicated voice loops simulation can be supported at one time (either for for real-time and simulations. Columbus or for ATV) and during critical mission phases of ATV only real-time operations (Columbus Facilities have to be doubled as well. The Col-CC and ATV) can be supported. This is due to the has three main control rooms: complexity of the ATV mission. - One for real-time operations - One for simulation and preparation Nearly permanent coverage - A back-up control room in a different building. The ISS uses a relay satellite system for communication with the ground. Consequently the ISS All three rooms are connected to different has a nearly permanent coverage. There is an average of monitoring and control system (3 instances), which can less than 10 times per day where a total Loss of Signal be connected to the different data sources (real-time (LOS) period is longer than 10 minutes. LOS periods data or simulators). During routine phases, where no greater than 30 minutes are very rare. back-up room in hot-stand-by is required, both, the back-up and the non-active control room, are used for Although very good and comfortable for experiment preparation activities. operations, this coverage is problematic for maintaining the ground systems as these activities should have less The necessity of this approach can be illustrated or no impact on on-going operations. To deal with this with the usage profile of all three control rooms for problem the ground operations/planning differentiates 2010. between two categories of maintenance:

Usage Description Planned Maintenance K4/3/11 365,0 d Usage for real-time operations These are activities someone would normally understand as ‘maintenance activities’. Regular work to K4 +259,0 d Simulation/Test/Maintenance keep the systems alive. Activities, which can be planned K3 +201,1 d (hopefully) and prepared way in advance. K11 + 77,3 d The Col-CC engineering team monitors the ground Table 2: Usage profile of all three main control rooms systems permanently and is in close contact with the during 2010 vendors. Whenever a maintenance activity is needed, an

input to the ground operations planning function The hours of usage above are summed up to days (described in the following sections) is done. The and the room preparation is not taken into account. It planner is looking for a feasible timeslot based on the was even surprising for us to find out that on 172,3 days following input: all 3 control rooms were used in parallel!

- urgency/need date For the ground segment operations two Ground - services affected Operations Control rooms are available: - external interfaces affected

IAC-11- B3.4 – B6.6.4 Page 4 of 11 Copyright © 2011 by Mr. Thomas Mueller. Published by the IAF, with permission and released to the IAF to publish in all forms. 62nd International Astronautical Congress, Cape Town, SA. 2011

- resources required (engineering team/operations Unplanned Maintenance teams/vendor) This is basically a more diplomatic expression for - impact on operations troubleshooting. Whenever a system has a failure or - possible impact on other systems shows a significant degradation of performance, which - maximum duration of outage/downtime requires an immediate action it is the task of the ground - possible workaround control team on console to coordinate the repair with all affected and involved parties. For activities with significant impact on services and operations, the ground operations planner will Taking into account the high service availability coordinate with the planners for on-board activities and needs for manned space operations and ATV, the core in parallel the ground controller team will coordinate network and the essential ground systems at Col-CC are with the flight control team(s) and external partners via redundant. For most cases the fail-over is automatic, for flight notes describing the activity and possible impacts. other cases the ground controller at Col-CC has It has been shown very useful in the past to include procedures and tools to manually fail-over or to restore major ground activities in the on-board timeline for services and to keep service interruptions to a minimum. awareness. However failures have to be localized and analyzed, Often such maintenance activities have to be done which is in such a complex ground segment often not an around LOS slots and sometimes an activity has to be easy task. Especially in an end-to-end service involving broken up in a sequence of smaller parts fitting in a systems of different entities (e.g. ESA and NASA), it sequence of some LOS slots. Performing such requires sometimes troubleshooting on both sides to maintenance in sequence requires a precise planning localize where a fault resides. and discipline of every involved person as it has to be taken into account to stop the maintenance work in-time Any broken equipment has to be exchanged or to be able to restore the service at the end of each LOS. repaired in a short time frame to restore redundancy again. Software patches may have to be implemented Another option, to perform maintenance during crew and installed. sleep, is not possible as the Col-CC engineering team is staffed during office hours only – everything else would In these cases the planning and coordination not be affordable. processes are the same as described above, but executed in near real-time by the on-console personnel. Whereby This planning is a very complex and iterative the configuration management and quality assurance process requiring a good coordination of all involved processes are as important as for planned changes at cadres and a sufficient lead-time. The most complicated least. Ad-hoc ‘emergency’ processes and decision task here is to reflect the needs of all current missions boards as well as a strong anomaly reporting system are and the possible impact of preparation campaigns of essential for this. future missions. It should be noted that the redundancy of the Nevertheless real-time operations, especially with systems and network can sometimes be more a burden astronauts are a very flexible task by nature. Therefore, then a help: after all the planning and preparation, the last and final word has still the acting flight operations team on-  Often the automatic failover mechanism is console. Every activity is started after the ‘go’ of the a single source of failure itself. Outages of flight director on-console and is performed in close a complete service can be caused by a coordination with him/her. faulty or unsuccessful switch-over to the redundant system. Or the failover Another measurement to minimize the impact on mechanism does not initiate a failover real-time operations is of course a strict configuration because of a wrong status interpretation or management process. If possible, every change or a non detection of a failure occurs. upgrade of a system is carefully tested and formally validated using the test facilities/instances described  After a failure and fail-over it has to be above. A successful validation including the required ensured that the system does not swap to an maintenance procedures is a prerequisite for starting the undefined state during the repair or at re- planning process of a change in the real-time system. activation.

IAC-11- B3.4 – B6.6.4 Page 5 of 11 Copyright © 2011 by Mr. Thomas Mueller. Published by the IAF, with permission and released to the IAF to publish in all forms. 62nd International Astronautical Congress, Cape Town, SA. 2011

 Switch back after a failure is sometimes For most cases it is more reasonable to perform more intrusive as the actual failure was. hardware maintenance event driven. Single components of high available equipment like fans, power supplies, On the other hand, if it is possible to decouple the hard drives, etc. are by default or have to be redundant redundant path, troubleshooting can be done on the implemented. Therefore it is more cost and resource passive path without interfering the operations on the effective to replace such components after a failure has active one. This strategy is sometimes also used for occurred than preventive after pre-defined operations planned upgrades and changes on systems either when it time. In general it should be considered that any is not possible to use a test facility to validate the preventive exchange does not necessarily guarantees the changes beforehand or during the change to avoid a availability of a component, therefore it is more service interruption. important to implement effective redundancies.

A continuous and regular (!) in deep monitoring is required to detect failures of components or even IV. LONG LIFETIME degradation in-time - to the most possible extend in At the moment the ISS life-time is targeted to until real-time. The Col-CC has implemented a hierarchical at least 2020. The implementation of the ESA ground monitoring system for real-time monitoring: so-called segment started in 2001 and first part was gone Element Managers monitor dedicated systems and operational already in 2004. Fully in operations is the provide summary monitoring information to an ground segment since 2006. This means that the umbrella monitoring system operated by the ground systems have to be kept alive for roughly 20 years. controllers on console. In addition to the on-console monitoring, engineers check and analyze off-line during This long life time has to be reflected in the design their office hours log files and perform routine system of the ground segment as well as in the maintenance. checks on a daily base.

Flexible Integration of New Experiments The difficulties of an effective system monitoring The ground segment must be designed modular to be are: capable to integrate new experiments, new user  Filtering/monitoring of meaningful data facilities, and new services. Among the mass of logging information and parameter a system provides, it is For this a good and early preparation of new essential to monitor the important data. experiments is required. Sometimes it would be even This is sometimes not easy to asses before optimal that the ground segment is involved as early as a failure occurs. possible in the development phase of an experiment to support the definition phase for communications  Polling rate of monitoring data requirements etc. But most of the time this is not the There is a trade of between the traffic the case. Therefore the ground segment engineering must be frequency of a monitoring parameter very flexible and sometimes just creative. request generates on a system and the delay of the detection of a malfunction. A Long-time Maintenance high polling rate can generate a higher load The main objective is of course to keep the services on a network than the actual business data. of the ground segment up and running over the entire It has also to be considered, which data life time with as less as possible service outages and rate a human operator can handle. interruptions. This is can be divided into two main  Correlation of monitoring data tasks: Which combination of monitoring values indicate which degradation. Preventive maintenance It is obvious, if you want to keep a system alive, you Such a monitoring is not set-up in the beginning and have to do maintenance on a regular basis. But what left untouched after. It is an iterative process and a daily does that mean for an IT-System? There is no regular task of the engineering team to refine the monitoring of change of oil or an exchange of a drive belt necessary the systems over the entire lifetime. for software after n hours of operations. There are very rare items for such regular mechanical maintenance like Again an effective and powerful anomaly tracking batteries, etc.. system (and process) is required to support the maintenance in the following aspects:

IAC-11- B3.4 – B6.6.4 Page 6 of 11 Copyright © 2011 by Mr. Thomas Mueller. Published by the IAF, with permission and released to the IAF to publish in all forms. 62nd International Astronautical Congress, Cape Town, SA. 2011

 Workflow of maintenance activities  To search for new products and either  Documentation and control of changes support a smooth transition part by part  Quality Assurance and Configuration when failure occur or Management  To replace each component at once  Knowledge base for later re-occurrence of failures In the later case a transition plan has to be defined  Correlation of malfunctions to detect with the scope of minimizing impact on operations. This systematic failures, etc. leads then to a more complex planned maintenance  Inventory control/spare part status activity to be handled as described above.

Spare part handling Software is often considered as ‘freezeable’ meaning To guarantee a fast replacement of failed equipment with a certain version of a software, the development (or a stock of spare parts is required, especially for critical procurement) of the software can be stopped. The services or components. system stays on this software and is not updated any longer. This is only possible in very rare cases if at all. Col-CC hosts a local spares stock. In addition essential spare parts are stored at external facilities and  Software is running on an operating system, international partner sites to avoid down times due to which, especially if it is a commercial shipment of equipment from Col-CC to a facility. operating system, is maintained as well and Where appropriate and more cost efficient the provision has a certain life time. It is very likely that of spares is part of the maintenance contract with it is required to update software to be dedicated vendors. compatible with a new operating system or patch. Col-CC tried to avoid this by using Inventory and spare part management is mandatory. mainly Linux systems, but… It has to be carefully assessed which spare parts are  Computer hardware has to be supported by required and distributed to which facilities. Spare parts the operating system (drivers, processors, have to be re-ordered in time after usage. In order to patches, etc.). If the computer hardware has guarantee a fast replacement the spare parts have to be to be replaced due to obsolescence, this either preconfigured or configurations have to be might require an update of the operating prepared and stored. system and in consequence an update of applications as well. So happened at Col- It should also not be forgotten that spare parts have CC e.g. with the exchange of console to be maintained as well! A bad example happened once computers. when the replacement of a CPU card in one of the Col- CC systems led to a total system outage as the spare  Applications interfacing with other card had a bad battery on-board which caused after a applications and/or services, which are failover and reboot a loss of the system configuration. subject to be changed. Either these Due to incomplete system documentation by the vendor applications are third party software with a the battery maintenance had been overseen. regular patch policy, applications are modified to implement new project Obsolescence management requirements, or applications are handled Especially for a long life time an obsolescence by partner organizations, … management in different aspects is essential. Last not least it should not be forgotten that the IT Every commercial vendor has to be innovative to world is changing very fast and sometimes even entire stay in business. Consequently at least hardware has technologies change or becoming obsolete. only a limited time of being produced and more important of being maintained. End of life and end of Two years ago the WAN of the ESA ground support times of equipments have to be carefully segment was migrated from ATM technology to MPLS monitored and taken into account as early as possible. because ATM is not longer supported and offered by Decisions have to be made: commercial service providers. Back-up dial in lines are still ISDN although ISDN is subject to become obsolete  To procure enough spare parts (in-time) to in the near future. survive a certain time after end of support – with this long life time this is not yet an The video system has to be migrated from analogue option. to HD digital as analogue equipment is barely available,

IAC-11- B3.4 – B6.6.4 Page 7 of 11 Copyright © 2011 by Mr. Thomas Mueller. Published by the IAF, with permission and released to the IAF to publish in all forms. 62nd International Astronautical Congress, Cape Town, SA. 2011 more expensive and – this is another driver – not network) at Col-CC was replaced by supported by the International Partners any longer. another system as major components reached end of support. The difficult part A last reason can be that a vendor of a system leaves was the migration of the redundant the market for a system for economical reasons (also operational systems and that the archive happened to Col-CC). In this case it might be necessary migration went hand-in-hand with a change to exchange components or an entire system with to a new operating system. another product of a different vendor.  LINUX The exchange of the console computers and Sustaining Engineering servers required an update of the operating The fast and innovative IT world gives also room for system together with the portation of nearly locking into new technologies over such a long life all software products including the main time. Apart from obsolescence there are other reasons monitoring and control software to the new for migrating systems: operating system.  VoCS  Better performance As the vendor left the market segment, the  Lower operating costs maintenance of the Voice Conferencing  Cheaper maintenance costs System at Col-CC became difficult and  Easier maintenance with less resources for expensive, especially taking into account engineering and/or operations, which leads the end of life of some components and again to cheaper operating costs upcoming changes of interfaces to  New user requirements international partners. The difficulty of the migration was that this system as the It is one task of the engineering team at Col-CC to central system in the ESA ground Segment evaluate new technologies and search for new, easier had ‘single’ interfaces to the international and cheaper solutions. partners not supporting hybrid connections but have to be migrated at once. The long life time forces the entire project to be very cost effective and to continuously watch for cost reduction options. It has to be permanently evaluated if the user community really requires the provided V. GROUND OPERATIONS functionality or if something can be made easier and cheaper. This has the high risk that user needs are In the terminology of ESA’s operations teams within overcome by budgetary constraints. the ISS project, the Ground Control Team (GCT) is the team operating the ESA ground segment from Col-CC. Already Performed Migrations In the opposite to the GCT, Flight Control Teams (FCT) Although a total operations time from about 7 years, for Columbus at Col-CC, and for ATV in ATV-CC the ESA ground segment has already undergone some control the spacecrafts Columbus and ATV. major system migrations or renewals: Organization  WAN The complex ground segment and the need to As already described above the entire support the different missions and activities via one WAN was changed from ATM technology ground segment led to the separation of the Col-CC to MPLS because ATM will not longer be Ground Control Team from the FCT. The GCT was supported by commercial service providers. organized together with the ground segment engineering This required a temporary change of the team at Col-CC in one department as shown below. central communication node at Col-CC to support both technologies in parallel and the step-by-step migration of the different facilities/IGS end points to MPLS. The most intrusive part was of course the migration of the International Partner gateways as data source for the entire ground segment.  SAN The central data archive (storage area

IAC-11- B3.4 – B6.6.4 Page 8 of 11 Copyright © 2011 by Mr. Thomas Mueller. Published by the IAF, with permission and released to the IAF to publish in all forms. 62nd International Astronautical Congress, Cape Town, SA. 2011

Ground Operations Team System Area

Ground Operations In difference to NASA’s approach the GCT at Col- Manager (GOM) CC is not part of the FCT and is not located in the main control room but has its own control room. This is done because the GCT has to deal with more users than the

Ground Controller Facility Engineering Columbus FCT and therefore it would more disturb to (GSOC GC) Maintenance Team Team have the GCT co-located than help.

Systems Controller ViDS Controller Ground Operations The Ground Operations Control Room (GOCR) is (GSOC SYSCON) (GSOC Video) Planner located in the system area where the main system/server rooms and the main back rooms for system engineering Ground Control Team are located. This supports a close coordination between engineering and ground control during maintenance Fig. 2: Ground Operations Team Organization activities. In addition the system area is a high restricted area in both aspects physical and IT access. The location The Col-CC Ground Operations Team is comprised of the control room in this area allows the direct of the Ground Control Team and the offline Engineering network access to the systems by the GCT on console. and Maintenance Teams. This is another difference to e.g. NASA’s approach: the GCT at Col-CC has a much more active role in Ground Control Team operating the ground systems. He/she has to actively The Ground Control Team has two console positions configure the systems and does level 1 troubleshooting  The Ground Controller (GC) by his own and therefore is equipped with dedicated GCT on-console lead, coordination with all tools and direct access to the ground systems. users and international counterparts and responsible for external services In the beginning, when we set-up this approach of  The System Controller (Syscon) GCT separation it was often said that it is essential that Technical expert for all Col-CC systems the GC is located in the main control room to ‘feel the and internal services. Cross-trained with atmosphere of the current real-time situation’ and so system engineering team to support off-line being able to react faster to the needs of the FCT. It can engineering. be said that the first years of operations clearly showed that it was more important to localize the GCT close to  Video Controller the engineering world as it is more difficult to Staffed by the video engineering team; only coordinate the system operations with the engineers on console for dedicated activities than to pass on the system status to the FCT and coordinate activities with the FCT via voice loops. Apart from on-console GC and Syscon, both Especially because the communication via voice loops positions have the off-line tasks to coordinate and is anyway the must between operator consoles. support increment preparation as interface between the users and the system engineering team. Ground Segment Planning One important part of the ground segment operations System Engineering Team is the ground segment operations and resource planning. The system engineering team is comprised by engineers/experts for each dedicated system at Col-CC. At Col-CC this planning is a completely self The engineers work pure off-line but have dedicated standing function, separate but of course with dedicated consoles or working places for accessing their systems interfaces to on-board, mission, and simulation planning in the system area. functions.

The close organization and coordination of both The Ground Operations Planner (GOP) is functions; ground operations and engineering led to a responsible for planning the resources and the support very effective and fast and well organized ground of the ground segment to real-time operations, segment handling. simulation and training activities, validation and test activities and ground segment maintenance. For migration projects as well as for increment preparation he/she plays an active role in coordinating the different parties and requirements.

IAC-11- B3.4 – B6.6.4 Page 9 of 11 Copyright © 2011 by Mr. Thomas Mueller. Published by the IAF, with permission and released to the IAF to publish in all forms. 62nd International Astronautical Congress, Cape Town, SA. 2011

A dedicated planning tool and process was This is a significant difference to former mission. The developed for the ground segment planning. The output GC takes various user requests and has to coordinate is a detailed timeline, which acts as the main working with all involved parties as well as he/she actively guideline for the GCT on console. transport ground segment status and information to users. The GOP concentrates on the mid-term and short- term planning. The short-term planning is detailed and During the first months of Columbus operations it needs a fine granularity to allow to derive the above was found that the GC needs to play an active role even mentioned ground operations timeline. in on-board troubleshooting of experiments. The user is usually an expert for his/her experiment, but has only In the past the long-term planning was done by each rare knowledge of on-board and on-ground dataflow. planning function more or less separately and was The dataflow after leaving Columbus and entering the coordinated with other planning functions rather late. US assets was -to some extend- terra incognita for the This sometimes led to situations, where resource Columbus module experts of the FCT as well. AS the conflicts could not be solved because of a late detection. Col-CC GC is responsible for the interfaces to NASA, It was experienced that due to the involvement of he/she has implicit a quite detailed knowledge about the International Partners, external support, or dependencies NASA ground system and the dataflow from the station on mission preparation milestones activities often have to the European user. It should be also noted that to be planned rather early and therefore a conflict different type of data are running over different paths detection and resolution must start in the long-term and on-board and on ground. Therefore the GC became a mid-term planning already. To deal with this situation a key position to support payload troubleshooting end-to- integrated planning working group chaired by all end from the experiment to the user console. planning functions was established to concentrate on the long-term planning with the main aim of early conflict This support function was extended further and the resolution. GC became a permanent member of a tiger team for supporting payload anomalies. Since then a part of the Maintenance Structure GC training and certification program is the With the organization described above, the Col-CC participation in the Columbus User Level and Columbus Ground Operations provides the following maintenance Payload Training courses in the European Astronaut structure: Training Centre in Cologne.

Team Avail. Description Deep System knowledge Level 1 GCT 24/7 User Helpdesk For the level 1 maintenance support as described Service Restoration / above, the GC and Syscon require deep system back-up activation knowledge. Although both are provided with a Level 2 Sys Eng 8/5 Preventive maintenance sufficient amount of very detailed troubleshooting and Trouble shooting recovery procedures, a complex and highly integrated Level 3 Vendor Vendor support acc. ground segment requires an understanding and a careful Maintenance contracts analysis of failures to decide which recovery procedure Table 1: Maintenance Structure to apply. Often it is the case that a failure of one system is visible by strange effects on other systems first. Operator Profile Starting with the usual operator profile for a satellite This necessary system knowledge is ensured by a mission performed by the control centre, we learnt that cross-training of the GCT in ground systems the ground operator for this complex ground segment engineering and a close coordination between the must provide additional skills. System Engineering Team and the Ground Control Team. Individual members of the GCT are nominated User coordination for dedicated systems and work very close together with A manned space mission has per default a need for the accordant engineers to jointly generate operations intensive verbal coordination (voice communication) on procedures and to support validation of system changes. various voice loops. In this project the large user This working relation ship proved to be very effective. community and the different mission objectives expand One limitation is that the GCT personnel have shift the need of verbal coordination on the loops even more, work on console and the majority of the co-operation especially for the Ground Controller, as he is the point tasks have to be worked during the rare office days. On of contact for service requests, ground anomaly the other hand the shift work requires alternative ways reporting and status, maintenance coordination, etc. to pass information of changes on to the entire team.

IAC-11- B3.4 – B6.6.4 Page 10 of 11 Copyright © 2011 by Mr. Thomas Mueller. Published by the IAF, with permission and released to the IAF to publish in all forms. 62nd International Astronautical Congress, Cape Town, SA. 2011

The Col-CC GCT uses some web-based information flow tools, a controlled update mechanism of operations Another alternative is a frequent position rotation. procedures and a wiki-style knowledge base for the This is not envisaged by Col-CC as we see the negative information propagation. effects at other centre where this is done. The persons stay in the project but the knowledge is lost to some Profile extend due to the amount of job rotations in parallel. The operators at Col-CC have usually an Especially operations teams have to re-build up engineering degree or equivalent. Each individual has to constantly. Furthermore it is still true that an undergo a formalized training and certification experienced operator get his experience by doing his programme, which lasts for the GCT approximately 6 (one) work for some years. months.

It is difficult to keep constant shift work attractive for such high profiled personnel over a longer period of VI. SUMMARY time. Therefore the individuals are assigned to different mini-projects like migrations, sustaining engineering The ESA Ground Segment for Columbus and ATV tasks, ATV support, etc. to enable them to do work off- in its distribution capability and requirement console and get further experience in other aspects of implementation led to quite some new approaches for the programme. the ground operations.

Nevertheless is has to be dealt with changes of The long life time and the continuous operations put personnel and a recognizable attrition rate. A positive additional constraints on operations and especially on and friendly working relation ship is essential to keep the maintenance of all systems. the team motivation. Several initiatives are done to keep the attrition rate low. It would be beneficial to be able to It can indeed be seen and applied as a whole or at offer alternative positions within the same project. This least in certain aspects as a precursor for future manned would keep the knowledge to some extend, which is the space missions, which require multi-national main problem. But although such an international collaboration. project provides always some vacant positions, the amount of free positions in total is of course very low.

IAC-11- B3.4 – B6.6.4 Page 11 of 11 Copyright © 2011 by Mr. Thomas Mueller. Published by the IAF, with permission and released to the IAF to publish in all forms.