NEOSTEL DATA PROCESSING CHAIN Jiří Doubek(1), Agnieszka Sybilska(2), Piotr Duźniak(3), Piotr Sybilski(2), Grzegorz Lech(2) (1) Iguassu Software Systems, Evropská 120, Praha 6, 160 00, Czech Rep. Email: [email protected] (2) Sybilla Technologies, Toruńska 59/004, 85-023 Bydgoszcz, Poland Email:[email protected] (3) GMV Innovating Solutions Sp. z O. O., Ul. Hrubieszowska 2, Warsaw, 01-209 Poland Email:[email protected] ABSTRACT purpose the NEOSTEL Data Processing Chain is being developed. The NEOSTEL DPC has several The NEOSTEL (Fly-Eye) telescope will detect NEOs components which are responsible for the following through wide-field survey observations. The Data parts: Processing Chain (DPC) will simultaneously process LightEngine – pre-processing of raw data, images from all cameras, extract detections, build tracklets and check it with the existing catalogue, astrometric and photometric data reduction. identify new discoveries and provide SST functionality. ASAP – building tracklets, comparing them The processing is divided into two main components – with existing asteroid catalogue, reporting LightEngine and the Automatic Asteroid and Satellite tracklets to the MPC. Processor (ASAP). LightEngine pre-processes raw data Orbit Calculator – providing the NEO scores and provides plate solution of images in near real time and other statistics for the observed tracklets. using a variety of catalogues. Identified detections from Archiving subsystem – storing all data in a all cameras are passed to the ASAP, which combines redundant system and providing additional them into a single field and builds tracklets from several services to the other software parts. images over the same sky region. Tracklets are All components are further described in the next compared to the Minor Planet Center (MPC) asteroid sections. catalogue and reviewed by an operator. Additional tracking data can be requested through the Scheduling 2 LIGHT ENGINE and Command Message (SCM) provided through the Orbit Calculator module, making use of a ranging LightEngine (LE) pre-processes frames retrieved from method for orbit determination. the Archive, performs astrometry and photometry and provides the results to ASAP (as shown in Figure 1, 1 INTRODUCTION where the protocols used in communication between LE components, ASAP and the Archive are presented). The Within the ESA’s SSA programme, the system in Figure 1 diagram shows all connections, not a specific charge of detecting natural objects such as asteroids, data flow. which can potentially impact Earth and cause damage, is the Near-Earth Objects (NEO) segment. Data on NEOs The Archive notifies the pre-processing and are collected from telescopes and radar systems processing services about new frames via worldwide. Each of these submits observations to the RabbitMQ. Minor Planet Center (MPC), which acts as a central The pre-processing service uses REST to clearing house for asteroid and comet observations. The download and upload frames from/to measurements collected are retrieved by the NEO AstroDrive. Coordination Centre (NEOCC). The processing service uses REST to download Regular sky surveys are important part of the NEO and upload frames from/to AstroDrive and discoveries. The same part of sky is scanned several notifies ASAP about processing completion via times in order to find moving objects. For performing RabbitMQ. It also uses REST to communicate the successful surveys, a telescope with a wide field of with the catalogue service. view (FoV) shall be available as well as the effective AstroDrive has its dedicated database way how to visit the different sky regions. For this manipulated via PostgreSQL engine. purpose the Near Earth Object Survey TELescope AstroDrive forwards frame storage and (NEOSTEL) is under construction. After it becomes retrieval operations to the Archive via Minio. operational it will produce a big amount of data that are The thumbnail generator service uses REST to needed to be processed in near-real time. For this download frames from AstroDrive. LProc.eave 1st foo NEOter and em Debrispty – DetectionThe Con Conference,ference f ooDarmstadt,ter wil lGermany, be adde 22-24d to Januarythe firs 2019,t page published of eac hby ptheap ESAer. Space Safety Programme Office Ed. T. Flohrer, R. Jehn, F. Schmitz (http://neo-sst-conference.sdo.esoc.esa.int, January 2019) ASAP downloads frame solutions from (TGS) AstroDrive and thumbnails from the thumbnail generator service via REST. Catalogue The catalogue service provides the service processing service with requested (CatS) catalogue data. Generally available services developed by Sybilla Technologies for LightEngine and other services used in NEO-DPC Supervisor The supervisor monitors other services’ service (SS) health and is able to restart them in case of failure. Logging The logging service serves as a service (LS) universal sink for logs emitted from the other services. The service can send e- mail notifications to configured receivers. Configuratio The config service is responsible for n service providing other services with Figure 1 Interactions between LightEngine components (ConS) configuration data. and other services Identity All clients connecting to the services in LightEngine is divided in the following software service (IS) the intranet from outside of the intranet components: (e.g. a CLI app, a Web app) must be authenticated and authorized. The Component Description service provides authentication and Pre- The pre-processing service is authorization services for both the processing responsible for: outside and inner entities. service (PPS) Basic correction of the frames. Combining darks, biases, flats All LightEngine services with user interface will into master darks, master flats, provide the current status of the services via SignalR master biases. updates and allow for retrieval of the historical data via Applying master calibration RESTful API, requesting the data. Commands will be frames on the scientific frames. executed via RESTful API and monitored via SignalR. Multiple instances of the pre-processing Each service will provide minimal status information service can be running simultaneously. BasicData and minimal set of commands listed in Table It is expected that there will be one 1. instance running for each camera. Table 1 BasicData for each service. Property name Type Description Processing The processing service is responsible Id String (UUID) Unique ID of the service (PS) for: service. Astrometry. Name String The name of the o Feature detection. service. o Plate solving. ServiceState String (enum): High level state o Catalogue matching. unknown, description of Photometry. starting, started, the service. Multiple instance of the processing normal Initial state service can be running simultaneously. operation, unknown is just malfunction, after the creation AstroDrive AstroDrive serves as the Archiving permanent of the service, (AD) Service’s overlay for the most of the failure, starting is LE. It provides a way to obtain and stopping, usually store frames and frame solutions. stopped automatically Thumbnail The thumbnail generator provides initiated. generator ASAP with frame thumbnails based on StartTime String (date and Service start service requested parameters. time) time Leave footer empty – The Conference footer will be added to the first page of each paper. ProductionTime String (date and Information time) provided in state creation time. Decay String Time during (duration) which the information provided in status may be trusted (usually three times the typical update loop). IsOperational Boolean Simple Boolean flag showing if the service is ready to accept new request, fulfil its role in the system. Figure 2. Combined image from 16 NEOSTEL cameras. 3 ASAP Several combined images from a same tessellation field The Asteroid and Satellite Automatic Processor (ASAP) are part of a sequence. Such sequence is used for is software that was developed for processing space building potential asteroid tracklets. debris and asteroids images from optical sensors in the Tracklets are formed from loners. Loners are detections context of ESA Test-Bed Telescopes. It is being that are present on one combined image but not on the redesigned for the NEOSTEL DPC to work effectively others. It shall be noted that loners can be points or for the large FoV and provide results in near-real time. streaks. Point loners shall change its position from one ASAP contains several services which are responsible image to another. The set of all images has to be for specific tasks. The first is the Detection Collector compared to find the point loners. Streak loners have a Service (DCS) which collects astrometric data coming noticeable speed and position angle with respect to the from LightEngine and stores them in a local database telescope tracking. for further processing. After all images from the same Tracklets represent moving objects – potential asteroid survey field are ready, the DCS inform the Loners and or satellites. They are composed of the combination of Tracklets Service (LTS). It combines data from all loners from different images. A loner from the first images, builds tracklets and store them in the database. image is combined with a loner from a second, third, Then the MPC Service (MPC) compares the found etc. Such a combination is selected on specific criteria. tracklets with the asteroid catalogue and identifies First tracklet position can be any loner from the first