RAWDAR: Raw Data Repository May 14th, 2018 RAWDAR Objectives & Status RAWDAR has the following, related but independent objectives: ✓ Storage of the missions’ historic raw data in a central repository. ✓ Systematic storage of the missions’ future raw data in this central repository. RAWDAR Data stored by Mission Category: • RSVTSC-0: Concluded Missions - Data not available • RSVTSC-1: Concluded Missions - Data fully fetched • RSVTSC-2: On-going Missions - Data stored externally • RSVTSC-3: On-going Missions - Manual retrieval GEDAIS • RSVTSC-4: On-going Missions - Automated retrieval • RSVTSC-5: Concluded Missions - Data stored externally as presented in the following diagram. RAWDAR Missions & Categories Data not available Data stored externally • GIOTTO RSVTSC-0 RSVTSC-2 • HUYGENS • BEPICOLOMBO • ULYSSES • GAIA • HUBBLE Data stored externally RSVTSC-5 • ISO Manual retrieval • LISA PATHFINDER RSVTSC-3 • MEX Data fully fetched going missions going - • EXOSAT RSVTSC-1 • HERSCHEL On Concluded missions Concluded Automated retrieval • HIPPARCHOS • IUE • CLUSTER RSVTSC-4 • PLANCK • ROSETTA • INTEGRAL • SMART-1 • SOHO • VEX • XMM-NEWTON RAWDAR Missions – High-Level View RAWDAR Missions’ Sizes – Pies & Charts Graphs RAWDAR Directory Structure The /data mount point contains the RADACER directory which is the repository root directory. For each mission there is a directory that contains all the available raw data of the mission. RAWDAR RSVTSC-0 Missions Data not available • GIOTTO RSVTSC-0 • HUYGENS • ULYSSES Concluded missions Concluded RAWDAR GIOTTO • Category: – RSVTSC-0: Concluded mission – Data not available – Old mission, whose raw data are not available at ESAC nor is there a possibility of finding it. • General Mission Information: – http://sci.esa.int/giotto/ • Data: – Spacecraft data (DID, RSVTSCEPA, GRE,-1 HMC, IMS, JPA, MAG, NMS, OPE, PIA, RPA) in PDS format – https://www.cosmos.esa.int/web/psa/giotto# RAWDAR HUYGENS • Category: – RSVTSC-0: Concluded mission – Data not available – Data (data products only) and relevant information available exclusively through the URL below. • General Mission Information: – http://sci.esa.int/cassini-huygens/ – https://www.cosmos.esa.int/web/psa/huygens# RSVTSC-1 RAWDAR ULYSSES • Category: – RSVTSC-0: Concluded mission – Data not available – Very old mission, whose raw data are not available at ESAC nor is there a possibility of finding it. • General Mission Information: – http://sci.esa.int/ulysses/ • Data: – http://ufa.esac.esa.int/ufa/RSVTSC-1 RAWDAR RSVTSC-5 Missions Data stored externally RSVTSC-5 • ISO • LISA PATHFINDER Concluded missions Concluded RAWDAR ISO • Category: – RSVTSC-5: Concluded mission – Data stored externally – Old mission for which all the raw data is already available from the ISO Data Archive (IDA) at the URL below. • General Mission Information: – http://sci.esa.int/iso/ • Data: – https://www.cosmos.esa.int/web/iso/accessRSVTSC-1 -the-archive – ISO DATA PRODUCTS (https://www.cosmos.esa.int/web/iso/guide-to-iso- data-products) • Raw Data Products: These are essentially unpacked telemetry in which no data reduction has taken place. • Basic Science Data Products: Data have been processed further to an intermediate level. • Fully Auto-processed Science Data Products: These data include a set of coherent, instrument-independent measurements of images or spectra designed to get as close as possible by automatic means to what could be produced by an astronomer using an interactive analysis system. RAWDAR Lisa Pathfinder • Category: – RSVTSC-5: Concluded mission – Data stored externally – Mission in post-operations phase. • General Mission Information: – https://www.cosmos.esa.int/web/lisa-pathfinder • Data: – Science Archive: http://archives.esac.esa.int/lpfsa – Products RSVTSC-1 • Raw Data Products: unpacked telemetry. They are to be ingested in the archive under the DDS placeholder soon. • Basic Science Data Products: processed data. In the current version of LPFSA Analysis Objects and Service Documents fall under this category. Eventually Analysis Packages will be ingested as well. • Fully Auto-processed Science Data Products: in a future release some products of this type will be part of the final LPFSA. RAWDAR RSVTSC-1 Missions Data fully fetched • EXOSAT RSVTSC-1 • HERSCHEL Concluded missions Concluded • HIPPARCOS • IUE • PLANCK • ROSETTA • SMART-1 • VEX RAWDAR EXOSAT • Category: – RSVTSC-1: Concluded mission – Data fully fetched • General Mission Information: – http://sci.esa.int/exosat/ • Data: – Estimated Total Size: 38.57 GB – Available Locally: 38.57 GB / 3 Directories / 6,686 Files – Data location: /data/RADACER/missions/EXOSAT/DATA/ – Data structure / content:RSVTSC-1 • Data currently stored in files named by their corresponding FOT identifier (eg. E0004.fot.tar.gz, E0007.fot.tar.gz, etc • These data files were downloaded from the EXOSAT web site and were also tar and gz-compressed. • Data Source – Data originally stored in 1600 BPI 12" magnetic tapes (Final Observation Tapes, or FOTs). – Also available from the ESA-ESTEC cosmos web site at: http://www.cosmos.esa.int/web/exosat/fots-1983 RAWDAR HERSCHEL • Category: – RSVTSC-1: Concluded mission – Data fully fetched • General Mission Information: – http://sci.esa.int/herschel/ • Data: – Estimated Total Size: 1.67 TB – Available Locally: 1.67 TB / 27 Directories / 97,641 Files – Data location: /data/RADACER/missions/HERSCHEL/DATARSVTSC-1 – Data structure/content: • The telemetry is organized as one directory per APID, with one folder per APID • Within APID folders, there are files: – each covering an interval of 16384 seconds – named in the form: 0016_106222057422848.TLM where 0016 is the APID and 106222057422848 is the onboard CUC time of the start of the period • Data Source: – Copied from an external HDD at ESAC. RAWDAR HIPPARCOS • Category: – RSVTSC-1: Concluded mission – Data fully fetched • General Mission Information: – http://sci.esa.int/hipparcos/ • Data: – Estimated Total Size: 1.28 TB – Available Locally: 1.28 TB / 1,044 Directories / 32,194 Files – Data location: /data/RADACER/missions/HIPPARCOS/DATA/RSVTSC-1 – Data structure / content: • at first level one folder for each usb stick, named set_1, set_2, …, set_8 • within these, there are files named in the form HA0284.tap, HA0285.tap, etc • Data source: – Copied from 8 HDDs of 250 GB each RAWDAR IUE • Category: – RSVTSC-1: Concluded mission – Data fully fetched • General Mission Information: – http://sci.esa.int/iue/ • Data: – Estimated Total Size: 213.94 GB – Available Locally: 213.94 GB / 133 Directories / 694,001 Files – Data location: /data/RADACER/missions/IUE/DATA/RSVTSC-1 – Data structure / content: • Raw telemetry is organized in directories under camera name and image sequence number • A separate directory exists for each camera (i.e., lwp, lwr, swp, swr) • Below each of these directories is a separate directory for each 1000 image numbers • These directories are named according to the starting image number in each group (i.e., 1000, 2000, 3000, etc.). • Data source: – Copied via ftp://archive.stsci.edu/pub/iue/data/ RAWDAR PLANCK • Category: – RSVTSC-1: Concluded mission – Data fully fetched • General Mission Information: – http://sci.esa.int/planck/ • Data: – Estimated Total Size: 2.52 TB – Available Locally: 2.52 TB / 1,237 Directories / 20,568 Files – Data location: /data/RADACER/missions/PLANCK/DATA/RSVTSC-1 – Data structure / content: • The raw packet telemetry data is organized in directories first by data stream, then followed by the SPID • The FARC area contains a number of additional files such as: – discarded packets in raw format – telecommand history files as provided by the DDS system – complete checkout of the FARC • Data source: – Copied from 2 HDDs at ESAC. RAWDAR ROSETTA • Category: – RSVTSC-1: Concluded mission – Data fully fetched • General Mission Information: – http://sci.esa.int/rosetta/ • Data: – Estimated Total Size: 425.71 GB – Available Locally: 425.71 GB / 42 Directories / 219,025 Files – Directory location: /data/RADACER/missions/ROSETTA/DATA/RSVTSC-1 – Data structure / content: • This data is distributed as Instrument: Science data size / Housekeeping data size • - ALICE: 1.6 GB / 137 MB • - CAM1: 12 GB / 259 MB • - CAM2: 4.8 MB / 204 KB • - COSIMA: 1.1 GB / 81 MB • - GIADA: 175 MB / 319 MB (continued) RAWDAR ROSETTA – Data structure / content (continued): • - MIDAS: 294 MB / 992 MB • - MIRO: 7.4 GB / 436 MB • - OSIRIS: 22 GB / 192 MB • - ROSINA: 4.5 GB / 243 MB • - RPC: 7.2 GB / 316 MB • - VIRTIS: 23 GB / 377 MB • Other: • - Command History Files: 148 MB RSVTSC-1 • - Event Files: 117 MB • Data source: – RAW telemetry received from MOC is stored in a Linux server at ESAC. – The raw data were copied to RADACER through rsync. RAWDAR SMART-1 • Category: – RSVTSC-1: Concluded mission – Data fully fetched • General Mission Information: – http://sci.esa.int/smart-1/ • Data: – Estimated Total Size: 170.75 GB – Available Locally: 170.75 GB / 1,286 Directories / 187,805 Files – Directory location: /data/RADACER/missions/SMARTRSVTSC-1 -1/DATA/ – Data structure / content: • The telemetry for the SMART-1 mission is structured first by year and subsequently by the day of the year, eg 2003269 corresponds to the 269th day of 2003 • Data source: – Copied from an external drive at ESAC. RAWDAR VEX (VENUS EXPRESS) • Category: – RSVTSC-1: Concluded mission – Data fully fetched • General Mission Information: – http://sci.esa.int/venus-express/ • Data: – Estimated Total Size: 1.09 TB – Available Locally: 1.09 TB / 25,534 Directories