ScienceScience DataData ReductionReduction PipelinesPipelines atat NASANASA 3030 YearsYears andand countingcounting

DonDon LindlerLindler SigmaSigma SpaceSpace CorporationCorporation JanuaryJanuary 24,24, 20072007 InternationalInternational UltravioletUltraviolet ExplorerExplorer JointJoint ProjectProject betweenbetween NASA,NASA, ESA,ESA, andand UKUK

„ LaunchedLaunched onon JanuaryJanuary 26,26, 19781978 „ TurnedTurned offoff onon SeptemberSeptember 30,30, 19961996 „ 18.718.7 yearsyears ofof operationoperation „ 104470104470 SpectraSpectra „ 96009600 astronoastronomicalmical sourcessources IUEIUE PipelinePipeline andand ArchiveArchive „ FirstFirst astronomicalastronomical andand satellitesatellite facilityfacility toto deliverdeliver fullyfully reducedreduced datadata withinwithin 4848 hourshours toto thethe worldwideworldwide communitycommunity ofof scientistsscientists „ thethe creationcreation ofof thethe firstfirst worldwideworldwide astronomicalastronomical reducedreduced--datadata archivearchive deliveringdelivering 44,00044,000 spectraspectra perper yearyear (5(5 spectraspectra perper hour)hour) toto astronomersastronomers inin 3131 countriescountries IUEIUE PipelinePipeline DevelopmentDevelopment „ UtilizedUtilized NASA/JPLNASA/JPL VICARVICAR systemsystem (Video(Video ImageImage CommunicationCommunication andand Retrieval)Retrieval) „ VICARVICAR portedported fromfrom IBMIBM mainframemainframe toto aa SigmaSigma 99 computer.computer.

Good Enough Much Better WhyWhy choosechoose VICAR?VICAR?

„ Analysis Tools for calibration „ Modules for geometric and photometric correction of images

BIGGEST PROBLEMS „ Expensive transition from IBM to Sigma 9 „ Astronomers were not using Sigma 9 computers ReasonsReasons forfor SuccessSuccess

„ Plenty of Funding! „ Modular coding „ Evolution of pipeline algorithms based on the Science and controlled by the users. Frequent meetings (Three Agency and calibration meetings) „ Astronomers were well informed (IUE newsletter) „ Dialog between astronomers and the observatory staff. Astronomers came to the control center for real time observing. ReasonsReasons forfor Success,Success, continuedcontinued

„ IntermediateIntermediate outputsoutputs suppliedsupplied toto astronomers.astronomers. „ DataData QualityQuality FlagsFlags includedincluded withwith outputoutput spectra.spectra. „ IUEIUE RegionalRegional DataData AnalysisAnalysis Facilities.Facilities. NEWSIPSNEWSIPS NewNew SpectralSpectral ImagingImaging ProcessingProcessing PipelinePipeline

„ ConsistencyConsistency betweenbetween archivedarchived productsproducts duringduring thethe extendedextended mission.mission. „ AfterAfter aa longlong studystudy ofof applicationapplication executives,executives, itit waswas developeddeveloped underunder MIDAS.MIDAS. „ MoveMove fromfrom VICARVICAR toto FITSFITS format.format. „ ImprovementsImprovements inin S/N,S/N, spectralspectral resolution,resolution, andand absoluteabsolute sensitivity.sensitivity. „ Again,Again, plentyplenty ofof funding!funding! HubbleHubble SpaceSpace TelescopeTelescope

Launched April 24, 1990 Five initial instruments - High Resolution Spectrograph - Faint Object Spectrograph - High Speed Photometer - Faint Object Camera - Wide Field and Planetary Camera Five Instrument Science Teams with Guaranteed Observing Time InstrumentInstrument TeamTeam SoftwareSoftware DevelopmentDevelopment WorkWork beganbegan inin thethe latelate 1970s1970s EachEach teamteam workedworked independently.independently. LittleLittle coordinationcoordination betweenbetween teams.teams. HSTHST LaunchLaunch scheduledscheduled forfor 19831983 GHRSGHRS andand FOSFOS selectedselected toto useuse IDLIDL asas thethe primaryprimary analysisanalysis tool/programmingtool/programming language.language. GHRSGHRS systemsystem waswas aa majormajor success!success! System used as recently as last year to analyze 20 year old spectral calibration lamp data. IDL used for data acquisition, archival, pipeline processing, and analysis All data raw data easily search and immediately accessible. “On-the-fly” pipeline processing Rapid Prototyping Propagation of errors. GHRSGHRS continued.continued.

GHRSGHRS approachapproach latterlatter extendedextended toto FOS,FOS, STIS,STIS, ACS,ACS, andand WFC3WFC3 instrumentinstrument developmentdevelopment andand toto somesome extentextent COS.COS. BiggestBiggest ProblemProblem withwith GHRSGHRS system:system: FormatsFormats defineddefined inin thethe prepre--FITSFITS eraera andand beforebefore STST ScIScI formatsformats werewere defined.defined. HST PODPS (First Try) Hubble Space Telescope Post Observational Data Processing System z Part of the Science Operations Ground System (SOGS) z 1980 to 81: 2 inch thick requirements document written by NASA appointed committee z 1981: Contract award to TRW z 1983: First software components delivered for a DEC/VMS based system. PODPS 1st try continued z End User was not involved in the design! 1st Try continued. z SOGS used last-generation programming technology. - Supply requirements. - Detailed Design. - No prototyping. z Non-modular. Fixing a bug in one part of the program can easily generate a bug somewhere else. z Very little contact with the instrument teams. PODPS (2nd Try) z The Space Telescope Science Institute enters the picture. z They inherit an unusable system. z Generation of calibration reference files was not part of the system! z They negotiate with the instrument teams for a joint development. This time with ST ScI oversight. z ST ScI Staffing increased from the proposed 100 people to around 400 people at launch. PODPS (2nd Try) continued z Pipeline software under SDAS (Science Data Analysis System) running under IRAF. z Prototyping of software by Instrument Teams z Problems - Machine dependent (DEC/VMS) - Machine dependent FITS-like data formats (.HHH and .HHD files) - VMS FORTRAN extensions used. - Intermediate results not available for all instruments. PODPS (3rd Try) z Name changed from SDAS to STSDAS z First stage: Code no-longer machine dependent z Second stage: Data Formats no longer machine dependent. Changed to FITS. z Third stage: Development of OPUS OPUS z Operational since 1995. z Fully distributed for any series of applications. z Runs multiple instances of multiple processes in multiple pipelines on multiple nodes. z Easy integration of your own pipeline steps. z Includes monitoring tools. z Best of all: The developers of the pipeline modules do not need to know how to use OPUS. z Complete record of processing included in the output. OPUS continuing evolution

z Observation Associations z On-The-Fly calibration z Automatic generation of calibration reference files (e.g. darks, flats). OPUS sites z Hubble Space Telescope (HST). z Far Ultraviolet Spectroscopic Explorer (FUSE). z International Gamma-Ray Astrophysics Laboratory (Integral) z Chandra X-Ray Observatory, AXAF z BeppoSAX X-ray Observatory z Spitzer Space Telescope, SIRTF z The Gemini Observatory Continuing Problem Areas z Deciding what enhancements to make when manpower is limited. A well defined decision process is needed with decisions based on the scientific return. Continued Problem Areas z Systematic Errors z Standard Data Quality Flags – 8, 16, or 32 bit quantity with each bit representing a different condition – Numeric score with each condition representing a different condition. Only the “worst” condition is flagged. – Binary z Analysis software which uses the flags The Future?

zModel Based Calibration “Calibration based on instrument models has been demonstrated to provide better accuracy than empirical methods, but in addition it also provides a real understanding of the instrument that enables one to maintain it at maximum Award winning Astronomers performance and quickly prefer Model Based Calibration diagnose any deviations” prefer Model Based Calibration Michael Rosa, ST-ECF Conclusion

You work hard, develop new ideas, and improve your instrument calibration. At the end of the day, you go home and have a beer. You repeat this day after day. In the end all that really matters is the beer.