The Community-Developed Software Model for Astropy and JWST

Erik Tollerud @eteq Giacconi Fellow (STScI) Astropy Coordination Committee Member “Traditional” Approaches to Astronomy Software Monolithic/Top-down

Lone Wolf (or Wolf Pack) Programming Python is now the dominant Language in Astro

Momcheva & Tollerud 2015 Astropy’s Origin Story

Q. How do I use python to convert from Equatorial J2000 RA/Dec to Ecliptic coordinates (as of 2011)? A. Use any of: •pyast •Astrolib •Astropysics Lots of wasted effort! •Kapteyn •PyEphem Mutually incompatible! •PyAst •PyAstro •Probably more... Astropy’s Origin Story

Everyone agreed this was bad.

Do we as a community really need yet another separate python library for astronomy and yet another attempt at building a core set of routines ported from the IDL library?

Marshall Perrin on “astropy” mailing list, June 2011 Astropy’s Origin Story Everyone agreed this was bad. (Agreement ends up crucial to shared development.) A grassroots discussion started in June 2011, followed by a series of votes (~100 astronomers), and some initial dev work (incl. STScI programmers on JWST). The Result:

Check out http://bit.ly/astropyvision for the original “vision” What is the philosophy behind Astropy?

(Professional) Astronomers help This means both by write it and for the community It should be useful for them as part of their day-to-day work Astronomers and Software have a Rocky Relationship

(Of those who write code:) Do you have training in writing software?

Momcheva & Tollerud 2015 Contributions To Astropy are ⪞200 and rising

Most of these astronomers have never met! Getting Scientists to work together is like Herding Cats…

You just need to give them a structure in which to work and they’ll do it themselves Community Development: Github Community Development: Github PRs Community Development: Github PRs Community Development: Github PRs

Consensus! Community Development: Github PRs

This is what makes it a Do-ocracycommunity project - a The work that gets done is the work that someone does…

Applies both for authoring and reviewing. What is the philosophy behind Astropy?

(Professional) Astronomers help This means both by write it and for the community It should be useful for them as part of their day-to-day work How can this possibly Stay Stable? Keys To Distributed Development: Testing Keys To Distributed Development: Testing in the Code Keys To Distributed Development: Continuous Integration Keys To Distributed Development: Docs In the Code Keys To Distributed Development: Docs on the Web

I Keys To Distributed Development

• git + Github for sharing • Test everything (automatically) • Easy documentation (⇒ thorough) And It Actually DOES enable science

Astropy Collaboration et al. 2013 A&A 558 A33 http://docs.astropy.org And Large Projects

• James Webb Space Telescope (JWST) • Tuna at LAM, pipeline for reduction of Fabry-Perot interferograms • (HST) Pipelines • Gemini data reduction software, and • LSST Science Pipelines Science and Data quality pipeline

• Gran Telescopio Canarias (GTC) pipeline • Dark Energy Spectroscopic Instrument for EMIR (Infrared MOS) and MEGARA (DESI) (Optical MOS and IFU) • MUSE - MPDAF (MUSE Python Data • INAF - DISCOS project Analysis Framework)

• Cherenkov Telescope Array Pipeline • AAO - SAMI Galaxy Survey framework • Hubble Frontier Fields • SimCADO - the MICADO+E-ELT instrument simulator • Hubble RELICS Project

• Code Investigating GALaxy Emission • UTFSM - Universidad Técnica Federico (CIGALE) Santa María

• The Herschel Extragalactic Legacy Project • Daniel K. Inouye Solar Telescope (DKIST) data management software The “For” part

This means both by and for the community It should be useful for them as part of their day-to-day work Astropy Core Package (currently v1.3) Best place to look is always http://docs.astropy.org

• Units and “Quantities” (arrays • nddata: Image analysis and with units that act the way interoperability data structures you’d expect). Integrated with comprehensive astro- • WCS (pixel ↔ sky mapping) appropriate physical constants • Extensible I/O: FITS, VOTable, • Date/time good to nanoseconds hdf5, custom over a Hubble time • Data modeling and fitting

• Celestial coordinates and their • Common Astrostatistics tools transformations (incl. solar system-appropriate systems) • (Cosmology tools)

• Table manipulation, including • Configuration and plumbing to many arcane astro formats make it all work Astropy Core Package &

Astropy is the framework on which JWST’s pipelines and data analysis tools are (and will be) built. For example:

• astropy.io.fits is used to load FITS files • astropy.units for representing units and unit-ful quantities • astropy.table objects are used to store inputs/outputs from JWST codes • astropy.coordinates are used where coordinate transformations are needed • etc… How do you learn More about Using Astropy?

http://tutorials.astropy.org

http://docs.astropy.org Python in Astronomy Facebook group

http://gitter.im : astropy/astropy chat room Astropy mailing list astropy-dev mailing list

If you have ideas: http://astropy.userecho.com How does this connect back to the Broadest sense of “Community” Software? The astropy Project and package

Core package “astropy” ≠ “Astropy Project”

The core package is what’s in github repo astropy/astropy. I.e., what “pip install astropy” or “conda install astropy” gets you.

“Astropy Project” includes all the affiliated packages and associated community Astropy “Affiliated Packages”

An affiliated package is an astronomy-related Python package that is not part of the astropy core package, but has requested to be included as part of the Astropy project’s community. What Unites them?

• A common goal and vision: reducing duplication and embracing good coding practices (testing, docs) • Listing on http://affiliated.astropy.org (curated and reviewed by the Astropy coordination committee) • (For many) a package template Affiliated Package Template

• Contains a ready-to-go “copy” of the astropy package layout. (Leans heavily on astropy-helpers) • Provides documentation tools, testing framework, cython, configuration, etc. • Docs on how to actually make it all work! Affiliated Packages Best place to look is always http://affiliated.astropy.org • APLpy: astronomical plotting • ccdproc: ccd reductions • astroquery: access to internet- accessible astronomy resources • photutils: photometry

• ginga: interactive image viz • specutils: spectroscopy

• imexam: quick image analysis • gammapy: gamma-ray astronomy

• pydl: simple IDL ports • sncosmo: supernova light curves fitting/typing/etc • PyVO: VO access • halotools: high-performance tools • WCSAxes: WCS-aware matplotlib for using n-body simulations to plots model galaxy formation • pyregion: ds9 region files • galpy: tools for Galactic dynamics • -wrapper: image mosaicing Affiliated Packages for

Most JWST data analysis tools will either be actual affiliated packages, or in the same framework • photutils: Aperture and PSF photometry for NIRCam, MIRI • specviz, mosviz, cubeviz: Spectroscopic visualization for NIRSpec • specutils: “Gateway” to connect the above to other spectra • gwcs: Sky <-> Pixel supporting distortions, IFUs, etc. • webbpsf & poppy: Optical modeling of the JWST PSF (or other ST’s!) • (future) jwst_psftools, jwst_autoreduce, etc. JWST Data Analysis Ecosystem

Spectral visualization tools Specutils Image visualization tools io models training & documentation modeling fitting astropy-helpers gwcs: generalized wcs wcs asdf: advanced data format JWST Tools tutorials software distribution astroquery MAST JWST data structures nddata models & fitting Astronomy Python Tool coordination photometry tools Development at STScI

Jupyter notebook tutorials IRAF Add to existing python lib cosmology Astropy replacement Build replacement pkg constants (mostly external) IRAF switchers guide affiliated packages no replacement Photutils: Community Photometry Tools

Feel free to check out http://bit.ly/jwst-phot-tutorials v0.3, http://photutils.readthedocs.io Photutils: Object-Finding Photutils: Aperture Photometry Photutils: PSF Photometry SpecViz

v1.0, http://specviz.readthedocs.io These All Aim To Be Community Software How Can you Get Involved with JWST Data Analysis Tool Efforts?

• Try out the tools on your own data… • Report issues/suggestions on Github • Sign up for the DADF mailing list (http://bit.ly/jwst-dadf-org) • Comment on or contribute code via Pull Request! This then provides A context for Community Development For

• an_ers_programs_pipeline • your_subfields_jwst_science_code • my_subfields_jwst_science_code

Upshot: Community developed software, esp. Astropy, its affiliated packages, and their infrastructure, will help us all do our JWST (and other!) science together, better. Astropy’s Development (The spheres are files, and the avatars are contributors.)