Ten Challenges of Producing an Astronomical Gigapixel Image Best Practices Practices Best Mathias Jäger Lars Lindberg Christensen Keywords ESA/Hubble Public Information Officer Head of ESO education and Public Outreach Image processing, astronomical images, [email protected] Department (ePOD) VISTA, VVV survey [email protected] Summary Public outreach involves developing new methods, testing new technologies and integrating new ideas. Sometimes, the craft of outreach even leads into completely unknown territory. This is the story of a project that led into astronomical and technological terra incognita. It is about the production of a mosaic of the central parts of the Milky Way made with ESO’s VISTA telescope as part of the VVV survey. The outreach system at ESO was tested to its limits, and beyond, by the pro- duction of what is still likely to be the largest astronomical image in the world. Several significant challenges had to be over- come, extensive hardware and software upgrades were undertaken and compromises had to be made to produce this stunning image for the public. VISTA and the VVV survey source of stunning images to be used for public outreach team quickly realised that public outreach. As of March 2015 ESO the huge mosaic of the central Milky Way VISTA — the Visible and Infrared Survey has published 17 press releases based on could be the main focus of a release. When Telescope for Astronomy — is part of results delivered by VISTA, seven of which the work on the image began the team was ESO’s Paranal Observatory and is the larg- were directly related to the VVV survey. confronted with ten major problems which est survey telescope in existence. It is also had to be solved. the most powerful near-infrared survey tel- escope ever built. The telescope has a The final image 4.1-metre primary mirror and is dedicated Problem 1: Getting the astronomical to conducting wide-angle surveys of the The image which tested the limits of ESO’s data skies with its 67-megapixel digital camera. system, and its team members, covers Simply moving the raw data from place to about 315 square degrees (20.4 × 15.4 place turned out to be a great challenge. VISTA’s observing time is entirely devoted degrees) and shows the centre of the Early on it was decided — on an excep- to mapping the sky systematically and six Milky Way. The observations were carried tional basis and only because of the sheer huge public surveys will take up the major- out using three different near-infrared (JHK) amount of data — that the data had to be ity of the telescope’s first years of opera- filters and the resulting image is monu- down-sampled by a factor of almost two. tion. One of these surveys is the VVV — or mental. It is 9 gigapixels in size, measur- Ordinarily ESO always releases images VISTA Variables in the Vía Láctea — survey, ing 108 199 by 81 503 pixels1. The image pixel by pixel as they are observed, to which started in 2010 and was granted a is so large that, if printed with the resolution make sure that the images the end-user total of 1929 hours of observing time over of a typical book, it would be nine metres receives are optimal. With the support of a five-year period. The survey is scanning across and seven metres tall. This makes ESO’s helpdesk the available storage on the southern plane and bulge of our gal- it likely the largest astronomical image in the ESO FTP server was extended and the axy — 520 square degrees in total — in the world. vast quantities of data — 166 gigabytes five near-infrared filters. of FITS3 files — could be moved back The gigantic dataset contains about and forth between science team member When the survey is completed in 2016, the 173 million objects, out of which about Ignacio Toledo (Atacama Large Millimeter/ outcome will be a catalogue with around 84 million have been confirmed as stars, submillimeter Array [ALMA]) and ESO over one billion point sources, including about which is ten times as many stars as in any several iterations. a million variable objects. This will be used previous study. It is a major step forward to create a three-dimensional map of the in our understanding of our home galaxy. bulge of our galaxy, the Milky Way, to cal- Problem 2: Distortions over the large culate the ages of stellar populations and field study the evolution of globular clusters. Ten problems to solve The next problem occurred during the astronomical data processing. Due to the In addition to its scientific purpose, VISTA’s Work on the image began in 2012 with the large field of view, the image happened superb image quality and the wide field preparation of a science release2 based to have significant projection and distor- of view make the telescope an excellent on a paper by Saito et al. (2012). The ESO tion effects. Most of these problems were CAPjournal, No. 17, June 2015 29 Ten Challenges of Producing an Astronomical Gigapixel Image Figure 1. This striking view of the central part of the Milky Way was obtained with the VISTA survey telescope at ESO’s Paranal Observatory in Chile. This huge picture is 108 200 by 81 500 pixels and contains nearly nine billion pixels. It was created by combining thousands of individual images from VISTA. Credit: ESO/VVV Survey/D. Minniti 30 CAPjournal, No. 17, June 2015 CAPjournal, No. 17, June 2015 Ten Challenges of Producing an Astronomical Gigapixel Image 31 Ten Challenges of Producing an Astronomical Gigapixel Image Figure 2. VISTA is the largest telescope in the world dedicated to surveying the sky. Its primary mirror is 4.1 metres in diameter and it has a huge camera that is sensitive to near-infrared wavelengths. Credit: ESO corrected by Ignacio Toledo. However, a format, failed as the ESO/ESA/NASA FITS Problem 4: Reading BigTIFF much smaller residual misalignment of Liberator programme4 was unable, at the Not only did the available version of the only a few pixels between the three differ- time, to create tiff files above 2 gigabytes. FITS Liberator prove to be inadequate, ent exposures (JHK filters) had to be cor- Since the dynamic range in the dataset was but Photoshop 5, which was used by ESO rected manually in Photoshop later on. not extremely large, the team reverted to at the time, also fell short as it could not using a less interactive method, using the read BigTIFF files properly. Thankfully, software STIFF5 which can create BigTIFF6 and as a matter of pure luck, Photoshop Problem 3: Dynamic range compression files of almost unlimited size. Meanwhile, 6 was released only a few weeks before The normal dynamic range compression, the FITS Liberator programme has now the start of the project and this version was which converts files from FITS format to tiff been updated to write BigTIFF files as well. able to read BigTIFF files. With the new 32 CAPjournal, No. 17, June 2015 CAPjournal, No. 17, June 2015 Figure 3. This view compares the huge mosaic in infrared light from the VISTA survey telescope and a visible-light mosaic view of the same region taken with a small telescope. Credit: ESO/VVV Survey/D. Minniti/Serge Brunier. Acknowledgement: Ignacio Toledo, Martin Kornmesser software installed, the images could finally Problem 7: Cosmetic cleaning Problem 9: Zoomable version be opened in Photoshop; and due to their The next step was a cosmetic cleaning Assuming that very few users would be size had to be reduced to eight-bit colour stage to remove instrumental artefacts able to actually handle the full 9-gigapixel depth immediately — normally astronom- and other blemishes of a non-cosmic ori- image, ESO wanted to offer a zoomable ical images have 16-bit colour depth and gin. ESO normally uses an outsourcing image10 as the main vehicle for deliver- hence have twice the size. This is some- company, but in this case the transport of ing the experience to the public. However thing which is otherwise only done at the the individual layers was impractical and Zoomify11, the usual tool for creating zoom- last stage in the production of an astro- the clean itself would have been too costly able products at ESO, did not work with PSB nomical colour composite. as it is paid for per megapixel. Therefore it files. The tools Krpano12 and Panotour Pro was decided to perform a more modest, from Kolor13 were finally used and proved but sufficient, in-house cleaning. This led themselves to be very good alternatives. Problem 5: Swapping 600 gigabytes to some interesting feedback from the pub- As soon as the team started to work on lic later on, which will be discussed in the the image in Photoshop the next challenge lessons learned. Problem 10: Web serving emerged. Working on three 9-gigapixel JHK The final challenge that the team had layers with corresponding adjustment lay- to face was to actually serve the large ers in Photoshop used unforeseen amounts Problem 8: Distribution file format individual files and the panorama to the of memory — up to 600 ­gigabytes of mem- It was clear that the final image should be public. The news of the image spread like ory/swap space. To perform the swapping accessible to as many people as possi- wildfire. The image is the most successful a state-of-the-art solid-state disk (SSD) ble, so the team had to look for the most ESO release to date with many more than was quickly procured to give a workable appropriate and viewable final file format.