DOCSLIB.ORG

The Role of CPU-Based Correlators in Radio Astronomy

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
The Role of CPU-Based Correlators in Radio Astronomy The Role of CPU-Based Correlators in Radio Astronomy Walter Brisken [email protected] National Radio Astronomy Observatory 12 February 2020 1 / 15 CPU correlation ∗ I prefer not to over-use term \software correlator" ◦ GPU correlators are also software correlators ◦ All modern correlators contain significant amounts of software ◦ E.g., VLBA hardware correlator had more lines of code than DiFX at time VLBA switched! ∗ CPU correlation is distinguished by its relative portability 2 / 15 The Distributed FX (DiFX) correlator ∗ Was half of Adam Deller's PhD dissertation project ∗ Since then, developed and maintained by global team ∗ Full-featured F-X architecture VLBI correlator ∗ Now in operational use for: ◦ Very Long Baseline Array (VLBA) ◦ VLITE (see Tracy Clarke's talk) ◦ MPIfR Bonn ◦ Haystack Observatory ◦ US Naval Observatory (USNO) ◦ Korean VLBI Network (KVN) ◦ Australian Long Baseline Array (LBA) ◦ Event Horizon Telescope (EHT) ∗ More info at: https://www.atnf.csiro.au/vlbi/dokuwiki/doku.php/difx/start 3 / 15 DiFX features ∗ Continent-scale baselines and space VLBI ∗ Geodetic VLBI: pulse cal, model accountability ∗ Spectral resolution to 2 Hz ∗ Time resolution to milliseconds ∗ Pulsar gating and binning ∗ Massive (hundreds) multi-phase center capability ∗ Correlation of mis-matched bands (EHT is extreme case) ∗ Polarization conversion (mixed X-Y and R-L) ∗ Floating point input and output (approx. 20 ENOB) ∗ Parallelized on time axis: very flexible ∗ Accepts many data formats, real/complex, upper/lower sideband, interoperably ∗ Operates in realtime (e.g., VLITE) or off-line 4 / 15 DiFX: technologies employed ∗ C++ and C for heavy lifting code ∗ Python for peripheral code / scripts ∗ OpenMPI: cluster-level parallelization ∗ SMP threading: core-level parallelization ∗ Intel IPP: instruction-level parallelization ∗ Multicast XML: monitor and control system ∗ Uses standard VLBI data formats: Mark5B, VDIF, VEX, FITS, MarkIV 5 / 15 Platforms supporting DiFX Raspberry Pi MAGNUS supercomputer, Laptop (Linux or Apple) Perth, AU ∗ Don't underestimate the value of running a correlator on a laptop! 6 / 15 VLBA DiFX Correlator ∗ Upgraded late 2019 ∗ 20 processing servers, 24 CPU cores each ∗ 40 Gbps Infiniband backplane ∗ Can process 10 antennas at 1 GHz bandwidth ∗ Power consumption: 7 kW ∗ Rack space: about 1 m ∗ Cost: $100k ∗ Easy today 7 / 15 Some things learned ∗ Be careful about longevity and portability of third-party libraries ◦ So far DiFX has been mostly lucky ◦ pgplot and some Fortran code are getting quite annoying though. ∗ DiFX is thermal limited, not I/O limited or instruction limited ◦ Cache hit optimization has been done ◦ Code can only be sped up significantly by reducing mathematical operations ◦ Mind the Thermal Design Power (TDP) when choosing CPUs ◦ Benchmarking and optimization can yield very puzzling results ∗ Performance driven mostly by number of antennas and bandwidth; spectral and time resolution important only in extreme cases. 8 / 15 CPU correlation for EVLA? ∗ Performance requirement: 27 antennas at 8 GHz ∗ Would require 1000 servers (2019 technology) ∗ Power consumption: 350 kW ∗ Rack space: about 50 m (about 25 racks) ∗ Cost: $6M ∗ Failure rate: approx. 1 server per month ∗ Not crazy now; easy in 15 years? ∗ Would have been 1 km tall rack when EVLA was inaugurated 9 / 15 CPU correlation for ALMA2030? ∗ Performance requirement: 70 antennas at 20 GHz Where to put such a correlator? ∗ Would require 20000 servers (2019 technology) ∗ Power consumption: 7 MW ∗ Rack space: about 1 km ∗ Cost: $120M ∗ Failure rate: approx. 1 server per day ∗ Unlikely to be practical 10 / 15 CPU correlation for ALMA2030? ∗ Performance requirement: 70 antennas at 20 GHz ∗ Would require 20000 servers (2019 technology) ∗ Power consumption: 7 MW ∗ Rack space: about 1 km ∗ Cost: $120M ∗ Failure rate: approx. 1 server per day ∗ Unlikely to be practical ∗ Would fit nicely inside the Jeddah Kingdon Tower, Saudi Arabia 11 / 15 CPU correlation for ngVLA? ∗ Performance requirement: 260 antennas at 20 GHz ∗ Would require 300000 servers (2019 technology) ∗ Power consumption: 100 MW ∗ Rack space: about 15 km (could host SOFIA at top) ∗ Cost: $2B ∗ Failure rate: approx. 1 server per hour ∗ Keep dreaming! 12 / 15 Why am I still talking? ∗ Clearly ALMA2030 correlator won't be a CPU correlator ∗ There are many reasons to include CPU correlators in the ALMA2030 landscape. ◦ A reference for correctness and digital efficiency comparison ◦ Testbed for new features ◦ Antenna / array diagnostics ◦ Replacement for ACA correlator? 13 / 15 Proposed: a two correlator solution ∗ Correlator 1: a new GPU, FPGA, or ASIC real-time correlator ◦ Use for 99% of science cases ∗ Correlator 2: a CPU-based offline correlator ◦ Use for perhaps 1 hour of observing per week ◦ Pair with multi-petabyte baseband data storage array ◦ Use for special processing: planetary radar, pulsars, FRBs, . ◦ Use for development (correlator, calibration, . ) ◦ Use for wide variety of tests ◦ Allow inclusion of non-ALMA antennas in correlation (e.g., EHT) ◦ Reduce worst-case specifications from correlator 1 14 / 15 Conclusions ∗ CPU-based correlators offer considerable flexibility with low development overhead ∗ They won't displace GPU, FPGA, ASIC correlators for large-N correlation at high bandwidth ∗ They remain highly relevant for I/O restricted cases (e.g., VLBI) ∗ They remain useful in secondary roles alongside heavy lifting correlators ∗ Moving to CPU-based correlation moves support from electronics to software and could have socialogical effects 15 / 15.
Load more

Recommended publications
  • High Resolution Radio Astronomy Using Very Long Baseline Interferometry
    IOP PUBLISHING REPORTS ON PROGRESS IN PHYSICS Rep. Prog. Phys. 71 (2008) 066901 (32pp) doi:10.1088/0034-4885/71/6/066901 High resolution radio astronomy using very long baseline interferometry Enno Middelberg1 and Uwe Bach2 1 Astronomisches Institut, Universitat¨ Bochum, 44801 Bochum, Germany 2 Max-Planck-Institut fur¨ Radioastronomie, Auf dem Hugel¨ 69, 53121 Bonn, Germany E-mail: [email protected] and [email protected] Received 3 December 2007, in final form 11 March 2008 Published 2 May 2008 Online at stacks.iop.org/RoPP/71/066901 Abstract Very long baseline interferometry, or VLBI, is the observing technique yielding the highest-resolution images today. Whilst a traditionally large fraction of VLBI observations is concentrating on active galactic nuclei, the number of observations concerned with other astronomical objects such as stars and masers, and with astrometric applications, is significant. In the last decade, much progress has been made in all of these fields. We give a brief introduction to the technique of radio interferometry, focusing on the particularities of VLBI observations, and review recent results which would not have been possible without VLBI observations. This article was invited by Professor J Silk. Contents 1. Introduction 1 2.9. The future of VLBI: eVLBI, VLBI in space and 2. The theory of interferometry and aperture the SKA 10 synthesis 2 2.10. VLBI arrays around the world and their 2.1. Fundamentals 2 capabilities 10 2.2. Sources of error in VLBI observations 7 3. Astrophysical applications 11 2.3. The problem of phase calibration: 3.1. Active galactic nuclei and their jets 12 self-calibration 7 2.4.
    [Show full text]
  • EARLY SCIENCE with the KOREAN VLBI NETWORK: the QCAL-1 43 Ghz CALIBRATOR SURVEY
    The Astronomical Journal, 144:150 (6pp), 2012 November doi:10.1088/0004-6256/144/5/150 C 2012. The American Astronomical Society. All rights reserved. Printed in the U.S.A. EARLY SCIENCE WITH THE KOREAN VLBI NETWORK: THE QCAL-1 43 GHz CALIBRATOR SURVEY Leonid Petrov1, Sang-Sung Lee2, Jongsoo Kim2, Taehyun Jung2, Junghwan Oh2,3, Bong Won Sohn2,3, Do-Young Byun2,3, Moon-Hee Chung2, Do-Heung Je2, Seog-Oh Wi2, Min-Gyu Song2,JimanKang2, Seog-Tae Han2,3, Jung-Won Lee2, Bong Gyu Kim2,3, Hyunsoo Chung2, and Hyun-Goo Kim2 1 Astrogeo Center, Falls Church, VA 22043, USA 2 Korean VLBI Network, Korea Astronomy and Space Science Institute, 776, Daedeokdae-ro, Yuseong-gu, Daejeon 305-348, Republic of Korea; [email protected] 3 Yonsei University Observatory, Yonsei University, Seongsan-ro 50, Yonsei-ro, Seodaemun-gu, Seoul 120-749, Republic of Korea Received 2012 July 24; accepted 2012 September 4; published 2012 October 18 ABSTRACT This paper presents the catalog of correlated flux densities in three ranges of baseline projection lengths of 637 sources from a 43 GHz (Q band) survey observed with the Korean VLBI Network. Of them, 14 objects used as calibrators were previously observed, but 623 sources have not been observed before in the Q band with very long baseline interferometry (VLBI). The goal of this work in the early science phase of the new VLBI array is twofold: to evaluate the performance of the new instrument that operates in a frequency range of 22–129 GHz and to build a list of objects that can be used as targets and as calibrators.
    [Show full text]
  • Exploring the Cosmic Frontier Astrophysical Instruments for the 21St Century ABSTRACT BOOK
    Exploring the Cosmic Frontier Astrophysical Instruments for the 21st Century Berlin, 18{21 May 2004 ABSTRACT BOOK Talks 3 Exploring the Cosmic Frontier: Astrophysical Instruments for the 21st Century Seesion I: Future Astrophysical Facilities Radio Facilities R. Ekers ATNF, Sydney, Australia Five decades ago, astronomers ¯nally broke free of the boundaries of light when a new science, radio astronomy, was born. This new way of 'seeing' rapidly uncovered a range of unexpected objects in the cosmos. This was our ¯rst view of the non-thermal universe, and our ¯rst unobscured view of the universe. In its short life, radio astronomy has had an unequalled record of discovery, including four Nobel prizes: Big-Bang radiation, neutron stars, aperture synthesis and gravitational radiation. New technologies now make it possible to construct new and upgraded radio wavelength arrays which will provide a powerful new generation of facilities. Radio telescopes such as the SKA together with the upgraded VLA will have orders of magnitude greater sensitivity than existing facilities. They will be able to study thermal and non-thermal emission from a wide range of astrophysical phenomena throughout the universe as well as greatly extending the range of unique science accessible at radio wavelengths. Millimeter, submillimeter and far-infrared astronomy facilities J. Cernicharo IEM, Madrid I will present the future observatories Herschel and ALMA and their capacities for the observation of the Universe in the wavelength range 60-3000 microns. From the solar system to the most distant galaxies, both instruments will allow to observe the cold gas and dust with an excellent frequency coverage and with very high angular resolution.
    [Show full text]
  • ASTRONET ERTRC Report
    Radio Astronomy in Europe: Up to, and beyond, 2025 A report by ASTRONET’s European Radio Telescope Review Committee ! 1!! ! ! ! ERTRC report: Final version – June 2015 ! ! ! ! ! ! 2!! ! ! ! Table of Contents List%of%figures%...................................................................................................................................................%7! List%of%tables%....................................................................................................................................................%8! Chapter%1:%Executive%Summary%...............................................................................................................%10! Chapter%2:%Introduction%.............................................................................................................................%13! 2.1%–%Background%and%method%............................................................................................................%13! 2.2%–%New%horizons%in%radio%astronomy%...........................................................................................%13! 2.3%–%Approach%and%mode%of%operation%...........................................................................................%14! 2.4%–%Organization%of%this%report%........................................................................................................%15! Chapter%3:%Review%of%major%European%radio%telescopes%................................................................%16! 3.1%–%Introduction%...................................................................................................................................%16!
    [Show full text]
  • ATNF Science Priorities
    ATNF Science Priorities Australia Telescope National Facility ATNF Science in 2010-2015: Part 1 – Overview Lewis Ball & Naomi McClure-Griffiths Robert Braun, Philip Edwards, Ilana Feain, George Hobbs, Simon Johnston Version 1.0: 8 August 2008 Context and purpose of this document Over the period 2010-2015 the ATNF needs to deliver ASKAP, a $100M new facility, restructure its operation to accommodate the new facility, maintain the world-class scientific productivity of its telescopes that are used by astronomers from around the world, and continue to implement Australia’s strategy of maximising Australian involvement in the SKA. This document comprises five parts, an overview of the expected scientific priorities that will be addressed by the full complement of ATNF telescopes and instrumentation, and four attempts to detail the science expected to be the priority for users of specific telescopes. It is important that the documents are treated as a set as they have been prepared as such and no one part is expected to stand alone. The priorities outlined in this document have been identified through a process of extensive user consultation. They draw on, for example, the millimetre white paper, the Science with Parkes forum on 5 November 2007, the Compact Array Science day in June 2008, and the extensive ASKAP science case. Ongoing feedback on the document and the priorities identified here are welcomed. This document is intended to be a part of the Future ATNF Operations plan. A preliminary version of that plan was written for the Australia Telescope Steering Committee in December 2007 and provided publicly soon after.
    [Show full text]
  • European VLBI Network Newsletter
    European VLBI Network Newsletter Number 37 January 2014 Contents Message from the Chairman of the EVN Board of Directors .................................................................................... 2 Call for the EVN Proposals: Deadline 1st February 2014 ........................................................................................... 3 EVN Science Highlights .............................................................................................................................................. 6 Radio Jets Clearing the Way Through a Galaxy: Watching Feedback in Action .................................................... 6 No evidence for a counter-jet in the TeV-emitting radio galaxy IC 310 ................................................................ 7 EVN/JIVE Technical Developments ............................................................................................................................ 8 A phase error related to the WSRT phased array.................................................................................................. 8 Phased array mode at the JIVE SFXC software correlator ................................................................................... 10 “OLD” EVN Scheduler’s Report: January 2014 ........................................................................................................ 11 Reports from EVN Institutes .................................................................................................................................... 14 JIVE 20
    [Show full text]
  • The Title of My
    Research in Astron. Astrophys. Vol.0 (20xx) No.0, 000–000 Research in http://www.raa-journal.org http://www.iop.org/journals/raa Astronomy and (LATEX: main.tex; printed on April 16, 2021; 0:09) Astrophysics East Asian VLBI Network Observations of Active Galactic Nuclei Jets: Imaging with KaVA+Tianma+Nanshan Yu-Zhu Cui1;2, Kazuhiro Hada1;2, Motoki Kino2;3, Bong-Won Sohn4;5;6, Jongho Park7, Hyun-Wook Ro4;5, Satoko Sawada-Satoh8, Wu Jiang9;11, Lang Cui10;11, Mareki Honma1;2;12, Zhi-Qiang Shen9;11, Fumie Tazaki2, Tao An9;11, Ilje Cho4;6, Guang-Yao Zhao4;13, Xiao-Peng Cheng4, Kotaro Niinuma8, Kiyoaki Wajima4, Ying-Kang Zhang9, Noriyuki Kawaguchi2, Juan-Carlos Algaba14, Shoko Koyama7, Tomoya Hirota2, Yoshinori Yonekura15, Nobuyuki Sakai4, Bo Xia9, Yong-Bin Jiang9, Lin-Feng Yu9, Wei Gou9, Ju-Yeon Hwang4, Yong-Chen Jiang9, Yun-Xia Sun9, Dong-Kyu Jung4, Hyo-Ryoung Kim4, Jeong-Sook Kim4, Hideyuki Kobayashi2, Jee-Won Lee4, Jeong-Ae Lee16, Hua Zhang10, Guang-Hui Li10, Zhi-Qiang Xu9, Peng Li10, Jung-Hwan Oh4, Se-Jin Oh4, Chung-Sik Oh4, Tomoaki Oyama2, Duk-Gyoo Roh4, Katsunori-M. Shibata2, Wen Guo9, Rong-Bing Zhao9, Wei-Ye Zhong9, Jin-Qing Wang9, Wen-Jun Yang10, Hao Yan10, Jae-Hwan Yeom4, Bin Li9, Xiao-Fei Li10, Jian-Ping Yuan10, Jian Dong9, Zhong Chen9, Kazunori Akiyama17;18, Keiichi Asada7, Do-Young Byun4, Yoshiaki Hagiwara19, Jeffrey Hodgson4, Tae-Hyun Jung4, Kee-Tae Kim4, Sang-Sung Lee4, Kunwoo Yi20, Qing-Hui Liu9, Xiang Liu10;11, Ru-Sen Lu9;11;21, Masanori Nakamura7, Sascha Trippe20, Na Wang10;11, Xue-Zheng Wang9 and Bo Zhang9 1 Department of
    [Show full text]
  • European VLBI Network Newsletter
    European VLBI Network Newsletter Number 39 September 2014 Contents Message from the Chairman of the EVN Board of Directors .................................................................................... 2 Call for the EVN Proposals ......................................................................................................................................... 3 EVN Science Highlights .............................................................................................................................................. 7 EVN and eMERLIN constrain the progenitor system and environs of SN 2014J ................................................... 7 EVN/JIVE Technical Developments ............................................................................................................................ 9 The JIVE UniBoard Correlator (JUC) ....................................................................................................................... 9 EVN Scheduler’s Report ........................................................................................................................................... 12 Reports from EVN Institutes .................................................................................................................................... 13 The New Radome for the Onsala Space Observatory 20-m Telescope ............................................................... 13 Recent and upcoming meetings .............................................................................................................................
    [Show full text]
  • The EVN and JIVE
    What can VLBI do for your research? The EVN and JIVE By Dr. Francisco Colomer Joint Institute for VLBI ERIC (JIVE) EAVN @ PyonengChang (Korea), 4-7 September 2018 Contents • The European VLBI Network (EVN) • The Joint Institute for VLBI ERIC (JIVE) • (A few) recent VLBI results • Globalization of VLBI See also poster: European VLBI Network (EVN) • The EVN is a network of radio telescopes located primarily in Europe and Asia, with additional antennas in South Africa and Puerto Rico, which performs high angular resolution observations of cosmic radio sources. • It is the most sensitive VLBI array in the world, thanks to the collection of extremely large telescopes that contribute to the network, operating from 1.4 GHz to 45 GHz (some up to 90 GHz), also in real-time (eEVN). • The Joint Institute for VLBI ERIC (JIVE) correlates the EVN data and provides expert support to EVN users. http://www.evlbi.org/ www.jive.eu / www.evlbi.org EVN+MERLIN Joint Institute for VLBI ERIC (JIVE) An European Research Infrastructure Consortium (ERIC) • 6 partner countries: NL (host), FR, ES, UK, SE, LV • 4 associated institutions: INAF (IT), NRF (SA), MPIfR (DE), NAOC (Cn) JIVE and EVN Joint Institute for VLBI ERIC (JIVE) • Supports the European VLBI Network • Radio astronomy at its highest angular resolution • operations JIVE headquarters in Dwingeloo, the Netherlands. • correlation • data acquisition • Research & development • Software (CASA 5.3) • hardware • e-VLBI • Science support • Training http://www.jive.eu/ EVN/JIVE collaboration with KVN First e-VLBI fringes between EVN antennas (Onsala and Yebes) and two stations of the Korean VLBI Network at 22 GHz.
    [Show full text]
  • CASA News Issue 6 • 9 May 2018
    CASA News Issue 6 • 9 May 2018 From the Lead Ryan Raba I would like to start by saying thanks to everyone within the Common Astronomy Software Applications (CASA) development team and user community for welcoming me to this position and showing patience as I come up to speed on the complexity of this environment. CASA is a fascinating collection of utilities born from decades of hard work and deep scientific understanding. It is a profoundly challenging and rewarding endeavor to chart a course for something that is both science research and an operational program, a support library as well as a standalone application, used extensively by two observatories with different cultures and work processes, and included in a wide variety of external projects from around the world. The CASA team has had a busy six months since the last newsletter. We hosted the annual CASA Users Committee gathering to discuss progress and feedback on a variety of topics relevant to the user community. We also conducted an internal CASA Development Face-to-Face meeting bringing together developers from the National Radio Astronomy Observatory (NRAO), the European Organisation for Astronomical Research in the Southern Hemisphere (ESO), and the National Astronomical Observatory of Japan (NAOJ) for the first time in several years. We are wrapping up the CASA 5.3 release with several new features in imaging and calibration and making very good progress on a High Performance Computing (HPC) multi-core parallelization mode for CASA processing. Support for the Very Large Array (VLA) Sky Survey, as well as upcoming Atacama Large Millimeter/submillimeter Array (ALMA) needs for Cycle 6, continue to drive testing, bug fixing, and an overall increase in maturity.
    [Show full text]
  • The Program & Abstracts Book
    EVN Symposium & Users Meeting 2016 Table of Contents 2 Welcome .............................................................................................................................. 3 General Information .......................................................................................................... 4 Schedule of Events ............................................................................................................. 8 List of Posters ...................................................................................................................17 AGN Surveys .....................................................................................................................22 Astrometry, Geodesy and Space Applications ...............................................................27 VLBI Technology ...............................................................................................................32 Starbursts and Extragalactic Masers ..............................................................................34 VLBI Arrays ........................................................................................................................38 Stellar Evolution and Stellar Masers ...............................................................................39 Transients ..........................................................................................................................43 AGN Astrophysical Processes ..........................................................................................45
    [Show full text]
  • East Asia VLBI Network Observations of the Tev Gamma-Ray Burst
    East Asia VLBI Network observations of the TeV Gamma-Ray Burst 190114C Tao An1,2*, Om Sharan Salafia3, Yingkang Zhang1,2, Giancarlo Ghirlanda3, Giovannini Giovannini4,5, Marcello Giroletti4, Kazuhiro Hada6, Giulia Migliori4,5, Monica Orienti4, and Bong Won Sohn7 1Shanghai Astronomical Observatory, Key laboratory of Radio Astronomy, CAS, Nandan Road 80, Shanghai 200030, China; 2University of Chinese Academy of Sciences, 19A Yuquan Road, Shijingshan District, 100049 Beijing, China; 3Istituto Nazionale di Astrofisica, Osservatorio Astronomico di Brera, Via E. Bianchi 46, I-23807 Merate, Italy; 4Istituto Nazionale di Astrofisica, Istituto di Radioastronomia, via Gobetti 101, I40129, Bologna, Italia; 5University of Bologna, Department of Physics and Astronomy, via Gobetti 83, I40129, Bologna, Italia; 6National Astronomical Observatory of Japan, 2-21-1 Osawa, Mitaka, Tokyo 181-8588, Japan; 7Korea Astronomy and Space Science Institute, Yuseong-gu, Daejeon 34055, Korea Received , 2019; accepted , 2019 Observations of gamma-ray bursts (GRBs) at Very High Energy (VHE) offer a unique opportunity to investigate particle accelera- tion processes, magnetic fields and radiation fields in these events. Very Long Baseline Interferometry (VLBI) observations have been proven to be a powerful tool providing unique information on the source size of the GRBs at mas scales, as well as their accurate positions and possible expansion speeds. This paper reports on the follow-up observations of GRB 190114C, the first ever GRB detected with high significance at TeV photon energies by the MAGIC telescope, conducted with the East Asia VLBI Network (EAVN) at 22 GHz on three epochs, corresponding to 6, 15 and 32 days after the burst. The derived maps do not show any significant source above 5σ.
    [Show full text]