A Scalable Packetised Radio Astronomy Imager by Jason Ryan Manley Thesis presented for the degree of DOCTOR OF PHILOSOPHY in the Department of Electrical Engineering, Faculty of Engineering and the Built Environment at the University of Cape Town February 2014 Supervisor: Professor M Inggs University of Cape Town The copyright of this thesis vests in the author. No quotation from it or information derived from it is to be published without full acknowledgement of the source. The thesis is to be used for private study or non- commercial research purposes only. Published by the University of Cape Town (UCT) in terms of the non-exclusive license granted to UCT by the author. University of Cape Town Abstract Modern radio astronomy telescopes the world over require digital back-ends. The complexity of these systems depends on many site-specific factors, in- cluding the number of antennas, beams and frequency channels and the bandwidth to be processed. With the increasing popularity for ever larger interferometric arrays, the processing requirements for these back-ends have increased significantly. While the techniques for building these back-ends are well understood, every installation typically still takes many years to develop as the instruments use highly specialised, custom hardware in order to cope with the demanding engineering requirements. Modern technology has enabled reprogrammable FPGA-based process- ing boards, together with packet-based switching techniques, to perform all the digital signal processing requirements of a modern radio telescope array. The various instruments used by radio telescopes are functionally very dif- ferent, but the component operations remain remarkably similar and many share core functionalities. Generic processing platforms are thus able to share signal processing libraries and can acquire different personalities to perform different functions simply by reprogramming them and rerouting the data appropriately. Furthermore, Ethernet-based packet-switched networks are highly flexible and scalable, enabling the same instrument design to be scaled to larger installations simply by adding additional processing nodes and larger network switches. The ability of a packetised network to transfer data to arbitrary processing nodes, along with these nodes' reconfigurability, allows for unrestrained partitioning of designs and resource allocation. This thesis describes the design and construction of the first working radio astronomy imaging instrument hosted on Ethernet-interconnected re- programmable FPGA hardware. I attempt to establish an optimal packetised architecture for the most popular instruments with particular attention to the core array functions of correlation and beamforming. Emphasis is placed on requirements for South Africa's MeerKAT array. A demonstration system is constructed and deployed on the KAT-7 array, MeerKAT's prototype. This research promises reduced instrument development time, lower costs, improved reliability and closer collaboration between telescope design teams. Acknowledgements I would like to thank the following people for their assistance and support: Dan Werthimer for being a lighthouse in the fog of instrument specifi- cations. Your insight and guidance is legendary. Also, thank you for the opportunity to work with the energetic CASPER team and to your family, Mary-Kate and Willy, for hosting me in Berkeley. The late Don Backer for always making time in his busy schedule to advise me and for providing me with the opportunity to participate in the PAPER experiment. I will forever remember him fondly. Aaron Parsons for the groundwork of the CASPER correlator and his insights and novel solutions to engineering problems. The rest of the CASPER, RAL and BWRC teams at UC Berkeley for providing much of the infrastructure that made this project possible and for keeping me entertained during my nearly two years at BWRC. Thank you to the talented and dedicated DBE engineering team at SKA- SA: Francois Kapp for making the ROACH board a reality and for his en- couraging leadership of the team. Andrew Martens and Paul Prozesky for their help with the KAT-7 F-engines and beamformer implementations and for assisting with software when I was under time pressure. Mark Welz for his unending patience with me and my Linux questions. Ruby van Rooyen for her help with quantisation and PFB analysis simulations and, together with Alec Rust, for performing the KAT-7 ATP. Sias Malan for his willingness to help with journal papers and ongoing enthusiasm for the project. Dave George for the underlying ROACH infrastructure and ROACH-2. Etienne Bauermeister for the KATADC and MeerKAT high speed digital samplers. Adam, Henno, Luzuko, Phil, Shanly, SimonR, SimonX, Wesley, Vaughn and the others for always lending a helping hand when it was needed. SKA-SA's System Engineers, Adriaan, Paul and Warnich, thank you for your help in specifying the requirements for the KAT-7 and MeerKAT systems. Alan Langman for introducing me to the SKA-SA project. To my family: thank you for their unending support and encouragement and especially to my mother for all the proofreading! To my supervisor, Professor Michael Inggs, for his faith in me and the free reign to complete this project in my own time. Declaration This document and all of its contents represent my own work unless otherwise stated. I acknowledge that all contributions made by others have been cited and referenced. I have not, and will never allow this work to be copied by anyone with the intention of submitting it as their own work. Furthermore, I acknowledge that plagiarism is wrong and declare that this project represents my own work. Jason Manley 13th of February 2014 Glossary ADC Analogue to Digital Converter. ALMA The Atacama Large Millimeter Array located in Chile. AIPY Astronomical Interferometry in PYthon; An imaging package by Aaron Parsons, used primarily by PAPER. ASIAA The Academia Sinica Institute of Astronomy and Astrophysics in Taiwan. ASIC Application-Specific Integrated Circuit; a piece of electronic circuitry designed for a single purpose and contained within a single chip. ASKAP The Australian Square Kilometer Array Pathfinder located in the Mid-west region of Western Australia. ATA The Allen Telescope Array in Hat Creek, United States of America. ATP Acceptance Test Procedure. In System Engineering, a list of instruc- tions for executing an Acceptance Test, to verify the correct operation of a component, to ensure that it meets its specification. This test is typically run on each device off a production line before being placed into service. Backplane A backplane interconnect is one that uses a printed circuit board (PCB) to connect multiple other PCBs (as opposed to cables). For example, multiple processing cards that need to communicate with one another can slot into a single larger backplane, rather than multiple cables connecting to- and from- each one. v ATP Acceptance Test Procedure. A System Engineering term for a docu- ment that describes test routines to be run during an acceptance test. See also ATR. ATR Acceptance Test Report. A System Engineering term for a document containing the results of an acceptance test. See also ATP. Baseline Refers to a pair of antennas in an array. BEE2 The second-generation Berkeley Emulation Engine. An FPGA-based hardware computing platform. BRAM Block Random Access Memory; the FPGA's onchip memory. BWRC The Berkeley Wireless Research Centre, a UCB venture together with a number of industry partners. CARMA The Combined Array for Research in Millimeter-wave Astronomy in Cedar Flat, California, United States of America. CASPER The Center for Astronomy Signal Processing and Electronics Re- search, a research group at the University of California, Berkeley. CfA Centre for Astrophysics, a collaboration of Harvard College Observa- tory (HCO) and Smithsonian Astrophysical Observatory (SAO) and home to Harvard's Department of Astronomy. Corner-turn A matrix transpose operation. CPLD A Complex Programmable Logic Device, a device typically used to interconnect other onboard devices with simple combinatorial and se- quential logic. Typically less complex than an FPGA with reduced functionality. ROACH's CPLD interconnects the PPC, FPGA and SD-card busses amongst others. CW Continuous wave signal (a sine or co-sine tone). DBE Digital Back-End. The real-time digital processing system of a radio receiver. vi DDC Digital Down Converter. The digital equivalent of an analogue het- erodyne mixer and filter. DDR Double Data Rate. The ability to transfer data on the leading and the falling edges of a clock cycle. EVLA The Expanded Very Large Array, New Mexico, United States of America. FASR The Frequency Agile Solar Radiotelescope at the Owens Valley Radio Observatory in California, United States of America. FIR Finite Impulse Response; usually a reference to a digital filter imple- mentation who's response to a signal of finite duration is also finite, before returning to zero. As opposed to IIR. FPGA Field Programmable Gate Array; A reprogrammable device able to perform boolean logical and mathematical operations. FX Correlator A correlator who's order of operation is to first perform data channellisation (typically through the use of an FFT) followed by the multiplication operation (see also XF correlator). GMRT The Giant Metre-wave Radio Telescope north of Pune in India. IBOB The Internet Break-out Board; An FPGA-based processing board designed by CASPER. It was originally designed to digitise data and retransmit it into an Ethernet network for processing by BEE2s and thus has limited compute ability. IETF The Internet Engineering Task Force, responsible for a number of Internet protocol standards definitions. IIR filter Infinite Impulse Response; a filter design that employs internal feedback such that its response to a finite input signal may continue to be non-zero infinitely. As opposed to FIR. KAT-7 The first phase of the MeerKAT telescope. This initial deployment, comprising 7 dishes, was completed in 2010. See also MeerKAT vii LEDA The Large Aperture Experiment to Detect the Dark Ages, a project currently utilising the LWA.