Square Kilometre Array Computational Challenges Paul Alexander Paul Alexander SKA Computational Challenges What is the Square Kilometre Array (SKA) • Next Generation radio telescope – compared to best current instruments it is ... E-MERLIN • ~100 times sensitivity • ~ 106 times faster imaging the sky • More than 5 square km of collecting area on sizes 3000km eVLA 27 27m dishes Longest baseline 30km GMRT 30 45m dishes Longest baseline 35 km Paul Alexander SKA Computational Challenges What is the Square Kilometre Array (SKA) • Next Generation radio telescope – compared to best current instruments it is ... • ~100 times sensitivity • ~ 106 times faster imaging the sky • More than 5 square km of collecting area on sizes 3000km • Will address some of the key problems of astrophysics and cosmology (and physics) • Builds on techniques developed in Cambridge • It is an interferometer • Uses innovative technologies... • Major ICT project • Need performance at low unit cost Paul Alexander SKA Computational Challenges Dishes Paul Alexander SKA Computational Challenges Phased Aperture array Paul Alexander SKA Computational Challenges also a Continental sized Radio Telescope • Need a radio-quiet site • Very low population density • Large amount of space • Possible sites (decision 2012) • Western Australia • Karoo Desert RSA Paul Alexander SKA Computational Challenges Sensitivity comparison 12,000 Sensitivity Comparison 10,000 1 - K 2 8,000 SKA2 6,000 SKA2 SKA1 MeerKAT LOFAR ASKAP 4,000 Sensitivity: Aeff/Tsys m Sensitivity:Aeff/Tsys eVLA SKA1 2,000 EVLA LOFAR 0 10 100 1,000 10,000 100,000 Frequency MHz Paul Alexander SKA Computational Challenges SKA2 ~ 250 Dense Aperture Array Stations 300-1400MHz ~ 2700 Dishes 3-Core Central Region ~250 Sparse Aperture Array Stations 70-450MHz Artist renditions from Swinburne Astronomy Productions Paul Alexander SKA Computational Challenges SKA1 ~ 300 Dishes 2-Core Central Region ~50 Sparse Aperture Array Stations 70-450MHz Artist renditions from Swinburne Astronomy Productions Paul Alexander SKA Computational Challenges SKA Timeline 2019 Operations SKA1 2024: Operations SKA2 2019-2023 Construction of Full SKA, SKA2 €1.5B 2016-2019 10% SKA construction, SKA1 €300M 2012 Site selection 2012 - 2016 Pre-Construction: 1 yr Detailed design €90M PEP 3 yr Production Readiness 2008 - 2012 System design and refinement of specification 2000 - 2007 Initial concepts stage 1995 - 2000 Preliminary ideas and R&D Paul Alexander SKA Computational Challenges Work Packages in the PEP 1. Management 2. System 3. Science SPO 4. Maintenance and support /Operations Plan 5. Site preparation 6. Dishes 7. Aperture arrays 8. Signal transport & networks Work Package 9. Signal processing Contractors 10. Science data processor 11. Telescope manager 12. Power Paul Alexander SKA Computational Challenges Work Packages in the PEP 1. Management 2. System 3. Science SPO 4. Maintenance and support /Operations Plan 5. Site preparation 6. Dishes 7. Aperture arrays 8. Signal transport & networks Work Package 9. Signal processing Contractors UK (lead), AU (CSIRO…), NL (ASTRON…) 10. Science data processor South Africa SKA, Industry (Intel, IBM…) 11. Telescope manager 12. Power Paul Alexander SKA Computational Challenges Very Brief Introduction to Radio Astronomy Imaging Paul Alexander SKA Computational Challenges Standard interferometer s Astronomical signal (EM wave) B . s • Visibility: V(B) = E1 E2* Detect & = I(s) exp( i w B.s/c ) amplify • Resolution determined by 1 B 2 Digitise maximum baseline & delay qmax ~ l / Bmax Correlate • Field of View (FoV) X X X X X X determined by the size of Integrate each dish visibilities qdish ~ l / D Process Calibrate, grid, FFT SKY Image Paul Alexander SKA Computational Challenges Aperture arrays Digitise, delay & beam form Beam form: Correlate • Apply delay gradients to point X X X X X X electrically • Multiple delay gradients many beams and large FoV Process Calibrate, grid, FFT SKY Image Paul Alexander SKA Computational Challenges Aperture arrays Aperture-Array station ~25000 phased elements Equivalent to one dish These are then Digitise, cross-correlated delay & beam form Beam form: Correlate • Apply delay gradients to point X X X X X X electrically • Multiple delay gradients many beams and large FoV Process Calibrate, grid, FFT SKY Image Paul Alexander SKA Computational Challenges Formulation • What we measure from a pair of telescopes: • In practice we have to deal with this equation, but for simplicity consider a scalar model m • The delays allow us to follow a point on the sky • The Ji are direction dependent Jones matrices which include the effects of: l propagation from the sky through the atmosphere s scattering coupling to the antenna/detector gain Paul Alexander SKA Computational Challenges Formulation • Where we include time delays to follow a central point. In terms of direction cosines relative to the point we follow and for a b = (u,v,w) • If we can calibrate our system we can apply our telescope-dependent calibrations and then for small FoV m we can approximate l • And the measured data are just samples of this function s • In this case we can estimate the sky via Fourier inversion and deconvolution Paul Alexander SKA Computational Challenges Example inner and intermediate layout. 180 dishes on each spiral arm. 16 AAhi and 16Formulation Aalo stations on each spiral arm (not shown). Dish core 1500 dishes 5km diameter. AAhi and Aalo core 167 stations, 5km diameter. • Sampling of the Fourier -10 -8 -6 -4 -2 0 2 4 6 8 10 plane is determined by the 10 positioning of the antennas 8 and improved by the rotation of the earth 6 4 W e 2 0 m -2 l -4 -6 s -8 -10 AAhi core AAlo core Dish core Dishes mid range Paul Alexander SKA Computational Challenges The Science Aims and the Imaging Challenges Paul Alexander SKA Computational Challenges SKA Key Science Drivers ORIGINS Neutral hydrogen in the universe from the Epoch of Re-ionisation to now When did the first stars and galaxies form? How did galaxies evolve? Role of Active Galactic Nuclei Dark Energy, Dark Matter Cradle of Life FUNDAMENTAL FORCES Pulsars, General Relativity & gravitational waves Science with the Square Origin & evolution of cosmic magnetism Kilometre Array (2004, eds. C. Carilli & S. Rawlings, New Astron. Rev., 48) TRANSIENTS (NEW PHENOMENA) Paul Alexander SKA Computational Challenges Galaxy Evolution back to z~10? HDF VLA ~ 3000 galaxies ~15 radio sources Paul Alexander SKA Computational Challenges Galaxy Evolution back to z~10? HDF SKA Paul Alexander SKA Computational Challenges The Imaging Challenge This illustrates one of our main challenges • To make effective use of the improved sensitivity we face an immediate problem • Typically within the field of view of the telescope the noise level will be ~106 – 107 times less than the peak brightness • We have to achieve sufficiently good calibration and image fidelity to routinely achieve a “dyanamic range” of > 107:1 • With very hard work now we can just get to 106:1 in some fields Paul Alexander SKA Computational Challenges The Processing Challenge Paul Alexander SKA Computational Challenges SKA2 wide area data flow Aperture Array 0.4-1.4 GHz Station Wide FoV Central Processing Facility - CPF 16 Tb/sTo 250 AA Stations 4 PbCorrelator/s UV Processors Image formation Archive Tile & Dense AA Buffer Processor Buffer Processor Station ... AA slice Buffer Processor Processing Buffer Processor Buffer Processor Buffer Processor ProcessorsImaging ... AA slice Data 16 Tb/s Buffer Processor ... ... Buffer Processor Data ... Buffer Processor Archive Buffer Processor Buffer Processor Data switch Data Buffer Processor ... AA slice .. Buffer Processor Buffer Processor Dish & AA+Dish Correlation & AA+Dish Dish Buffer Processor Optical Buffer Processor ... .. Data Buffer Processor ...... Time links Processors Control Science 70-450 MHz Sparse AA Wide FoV ... Buffer Processor Buffer Processor Buffer Processor 20 Gb/s Buffer Processor DSP Buffer Processor 1.2-10 GHz 24Pb/s Tb/s Tb/s Gb/s Gb/s WB-Single Pixel feeds Control Processors & User interface 10 Gb/s DSP 20 Gb/s Time Standard 15m Dishes ... To 1200 Dishes User interface via Internet Paul Alexander SKA Computational Challenges SKA1 Data rates from receptors • Dishes – Depends on feeds, but illustrate by 2 GHz bandwidth at 8-bits – G = 64 Gb/s from each dish • For Phased Array feeds increased by number of beams (~20) – G ~ 1 Tb/s • For Low frequency Aperture Arrays : – Bandwidth is 380 MHz – Driven by the requirements of Field of View from the science requirements which from DRM is 5 sq-degrees 20 beams – G = 240 Gb/s • These are from each collector into the correlator or beam former – 300 dishes – 285 75-m AA stations – G(total) ~ 68 Tb/s Paul Alexander SKA Computational Challenges Data Rates • After correlation the data rate is fixed by straightforward considerations Must sample fast enough (limit on integration time) dt Baseline B/l UV (Fourier) cell size D/l max B/l – B/(l+dl) Must have small-enough channel width to avoid chromatic aberration max Wdt Paul Alexander SKA Computational Challenges Data rates from the correlator • Standard results for integration/dump time and channel width • Data rate then given by # antennas # polarizations # beams word-length • Can reduce this using baseline-dependent integration times and channel widths Paul Alexander SKA Computational Challenges Example correlator data rates and products SKA1 • Aperture Array Line experiment (e.g. EoR) • 5 sq degrees; 170000 channels over 250 MHz bandwidth ~ 30 GB/s reducing quickly to ~ 1GB/s Up to 500 TB UV (Fourier) data; Images (3D) ~ 1.5 TB • Continuum experiment with long baselines with the AA • 100 km baseline with the low-frequency AA 1.2 TB/s reducing to ~ 12.5 GB/s Up to 250 TB/day to archive if we archive raw UV data • Spectral-line imaging with dishes Data rates ~ 50 GB/s; Images (3D) ~ 27 TB Paul Alexander SKA Computational Challenges Example beam-formed data rates SKA1 • Pulsar search • Galactic-plane survey for pulsars ~ 400 GB/s to de-disperser (hardware?) Data products are of small volume as all analysis is done in pseudo real-time.