The Square Kilometre Array

John Bunton | CSP System Engineer 6-8 March 2013

CSIRO ASTRONOMY AND SPACE SCIENCE SKA1 Low antenna station (Australia) SKA1 MID antennas (South Africa)

Square Kilometre Array

SKA1 Overview

SKA1-low stations include Station Beamformer

CSP includes Correlator, Tied Array beamformers, Pulsar Search Engine and I work here Pulsar Timing Engine My work

The Low Correlator and Beamformer (CSP_Low.CBF) consortium are part of the larger Central Signal Processing (CSP) Consortium CSP is lead by NRC Canada with MDA managing

CSP_Low.CBF is lead by CSIRO (Australia) With ASTRON (Netherlands) and AUT (New Zealand) as collaborator SKA Science

Square Kilometre Array Rainer Beck

Square Kilometre Array Radio Telescope Sensitivity

Square Kilometre Array Ideal Antenna and Receiver

-270C =3K

27C =300K

Power=1 Power=100 Output power proportional to Temperature

Square Kilometre Array A real system

A real receiver has internal noise Model as a resistor with an equivalent temperature Tr

Total power Tsky + Tr Want to minimise T r Tsky A good system Tr=20K Signal = Tsky + Tr

Tr

Square Kilometre Array Power in a resistor

Power = kTB  k Boltzmann’s Constant = 1.38 x10-23 Joules/Kelvin  T Temperature of the resistor (20 Kelvin)  B Bandwidth For 1Hz bandwidth, 20K system temperature  Power at receiver input approx. 2.8 x 10-22 W For a 1 MHz bandwidth its 2.8 x 10-16 W

 A 1W transmitter at 100 km is ~10-11 W/m2

Square Kilometre Array The Jansky

15 m antenna, area 177 m2, and after normalising for bandwidth Equivalent incident power = 1.6 x 10-24 W/m2Hz

This is an inconveniently small quantity So Astronomers invented the Jansky 1 Jansky = 10-26 W/m2Hz

Our antenna needs a 160 Jansky source for the power to double at the output. Or a 1 nanoW transmitter at 100 km with at a 1MHz bandwidth

Square Kilometre Array Antenna Signal to Noise

A strong source (radio star) is 10 Jansky  Signal to Noise Ratio 0.06, -12dB (note our signal is less than unwanted noise) In the 80s I worked for the Fleurs Radio Telescope just outside Sydney  It could detect sources of ~1 mJansky  SNR 0.000 006 = -52dB ASKAP can reach ~10 μJansky, SNR = -72dB (10-7.2) SKA less than 1 μJansky, SNR less than -82dB (~1:1billion)

Square Kilometre Array Interferometry

How the SKA work

Square Kilometre Array The Interferometer

The SKA uses interferometers Signals from pairs of antennas are Correlated A correlations is a multiply and accumulate.  We average to improve signal to q noise ratio The conjugate * cancels phase offset u (wavelengths) Signals from star are in-phase X * ++

Square Kilometre Array Drunken Walk

Consider a drunk who is trying to get home. He cab stagger from one lamp post to the next. But when he gets there he can’t remember which way to go. So each time he sets of in a random direction After N lamp posts how far, on average, has he gone  Answer √N After 100 lamp post he has gone about 10

Square Kilometre Array The Correlator

The signal from each antenna has two component  The sky signal (wanted)  Antenna (System) noise (unwanted) The system noise is randomly in-phase and out-of-phase. The signal adds in the same manner as a drunken walk - grows as √N The sky signal is maintained in phase – grows proportional to N Improvement ~√N X * ++

Square Kilometre Array How good does it get

For the SKA bandwidths as much as 5 GHz  5 x 109 samples per second Observations can be more than 24 hours  105 seconds Number of sample = Bτ  B Bandwidth, τ Time = 5 x 1014 SNR improvement √Bτ ~ 2 x 107 (73dB)  For a single pair of antennas.

SKA has 20,000 (Mid) and 256,000 (Low) pairs of antennas

Square Kilometre Array Warning: Complex Arithmetic coming up

Phase and phase difference are important

Product of two real sinusoids cos(ωt) cos(θ ) = (cos(ωt + θ) + cos(ωt - θ))/2 Yuk!

Complex sinusoids ejωt = cos(ωt) + j sin(ωt) j= √-1 think real in-phase, imaginary (j term) quadrature phase Now product of two complex sinusoids ejωt ejθ = ej (ωt + θ) Easy

Square Kilometre Array Interferometry 101

Star brightness B at position l = cos(q) (direction cosine)

) Distance between two antennas u q s( o c in wavelengts u

q Extra path length to one antenna of D= u cos(q) = ul u (wavelengths) j2πul Adds a phase term of e Complex Correlator output  Be j2 ul

Square Kilometre Array Interferometry 101

output  Be j2 ul Consider real part of output

= Bcos(2πul) q Cosine variation as θ varies

 l = cos(q) and change in θ small u (wavelengths) Interferometer is sensitive to a X * ++ Dump Sum once a second “spatial” frequency  A component of the sky image with a sinusoidal variation across the sky q Period proportional to1/u

Square Kilometre Array Interferometry 101

output  Be j2 ul Summing over all l we find output(u)   B(l)e j2 uldl If we measure at all separation we measure all “spatial” frequencies q We have measured the spectrum of the image u (wavelengths) Measure different “frequencies” by X * having antennas at different spacings ++ Dump Sum once a second Fourier transform “spectrum” to get image q Period proportional to1/u

Square Kilometre Array Synthesis Mapping

Correlation between pairs of antennas generate spatial frequency (Fourier) components of the image N antennas gives N(N-1)/2 spacings (usually incomplete) Transform to give (dirty) image (further processing needed)

Antenna location Fourier Components Image of Point Source

400 1 500 200 400 0.5 0 300

-200 200 0 40 40 300 400 500 600 700 20 -200 0 200 400 20 0 0

Square Kilometre Array Musical Analogy

Music score defines the notes (frequencies) and their position (time) Fourier components defines spatial frequency and also a position (in Fourier Components this case and angle) 400

Need and orchestra to process the 200 score into music 0 Need a complex computer program to convert Fourier component to an -200 image -200 0 200 400

Square Kilometre Array Spectral Response

Many radio astronomy signal are due to atomic transition – Neutral Hydrogen at 1.42 GHz, OH masers at 1.665 GHz… Want spectrum of signal – Colour image, not just black and white EITHER Measure correlation with different delays – Cross Correlation function and Fourier Transform – XF correlator OR Split signal into subbands and measure correlation in each – Cross Power Spectrum - FX correlator

Antenna 1 Antenna 2 Antenna 1 Antenna 2

S S S S

h h h h

i i i i

D D D D D D f f f f

t t t t

R Frequency R R Frequency R

e e e e

g g g g

X X X X X X X i Transform i i Transform i

s s s s

t t t t

e e e e

nD r r r r x A A A A A A A Accumulator Note: no change in Shift Register + sample rate

Square Kilometre Array Which to Use for the SKA

XF – complex multiplies per input sample = No. Channels ~104 FX – complex multiplies per input sample = 1  Plus overhead for frequency transform ~12 per complex antenna per sample  But number of correlations is N(N-1)/2 where N is number of antennas FX computation more efficient when  No. Channels > 1 + 24/N

Antenna 1 Antenna 2 Antenna 1 Antenna 2

S S S S

h h h h

i i i i

D D D D D D f f f f

t t t t

R Frequency R R Frequency R

e e e e

g g g g

X X X X X X X i Transform i i Transform i

s s s s

t t t t

e e e e

nD r r r r x A A A A A A A Accumulator Note: no change in Shift Register + sample rate XF FX

Square Kilometre Array Polarisation and compute load

Each antenna generates signal for two polarisations – X,Y linear or left and right circular * * * * Must form all combination X1 X2 , Y1 Y2 , X1 Y2 , Y1 X2 , – Stokes parameters Compute load for a single pair of antennas  4 multiplies and 4 adds per complex multiply  4 complex multiplies per data sample  1 data sample per Hz (complex data)  32 Gflops per GHz of bandwidth

For correlation (excluding image processing, beamforming etc) • SKA Low 1.26 Pflops • SKA Mid 2.95 Pflops

Square Kilometre Array Filterbanks

From 1GHz to 1kHz

Square Kilometre Array Generalised Filterbank

I/Q mixer on each channel, selects band centre

Filter to required bandwidth (impulse response ho(n), ho(t) analogue) Down sample to required sample rate Low Pass filter IQ Mixer

Down convert

Cos(ωt) or ADC if mixer and filter analogue Sin(ωt)

Square Kilometre Array Single output from filterbank

Multiplication by complex sinusoid and impulse response can be interchanged. Let w=e-jω

0.n ho(n) w

Xo Sum

w1.n

X1 Sum x(n)

m.n ho(n) w

XN-1 Sum

Square Kilometre Array Single output, simplified

Need to do multiplication of input by ho(n) once

Take block of input data and value by value multiply x(n).ho(n) On the right is a Fourier Transform Choose w and m appropriately  Have a Fast Fourier Transform

Fourier Transform

Pre Multiply

Square Kilometre Array Decimating the FFT

On 2GS/s data a 1MHz bandwidth filter and ho(n) ~20,000 samples long. Therefore the channel spacing is 0.1MHz We only want to keep every tenth output from the FFT. The rest can be discarded. So why calculate them at all? Have N 1 X(k)  h(n).x(n).e j(2 / N)nk n0 Want ever rth output: N 1 X(k')  h(n).x(n).e j(2r / N)nk' n0

Square Kilometre Array Apologies for the Maths

If the length of the original impulse response and FFT is rM then

r1 M 1 X(k')    h(n  mM ).x(n  mM ).e j(2r / N )( nmM )k' m0 n0 r1 M 1    h(n  mM ).x(n  mM ).e j(2r / N )nk'e j2 .m.k' m0 n0 r1 M 1    h(n  mM ).x(n  mM ).e j(2r / M )nk' m0 n0 Interchanging the order of summation gives (ALMA memo 447) r1 M 1 X(k')    h(n  mM ).x(n  mM ).e j(2 / M )nk' m0 n0 Reduced size (one rth) M 1 r1 Fourier transform    j(2 / M )nk'    h(n  mM ).x(n  mM ).e n0 m0 

Square Kilometre Array Example r = 2

Consider an 8 point Fourier Transform and only every second output wanted Delete every second row. What is left is two 4-point Fourier Transforms Sum the data before the transform and do one Transform

1 1 1 1 1 1 1 1  x0   f0   1 2 3 4 5 6 7  1 w w w w w w w  x    Note shaded   1     1 w2 w4 w6 w0 w2 w4 w6  x2   f2  blocks are the  3 6 9 12 15 18 21     1 w w w w w w w  x3       same after  4 0 4 0 4 0 4 x   f  1 w w w w w w w  4 4 5 10 15 20 25 30 35 x     deletions 1 w w w w w w w  5      1 w6 w4 w2 w0 w6 w4 w2 x6   f6       7 14 21 28 35 42 49 x  1 w w w w w w w  7   

Square Kilometre Array Graphical representation

Multiply input by prototype filter Data Segment into r section of length M x Sum the r section to give output of length M

Filter h0(n) Fourier Transform M samples

Generates a single output for each frequency channel Segment and sum 

M-Point FFT Overlapped filter Filter bank output segments

Square Kilometre Array Critically sampled filterbank

Want to continuously process the data Data Next output obtained by shifting the x data) and applying the same process. If the shift is by M samples then each input to the FFT is processing largely the Filter h0(n) M samples same data. Gives a simple architecture

Pre-summation Segment and sum  po(m) p1(m) M-Point FFT M-Point FFT pM-1(m) Overlapped filter Implementation of Critically Sampled Filterbank Filter bank output segments

Square Kilometre Array SKA1 Low

to be built in Australia

Square Kilometre Array Square Kilometre Array Square Kilometre Array SKA1 Low Location

Boolardy

Geraldton Regulation of RFI

Geraldton Low Population Density

Geraldton gazetted towns: 0 population: “up to 160” Australian Strengths

Geraldton gazetted towns: 0 population: “up to 160” Square Kilometre Array SKA1 Low Station processing (Europe)

Bandwidth 300 MHz (Sky frequency 50-350 MHz) 512 antenna stations Each station 256 dual polarisation antenna (log periodic) Digitise 262k signal at 0.8 GS/s – total of 105 THz of bandwidth Beamform to produce a 300MHz beam – Frequency domain processing. – Apply filterbank 1024 channels (800Mhz to 781kHz) – Beamforming in a single channels is just – Multiply each input by a ejθ, adjust phase, and add  6 Petaflops  Total power ~1MW SKA1 Low Correlator Base Requirements (CSIRO, ASTRON, AUT) Bandwidth 300 MHz (Sky frequency 50-350 MHz)  From low station as 384 channels of 0.781kHz each Full Stokes 512 antenna stations Correlator compute load 1.25 Pflops equivalent  About order of magnitude more than JVLA correlator (US biggest current)  About half of the proposed SKA1 Mid correlator

Basic operating mode is a “zoom mode” with at least 64k frequency channels –across 300MHz, ~172 per 781 kHz beamformer channel  Resolution 4.07 KHz Plus Zoom Modes

Four Independent Zoom Bands nominally either  4 MHz  8 MHz  16 MHz or  32 MHz each – 256 possible combination At least 16k channels in each zoom band In addition “continuum” required at frequency within 300 MHz observing band, but not in a zoom band Plus Subarraying

The telescope can operate as 1 to 16 INDEPENDENT subarrays No antenna station can belong to the same subarray Plus provision is made for a maintenance (17th) subarray Up to 512 antennas when there is a single subarray Each subarray operates independently

Last implies we cannot change firmware to change modes.  FPGA based correlator Additional outputs from SKA1_Low.CBF

16 voltage beams full bandwidth for pulsar timing 500 pulsar search beams at 128MHz bandwidth Pulsar search can trade beams for bandwidth – 500 beams 128 MHz, – 250 beams 2x128 MHz, or – 133 beams 3x128 MHz

Again frequency domain beamforming. Plus Multibeaming

Each subarray can apportion its 300MHz to 8 beams Each beam is a contiguous section of bandwidth Each beam pointing is independent of the other beams Spectrum of beams can overlap  Example 8 beams all covering 200-237.5 MHz on the sky

Large number of mode “Normal” observing with zoom modes, subarraying, multibeaming, and independent scheduling blocks.

In the following will show how these are implemented in SKA1_Low Correlator and beamformer Implementing Delay Correction

) q s( o c Must bring all signal to same wavefront u before correlation in effect add delay of q u.cos(θ) to second antenna

u (wavelengths) Complex Correlator Step 1 Remove bulk delay by delay sampled data (~1us accuracy for LOW). Left with fractional delay error of up to 0.5us across ~1MHz delay = -dθ/dω (dω.delay = 180 degrees) Phase changes by up to 180 deg across ~1MHz Fine Delay Correction with Fractional Time Delay filters

FIR filter with values sampled from a continuous time filter. Change the initial sampling point of continuous time filter (red + and blue dot are a half sample different in delay Problems  Residual amplitude and phase errors, possibly Filter Impulse Response delay errors – Changes with delay value – Improve by making filter longer  In practice filter length large – more compute intensive than a filterbank Fine Delay Correction with Phase Slope

In frequency domain delay is equivalent of a phase slope “Normal” frequency resolution of correlator requires ~256 channel filterbank. Phase slope across a fine channel (4 kHz) is less than 1 degree (±0.5deg), average error is zero Apply correct phase, as a function of time, to each fine channel

Problem: Antennas with different velocities = different rate of change of delay = Doppler Shift,  Different Doppler shifts on different antennas (less than 10Hz)  Fine channels select different part of the sky spectrum – correlation loss Solution apply Doppler correction before fine filterbank Implementing Spectral Resolution

Six spectral resolution required: – Continuum, “normal” (4kHz) and four zoom modes Solution 1 Five Separate filterbanks – Implement a separate filterbank for each resolution – No filterbank for Continuum so fine delay must be fractional delay filter Solution 2 Combine zoom filterbanks – Zoom filterbanks are power of 2. Build variable length filterbank – For data from stations Normal mode is resolution not a power of 2, separate filterbank – Have two filterbanks and fractional delay filter Solution 3 Frequency averaging – Implement filterbank at finest frequency resolution and average in frequency to achieve other resolutions – 4096 channel filterbank -226Hz resolution, – Average 1, 2, 4, 8 channels gives 226, 452, 904Hz, 1.8kHz - all zoom modes – Average 18 channels give “normal” observing of 4kHz, average 3456 channels gives continuum (781kHz) SKA_Low solution

Use frequency averaging to implement all resolutions  A single filterbank,  Uniform data flow into correlator (same for all resolutions)

Use phase slope method for fractional delay  4096 channel filterbank = 0.02degree maximum phase error across channel  Very small phase error  No added amplitude error  Small compute load Correlation – six frequency resolutions

Correlator 1.25PFlops require 192 FPGAs Data output from filterbank FPGAs on 8 x 25Gbps links One link to group of 24 – cross connect with group of 24 Each FPGA process 1.56 MHz

Frequency averaging Passive Passive Optical Optical for different frequency res. Backplane Backplane From Station Correlations 24-way Visibility FPGAs Frequency to SDP Gearbox cross Correlator Sharing & Accumulation connect Packetisation

HMC HMC Memory Memory Multibeaming

For correlator difference in beam is simply a different delay polynomial (description of delay as a function of time) Each antenna station is in a single subarray. Eight beam per subarray require 8 delay polynomials for each antenna. For each coarse channel apply the appropriate delay polynomial. All extra complexity is with the filterbanks Correlator independent Subarraying

If there is NO SUBARRAYING correlation between all pairs of antennas is performed. In this design all correlation for single frequency channel occurs in a single FPGA and are stored to DRAM A subarray selects a subset of all possible correlation Arrange data in DRAM so that a block read are for correlation where one the of antenna station inputs is common SUBARRAYING becomes  For each antenna station in a subarray read blocks for that antenna station  Select the subset of correlation for the subarray and transmit for data processing Hardware Proposal

CSIRO-ASTRON Colaboration

Square Kilometre Array Hardware implementation Gemini Processing board

CSIRO ASTRON(Netherlands) collaboration Optical Transceiver and FPGA SERDES number now allow FULL OPTICAL DATA CONNECTIONS Proposed processing board is a simple single board unit with • A single FPGA • Parallel Optical transceivers for I/O (48 fibres total) • Four HMC for data storage • SFP+ 10GE for Monitor&Control

Square Kilometre Array Perentie Rack Unit

Packaging • Four Gemini processing boards • Liquid cooling • COTS Power • Optical to Front Panel • In a standard 1U rack unit • 32Tflop equivalent • 48 Rack units for correlation

Square Kilometre Array Interconnecting Perentie Rack Units

If Front Panel had single fibres simply route each fibre to destination But to keep density down have multi-fibre ribbons (up to 24/ri) COTS Passive Optical Backplanes connect a single fibre from one ribbon to another. User writes specifications and has it built. Similar to acquiring circuit boards.

Square Kilometre Array Putting it all together

Square Kilometre Array Thank you CASS John Bunton SKA1 CSP System Engineer t +61 2 9372 4420 e john.bunton@csiro.au w www.atnf.csiro.au/projects/askap

PO BOX 76 EPPING, 1710, AUSTRALIA Example ASKAP

Major Cross connects for SKA1_Survey  Filterbank to Beamformer ~2Tb/s per antenna  Beamformer to Correlator ~100Tb/s

Square Kilometre Array