Robust Mean Estimation for Real-Time Blanking in Radioastronomy

ROBUST MEAN ESTIMATION FOR REAL-TIME BLANKING IN RADIOASTRONOMY

Philippe Ravier1, Cedric Dumez-Viou1−2

1Laboratory of Electronics, Signals, Images, Polytech’Orleans´ - University of Orleans 12, rue de Blois, BP 6744 Cedex 2, F-45067 Orleans,´ France 2Observatoire de Paris/(CNRS No.704), Station de radioastronomie, F-18330 Nanc¸ay, France phone: + (33).2.38.49.48.63, fax: + (33).2.38.41.72.45, email: [email protected] web: www.obs-nancay.fr/rﬁ

ABSTRACT assumption does not hold anymore and the exact PDF must be taken into account for precise threshold derivation. Radio astronomy observations suffer from strong interferences that need to be blanked both in the time and frequency domains. In order to achieve real-time computations, in- 2. PROCEDURE USED FOR REAL-TIME terference detection is made by simple thresholding. The In order to achieve real-time computation, a simple thresh- threshold value is linked to the mean estimation of the power olding drives the time and/or frequency blanking decision. 2 of clean observed data that follows a χ distribution. Spu- The problem is formulated as the following binary hy- rious values insensitivity is obtained by replacing mean by pothesis test: robust mean. A theoretical study of the variance of three estimators of the mean is presented. The study leads to a prac- tical trimmed percentage that is chosen for the robust mean. H0 : the SOI only is present, H1 : the SOI is polluted with additive interference. 1. INTRODUCTION The SOI is supposed to be Gaussian. On the other hand, Radioastronomy benefits from protected bands for the study no a priori information about the interference PDF is avail- of radio-sources. However, useful information is some- able. Nevertheless, the interference is usually powerful com- times only available in bands occupied by telecommunica- pared to the SOI. So the decision of blanking is made accord- tion users. In those cases, signals of interest (SOI) are gener- ing to the false alarm probability (Pf a) that may be acceptable ally polluted by strong radio frequency interferences (RFI). by the radio astronomers. The compromise for a good rejec- Observation is still possible if a receiver is designed to han- tion of the interferences is to give up the smallest number of dle high dynamic signals composed of weak SOI with su- the highest values of the observations without compromis- perimposed strong RFI. Then, signal processing make RFI ing clean data with residual RFI. In practice, the Pf a is taken mitigation possible to retrieve the SOI. between 0.1% and 0.0001%. One solution to make the deci- Such a receiver is one of the current projects at the sion concerning the hypothesis testing problem is to compare 3 Nançay Observatory [1]. Robust Radio Receiver (R ) can the observation to a threshold: the H0 hypothesis is realized connect to 3 instruments (Nançay Radio Telescope, Nançay when the observation is smaller than the threshold. The H1 Decameter Array, Surveillance Antenna) and process up to 8 hypothesis is realized in the other case. When H0 is realized, independent bands ranging from 875 kHz to 14 MHz band- the threshold value is updated with the new observation in- width that can be combined to provide a larger band of anal- put. When H1 is realized, the blanking decision is made and ysis. This band can be channelized into 64 to 8192 sub- the threshold value remains unchanged. bands. Either Blackman windowing or 49152-coefficients The threshold is calculated for a desired Pf a. Theoretical polyphase filter can be used for FFT apodisation. RFI mit- values may be derived when the PDF is known. Since the igation procedures can be applied in time and/or frequency SOI is Gaussian, the power of the observation follows a χ 2 domain, i.e. before and/or after FFT calculation. Current im- distribution in both time and frequency domains. Assuming plementations are based on a robust threshold power detector. the SOI follows a N (0,σ 2) distribution, the observation re- They allow real-time RFI mitigation : interference rejection sults in the sum of ν squared SOI independent realizations. is processed while observation is running. The χ2(ν) cumulative distribution function may be written For long time exposure, noise power probability density as an incomplete normalized Gamma function P : function (PDF) is assumed to be Gaussian. However RFI mitigation may require high time resolution. The Gaussian ν x Fχ2(ν)(x) = P , , The Nançay Radio Observatory is the Unite´ scientifique de Nançay of 2 2 the Observatoire de Paris, associated as Unite´ de Service et de Recherche No. 704 to the French Centre National de la Recherche Scientifique where the function P is defined as (CNRS). The Nançay Observatory also gratefully acknowledges the financial support of the Conseil Regional´ de la Region´ Centre in France. Γ(ν,x) 1 x The work of C. Dumez-Viou was financially supported by Observatoire P(ν,x) = = tν−1e−t dt de Paris and by the European Social Fund. Γ(ν) Γ(ν) Z0 and Γ(ν) = Γ(ν,+∞) is the Gamma function [2]. Given an admissible false alarm rate Pf a = α leads to the following 2 −1 ν −1 value threshold = 2σ P 2 ,1 − α where P is the in- verse tabulated P function. Ho wever,the variance σ 2 of each SOI realization is not precisely known and that value fluctu- ates over time due to changes in signal transmission (antenna position, ionosphere properties, . . . ). Since the mean value 2 2 µχ2(ν) of a χ (ν) pdf equals νσ , the threshold is finally estimated as:

µχ2(ν) ν threshold = 2 P−1 ,1 − α (1) b ν 2 Power (log arbitrary unit) The key point therefore relies on a good estimation of the mean value of the power of the data that has to be RFI- mitigated. Different estimators have been computed in time and/or frequency domains for the best detection possible fit- ting the nature of the interferences. 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 Frequency (MHz) 2.1 Time Blanking Time blanking needs to detect aberrant points, with real-time Figure 1: Illustration of sky noise background observation constraints. The input data are the real part and imaginary without (top) and with (bottom) radar blanking using the part of the complex-demodulated band-shifted received data. Nançay Radio Telescope. Time-exposure of 40s. The curves Time blanking is processed on the waveform that holds the are shifted for display purpose. instantaneous power of the received data. In this case, the thresholding procedure involves estimating the mean µx of independent realizations following χ 2(2) distributions. On- based on 50% overlapped temporal slices. Beyond weight- line computation naturally leads to an adaptive way forb the ing functions, the spectrum analyser offers two degrees of estimation of µx with respect to time: freedom for the computation of the power spectral density (PSD): the FFT length ranging from N=64 to N=4096 and b the number of accumulations A (from 20 to 217) generating µx[t] = µx[t − 1] + ε · (x[t] − µx[t − 1]) 2 the average PSD. Each periodogram is computed as |Xi( f )| 2 2 th The wbaveformb is computed as x[tb] = xQ[t] + xI [t]. The with Xi( f ) being the FFT of the i temporal slice. components xQ[t] and xI [t] follow a centered Gaussian dis- The resulting estimator writes: tribution with unknown variance σ 2. The time t − 1 stands for the acquisition time of the samples preceding the current A one. The choice of ε is a compromise between a short time 2 P( f ) = ∑ |Xi( f )| response, a low variance of the mean estimator and RFI in- i=1 sensitivity. First, a short settling time of the estimator is required to so that P( f ) follows a χ2(ν) with ν = 2A varying from 21 perform RFI mitigation as soon as possible after the begin- to 218 degrees of freedom. Note that the average periodogram 1 ning of the acquisition sequence. Then, the mean estima- theoretically rescales P( f ) by A . This stage is performed a tor is unbiased and the small value of ε corresponds to a re- posteriori for implementation constraints. 4 duced variance of the estimator since Var(µx) = 2εσ (with A frequency blanking procedure is applied each time a σ 2 generally weak). Finally, RFI insensitivity is achieved by PSD is estimated. The threshold procedure is applied on the using a time constant longer than RFI events.b For example, whole PSD. The threshold calculus (1) therefore holds, how- specifications of ground-based aviation radars give a pulse ever the mean µ is estimated on the PSD samples. 3 rate frequency of To = 1 ms for the emitted signal. On the R Nevertheless, the mean of the PSD may be strongly cor- receiver, ε is set to 0.00012 resulting in a settling time of 2-3 rupted, due to bsome channels that may be occupied by RFI To. polluters. The median estimator may be preferred instead. Note that when interference is detected, the mean esti- The problem is the computation cost of such an estimator as mator value is not updated with the new sample in order to well as its deviation to the desired mean when the distribu- prevent the estimation to be polluted by incoherent values. tion is asymmetric (i.e. for low ν values). Another solution The last computed estimator is kept unchanged. This time relies on robust mean estimators such as the trimmed mean blanking procedure is real-time operational in the R3 instru- or the Winsorized mean. ment at the front-end part of the spectrum analyser. Its use In the following part, we compare the mean estimator of for radar blanking allows to freely observe in the highly cor- the threshold value with the robust mean and the median es- rupted bands occupied by radars (Figure 1). timators.

2.2 Frequency Blanking 3. ROBUST ESTIMATION OF THE MEAN For scientiﬁc analysis purposes, time-frequency maps are For an easy comparison, we hereafter give the bias and vari- computed on the ﬂy: an averaged periodogram is estimated ance of each estimator. The comparison will help choosing the suitable parameters for the retained method.

3.1 The mean estimator 1.1 ν = 2

The reference mean estimator in (1) is theoretically ex- / median 1 pressed as the expectation of the P( f ) distribution: 0.9 (m +/− s) = E {P( f )} = 2 0.8 µMN νσ 32 64 128 256 512 1024 2048 4096 8192 Number of frequency channels N Let us estimate µMN by:

1.1 ν = 8

1 N / median µMN = ∑ P( f ) 1 N f =1 0.9

b (m +/− s) This classical estimator is consistent with the number of 0.8 Fourier coefﬁcients N since the estimator is unbiased with 32 64 128 256 512 1024 2048 4096 8192 Number of frequency channels N variance 2νσ 4/N and the P( f ) distribution is supposed to be white. 1.1 ν = 16 3.2 The median estimator 1 Let us note the theoretical median of P( f ) as: / median 0.9

ν 1 (m +/− s) 0.8 µ = median {P( f )} = 2σ 2P−1 , . 32 64 128 256 512 1024 2048 4096 8192 MD f ∈IR 2 2 Number of frequency channels N The median is estimated on N values: Figure 2: Mean and standard deviation for the median estimator (normalized with respect to the true median) for ν=2, µMD = median f =1...N {P( f )}. 8 and 16. The biasband variance of this estimator needs to derive the law of an order statistics for a χ 2(ν) parent distribution. The k-moments of the median estimator are expressed as: ν a = 2σ 2P−1( ,δ) 1 k 2 k 2 −1 ν u N −1 N E µ = K 2σ P , u 2 (1 − u) 2 du, 2 −1 ν MD N Z b = 2σ P ( ,1 − δ) n o 0 h 2 2i 2 b N Γ(N + 1) with KN = 2 (Γ(N/2 + 1))2 The function F(x,ν) stands for the χ 2(ν) cumulative distribution function. Note that µMR(0) = µMN . Identically, we The bias and variance are numerically computed. δ →0.5 ( ) → (i) th Figure 2 shows that the estimator becomes unbiased with have µMR δ µMD. Let us note P the i order statis- tic of P( f ) when the data are arranged in increasing order. the number of channels N growing and degree of freedom ν growing. The bias for low N values is actually negligible. The δ trimmed mean is estimated by

3.3 The trimmed mean estimator N−q 1 (i) The trimmed mean, like the median, is insensitive to spurious µMR(δ) = P , with q = bδNc N − 2q ∑ data or outliers. The trimmed mean is the mean computed af- i=q+1 ter the q smallest observations and the q largest observations b in the sample are deleted. The variance of this estimator approximately writes: Considering Pδ ( f ) the δ trimmed distribution of P( f ). The parameter δ = q/N is the percentage of trimmed values ν2σ 4 at each side. The theoretical trimmed estimator writes: Var {µ (δ)} =∼ . MR N − 2q 2 P( ν + 2, b ) − P( νb+ 2, a ) − P( ν + 1, b ) − P( ν + 1, a ) µMR(δ) = E {Pδ ( f )} 2 2 2 2 2 2 2 2 [F(b,ν) −F(a,ν)] P( ν + 1, b ) − P( ν + 1, a ) = νσ 2 2 2 2 2 F(b,ν) − F(a,ν) 4. DISCUSSION The parameters a and b correspond to the low and high δ Figure 3 (top) shows the bias of estimation of the robust esti- 2 trimmed values of the χ (ν) PDF: mators with respect to the desired mean value µMN . Since the µMR(δ) estimators are unbiased with respect to their theoret- This mistake may be overcome by calculating a corrected ical values, the bias of µMR(δ) lies between 0 (case δ = 0, trimmed mean. The trimmed values can be numerically com- equib valent to µMN ) and 30 % that is the bias for µMD (case puted in order to make the PDF symmetric such that the cor- 0 δ → 0.5). A normalizedb bias with respect to µMN is repre- rected trimmed mean equals the mean. The q largest obser- sented: its value rapidly falls with an increasingbdegree of vations and the q00 smallest observations are deleted such that freedom. Note that the normalized bias is independent of the δ = (q0 + q00)/2N is the average percentage of trimmed val- window length N. ues and both the PDF and the trimmed PDF share the same ﬁrst moment (Figure 4). These constraints yield to solve the Normalized bias of µ and µ with respect to µ 0 0 MR MD MN nonlinear system with the two unknown quantities a and b 35

30 µ 0 0 MD P ν , b − P ν , a = 1 − 2δ (%) µ (40%) 2 2 2 2 25 MR 

MN µ (10%) MR

µ N = 64,...,4096  µ (1%)  b0 a0 20 MR ν ν 0 0 ) / P 2 + 1, 2 − P 2 + 1, 2 = F(b ,ν) − F(a ,ν) MN 15  µ  Solutions are numerically computed for all the values of

− 10 ν and for the retained value of δ = 30% giving a normalized

MR standard deviation of the unbiased estimator less than 5 %. µ 5 ( 0 0 5 10 15 20 Degree of freedom (power of 2) Normalized std of µ , µ and µ with respect to µ MN MR MD MN 15 µ MD µ (40%) N = 64 MR

(%) µ (10%) 10 MR µ (1%) MN MR µ µ

µ MD MN µ MN b b’ ) / a a’ µ (10%) MR

MR 5 µ (

σ Figure 4: Trimmed values after correction of the trimmed mean estimator. µMR(α) is now equal to µMN . 0 0 5 10 15 20 Degree of freedom (power of 2) 5. CONCLUSION Thanks to time-blanking and frequency-blanking, the celes- Figure 3: Normalized bias (top) and standard deviation (bot- tial observation of certain frequency bands that were tradi- tom) for the mean, median, and trimmed mean estimators. tionally highly polluted may finally be explored for scien- tific purposes. Such blanking methods have proved their The smallest value N = 64 yields the highest normalized efficiency in real-time survey experiments (especially with standard deviation as shown figure 3 (bottom). The µMD es- radar polluters). However, time-blanking assumes very short timator is the poorest. However it does not exceed 12.5 % of wide band interferences manifestations and does not take into normalized standard deviation. The µMD estimator isbslightly account permanent or narrow band polluters. These latter better. The normalized standard deviation of µMR(δ) tends change the distribution of the temporal waveform and may towards zero with an increasing valueb of δ. This is not sur- have an impact on the threshold. prising since the pdf becomes narrower and narrob wer when In practice, front-end analog filters distort the flatness of the trimmed parameter δ increases. When δ tends to 0.5, the expected white noise leading to a reduced RFI blanking the samples are more and more confined between two cer- efficiency. This work allows a fast and robust estimation of tain values around the median value: the standard deviation the mean for each single channel resulting in a good estima- of the trimmed pdf decreases (although the samples num- tion of analog filters responses prior to any acquisitions. ber becomes smaller). On the other hand, the µMD estimator has a completely different pdf that depends on order statistics REFERENCES showing bad performances in terms of standardb deviation of the estimator. [1] R. Weber, L. Amiaud, A. Coffre, L. Denis, A. Leca- In practice, a high trimmed value is preferable for sake cheux, C. Rosolen, C. Viou and P. Zarka, “Digital im- of robustness with respect to RFI. The normalized standard plementation of a robust radio astronomical spectral deviation is therefore weak. The counterpart is the important analyser: towards IIIZW35 reconquest,” in Proc. EU- SIPCO 2004, Vienna, Austria, September 6-10. 2004, bias with respect to µMN for low degrees of freedom. As an example, the observed normalized bias of 27 % pp. 773776. actually produces an error which is as big as taking 0.01 % [2] M. Abramowitz and I. A. Segun, “Handbook of math- instead of 0.0001 % in the false alarm probability, when ac- ematical functions,” Dover Publications, New York, cumulating 16 spectra (ν=32). 1965.