Robust Statistics

Arnaud Delorme (with feedback & slides from C. Pernet & G. Roussellet)

Robust statistics

Parametric & non-parametric statistics: use mean and standard deviation (t-test, ANOVA, …)

Bootstrap and permutation methods: shuffle/bootstrap data and recompute measure of interest. Use the tail of the distribution to asses significance.

Correction for multiple comparisons: computing statistics on time(/frequency) series requires correction for the number of comparisons performed. Take-home messages

• Look at your data! Show your data!

• A perfect & universal statistical recipe does not exist

• Keep exploring: there are many great options, most of them available in free softwares and toolboxes References Parametric statistics

Assume gaussian distribution of data

Paired T-test: Compare paired/ Mean_ difference tN= −1 unpaired Standard_deviation Samples for continuous data. In EEGLAB, used for Unpaired grand-average ERPs. Mean− Mean tN= AB 22 ()()SDAB− SD

ANOVA: compare several VarianceinterGroup groups (can test interaction NGroup −1 F = between two factors for the VarianceWithinGroup repeated measure ANOVA) NN− Group

Why the standard figure is not good enough

Add confidence intervals Add plot of the difference How many subjects show an effect?

Robust measures of central tendency (location)

• Non-robust estimator – Mean: mERP = mean(EEG.data,..)

• Robust estimators of central tendency - Median: mdERP = median(EEG.data,...) - Trimmed mean tmERP = trimmean(EEG.data,...)

Trimmed means

• 20% trimmed means provide high power under normality and high power in the presence of outliers • Rand Wilcox, 2012, Introduction to Robust Estimation and Hypothesis Testing, Elsevier ERP application: Rousselet, Husk, Bennett & Sekuler, 2008, J. Vis. + Desjardins 2013

Non-parametric statistics

Paired t-test Wilcoxon Unpaired t-test Mann-Whitney One way ANOVA Kruskal Wallis

Values Ranks

BOTH ASSUME NORMAL DISTRIBUTIONS Problems

• Not resistant against outliers

• For ANOVA and t-test non-normality is an issue when distributions differ or when variances are not equal.

• Slight departure from normality can have serious consequences

Solutions 1. Randomization approach

2. Bootstrap approach Bootstrap: central idea

• “The bootstrap is a computer-based method for assigning measures of accuracy to statistical estimates.” Efron & Tibshirani, 1993

• “The central idea is that it may sometimes be better to draw conclusions about the characteristics of a population strictly from the sample at hand, rather than by making perhaps unrealistic assumptions about the population.” Mooney & Duval, 1993 Sample and population

Sample Population

H0: the mean is not 0 for the given that we have no other information population about the population, the sample is our best single estimate of the population Percentile bootstrap: general recipe

• sample = X1, ..., Xn • resample n observations with replacement • compute estimate • repeat B times

with B large enough the B estimates provide a good approximation of the distribution of the estimate of the sample

Bootstrap philosophy Percentile bootstrap estimate of confidence intervals Percentile bootstrap estimate of confidence intervals

2.5% 97.5% Distribution can take any shape

2.5% 97.5% 2.5% 97.5% 2.5% 97.5%

Once you have the 95% confidence interval, you can perform inferential statistics. Confidence interval for the difference Bootstrap approach H0

a2 a1 a3 a7 a6 analyze a4 a5

b2 difference Xorg b1 b3 b7 b6 analyze b4 b5 Confidence interval for the difference Bootstrap approach H0

a2 b3 a1 a3 b1 a2 analyze b7

b3 difference X1 a7 a5 b6 analyze b3 b1 a7 Confidence interval for the difference Bootstrap approach H0

a2 b7 a3 a3 b2 a7 analyze b7

difference X a3 2 a7 b2 b6 analyze b3 b1 a3 ConfidenceInferences interval based foron thepercentile difference bootstrapBootstrap method approach H0 2

Permutation /bootstrap

Sorted values

Thresholds

2.5% 97.5% Testing for difference 2 bootstrap approaches

• Bootstrap 1 is testing against H0: the two samples originate from the a&b same distribution.

0 2.5% 97.5% Original value Testing for difference 2 bootstrap approaches

• Bootstrap 1 is testing against H0: the two samples originate from the a&b same distribution.

0 2.5% 97.5% Original value • Bootstrap 2 is testing against H1: the two samples originate from the different distributions.

a b

2.5% 97.5% 2.5% 97.5% Measure for the bootstrap

b b a a a b a

t-test X2 a a b b a b

2.5% 97.5% Measure for the bootstrap c b a c a b a

Anova X2

c a b b a b

a c b c a b 2.5% 97.5% Resampling strategies: follow the data acquisition process

Independent sets: Dependent sets: • 2 conditions in single- • 2 conditions in group analyses subject analyses • Correlations • 2 groups of subjects, e.g. • Linear regression patients vs. controls Confidence interval for the difference Bootstrap approach H0 (paired)

a1 a2 a3 a4 a5

Pairwise Mean X Difference org b1 b2 b3 b4 b5 Confidence interval for the difference Bootstrap approach H0 (paired)

a3 a2 a2 a4 a1

Pairwise Mean X Difference 1 b3 b2 b2 b4 b1

a-b

2.5% 97.5% 0 Confidence interval for the difference Bootstrap approach H0 (paired) permutation

a1 b2 a4 b4 b5

Pairwise Mean X Difference 1 b1 a2 b2 a4 a1

a-b

0 2.5% 97.5% Original value Husband Wifes Husb. boot Wifes boot Husb. boot Wifes boot 22 25 27 62 24 71 Are the two groups 32 25 24 26 25 25 different: thats an 50 51 16 29 36 54 unpaired test 25 25 28 30 25 35 33 38 23 26 27 29 (comparing the mean 27 30 28 35 60 31 or median of husband 45 60 16 60 27 29 and the mean or 47 54 27 31 47 23 median of wife) 30 31 73 60 27 19 44 54 45 38 39 29 23 23 26 25 27 35 39 34 24 31 26 31 24 25 32 25 39 25 22 23 26 25 28 31 16 19 27 38 39 62 73 71 50 51 28 34 27 26 73 35 73 26 36 31 45 62 39 23 24 26 39 54 32 25 60 62 30 38 30 60 26 29 47 31 45 34 23 31 16 31 25 23 28 29 22 29 73 30 36 35 30 25 36 25 Diff= -1.88 Diff= -4.29 Diff= -2.83 2.5% 97.5% Husband Wifes Difference Sign perm Sign perm Sign perm 22 25 -3 3 3 -3 Are the two groups 32 25 7 -7 7 7 different: thats an 50 51 -1 -1 -1 1 25 25 0 0 0 0 unpaired test 33 38 -5 5 5 5 (comparing the mean 27 30 -3 3 3 3 or median of husband 45 60 -15 15 15 15 and the mean or 47 54 -7 -7 7 7 median of wife) 30 31 -1 -1 1 -1 44 54 -10 -10 -10 -10 23 23 0 0 0 0 Are husbands older than wifes: 39 34 5 5 5 -5 thats a paired test. Compute difference 24 25 -1 1 1 -1 between the two and change sign to 22 23 -1 1 -1 1 bootstrap (permutation) 16 19 -3 -3 3 3 73 71 2 -2 -2 -2 27 26 1 -1 1 1 36 31 5 5 5 -5 24 26 -2 -2 2 2 60 62 -2 -2 2 -2 26 29 -3 -3 3 3 23 31 -8 8 -8 8 28 29 -1 1 1 1 36 35 1 -1 -1 -1

Median -1 -0.5 1.5 1 2.5% 97.5% Bootstrap versus permutation

Permutation Bootstrap

each element only each element can get picked once get picked several times

Draws are dependent of each others Draws are independent of each others Use bootstrap when possible! Assessing significance

Original Difference 1 Difference 2 Difference 3 Difference 4 Difference …

Difference mask at p<0.05

2.5% 2.5% EEG1 EEG2 difference

_ = frequencies frequencies frequencies frequencies subjects subjects electrodes electrodes difference signs* difference*

* =

Correcting for multiple comparisons

• Bonferoni correction: divide by the number of comparisons (Bonferroni CE. Sulle medie multiple di potenze. Bollettino dell'Unione Matematica Italiana, 5 third series, 1950; 267-70.)

• Holms correction: sort all p values. Test the first one against α/N, the second one against α/(N-1)

• Max method

• False detection rate

• Clusters

Max procedure

• for each permutation or bootstrap loop, simply take the MAX of the absolute value of your estimator (e.g. mean difference) across electrodes and/or time frames and/or temporal frequencies.

• compare absolute original difference to this distribution

2.5% 97.5% FDR procedure

C1 C2 C3 Procedure: Index "j" Actual j*0.05/10 C2-C1 - Sort all p values (column C1) 1 0.001 0.005 -0.004 C3 2 0.002 0.01 -0.008

3 0.01 0.015 -0.005 - Create column C2 by computing j*α/N 4 0.03 0.02 0.01 - Subtract column C1 from C2 to build 5 0.04 0.025 0.015 column C3 6 0.045 0.03 0.015 7 0.05 0.035 0.015 - Find the highest negative index in C3 8 0.1 0.04 0.06 and find the corresponding p-value in C1 9 0.2 0.045 0.155 (p_fdr) 10 0.6 0.05 0.55

- Reject all null hypothesis whose p-value are less than or equal to p_fdr

FDR procedure

C1 C2 C3 Procedure: Index "j" Actual j*0.05/10 C2-C1 - Sort all p values (column C1) 1 0.001 0.005 -0.004 C3 2 0.002 0.01 -0.008

- Reject all null hypothesis whose p-value are less than or equal to p_fdr

FDR procedure

C1 C2 C3 Procedure: Index "j" Actual j*0.05/10 C2-C1 - Sort all p values (column C1) 1 0.001 0.005 -0.004 C3 2 0.002 0.01 -0.008

- Reject all null hypothesis whose p-value are less than or equal to p_fdr

FDR procedure Bonferoni

C1 C2 C3 Procedure: Index "j" Actual j*0.05/10 C2-C1 - Sort all p values (column C1) Holms 1 0.001 0.005 -0.004 C3 FDR 2 0.002 0.01 -0.008 - Create column C2 by computing j*α/N 3 0.01 0.015 -0.005 4 0.03 0.02 0.01 - Subtract column C1 from C2 to build column C3 5 0.04 0.025 0.015 6 0.045 0.03 0.015 - Find the highest negative index in C3 7 0.05 0.035 0.015 and 8 0.1 0.04 0.06 find the corresponding p-value in C1 (p_fdr) 9 0.2 0.045 0.155 10 0.6 0.05 0.55 - Reject all null hypothesis whose p-value are less than or equal to p_fdr

Uncorrected

Cluster correction for multiple comparisons

Original difference

2.5% 97.5%

44 pixels

Difference bootstrap 1 Difference bootstrap 2 Difference bootstrap 3

….

35 pixels 27 pixels

Control for multiple comparisons cluster method

Control for multiple comparisons cluster method References

Delorme, A. 2006. Statistical methods. Encyclopedia of Medical Device and Instrumentation, vol 6, pp 240-264. Wiley interscience.

Genovese et al. 2002. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. NeuroImage, 15: 870-878

Nichols & Hayasaka, 2003. Controlling the familywise error rate in functional neuroimaging: a comparative review. Statistical Methods in Medical Research, 12:419-446

Maris, 2004. Randomization tests for ERP topographies and whole spatiotemporal data matrices. Psychophysiology, 41: 142-151

Maris et al. 2007. Nonparametric statistical testing of coherence differences. Journal of Neuroscience Methods, 163: 161-175

Groppe, D.M., Urbach, T.P., & Kutas, M. (2011) Mass univariate analysis of event-related brain potentials/fields I: A critical tutorial review. Psychophysiology, 48(12) pp. 1711-1725.

Thanks to G. Rousselet

Exercice

• Experiment with the statcond function – Create 2 random vectors of values – Add signal to one of the variable – Use statcond EEGLAB function and compare permutation and parametric results – Repeat 100 times and plot the histogram of p- values