DOCSLIB.ORG

Benjamini-Hochberg Method

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Benjamini-Hochberg Method P-values and statistical tests 7. Multiple test corrections Marek Gierliński Division of Computational Biology Hand-outs available at http://is.gd/statlec 2 Lets perform a test � times False positives True positives H0 true H0 false Total Number of Significant �� �� � discoveries Not �� �� � − � significant Total �( �) � Number of tests True negatives False negatives 3 Family-wise error rate ���� = Pr (�� ≥ 1) 4 Probabilities of independent events multiply 1 1 � � and � = �(�)×�(�) 2 4 1 2 1 2 Toss two coins 1 2 1 2 1 2 5 Probabilities of either event is 1 − 1 − � = 1 1 � � or � = ? 2 4 1 � � = 1 − �(�) 2 1 1 2 4 � � and � = � � ×� � = Toss two = 1 − � � coins 1 1 � � or � = 1 − � � and � 2 4 1 = 2 � � or � = 1 − 1 − � � 1 1 2 4 1 = 3 �(� or �) = 1 − 1 − = 2 4 6 False positive probability H0: no effect Set � = 0.05 One test Probability of having a false positive �) = � Two independent tests Probability of having at least one false positive in either test = �= = 1 − 1 − � � independent tests Probability of having at least one false positive in any test F �F = 1 − 1 − � 7 Family-wise error rate (FWER) Probability of having at least one false positive among � tests; � = 0.05 F �F = 1 − 1 − � Jelly beans test, � = 20, � = 0.05 FWER =( �=( = 1 − 1 − 0.05 = 0.64 � 8 Bonferroni limit – to control FWER Probability of having at least one false positive Controlling FWER among � tests; � = 0.05 We want to make sure that F �F = 1 − 1 − � ���� ≤ �J. Then, the FWER is controlled at level �′. Bonferroni limit FWER � �J = � � F � = 1 − 1 − ≈ � F � 9 Test data (1000 independent experiments) Random samples, size � = 5, from two normal distributions Negatives �( = 20 g �( = 970 Positives �) = 40 g �) = 30 10 One sample t-test, H0: � = 20 g No correction H0 true H0 false Total Significant 56 30 86 Not 914 0 914 significant Total 970 30 1000 30 positives 970 negatives 11 One sample t-test, H0: � = 20 g No correction H0 true H0 false Total Significant �� = 56 �� = 30 86 Not significant �� = 914 �� = 0 914 Total 970 30 1000 Family-wise error rate ���� = Pr (�� ≥ 1) 56 ��� = = 0.058 �� �� 56 + 914 False positive rate ��� = = � �� + �� ( 0 ��� = = 0 �� �� 0 + 30 False negative rate ��� = = �) �� + �� 12 Bonferroni limit No correction Bonferroni � 0.05 5×10ST ��� 0.058 0 ��� 0 0.87 No correction Bonferroni H0 true H0 false Total H0 true H0 false Total Significant 56 30 86 Significant 0 4 4 Not Not 914 0 914 970 26 996 significant significant Total 970 30 1000 Total 970 30 1000 13 Holm-Bonferroni method Sort p-values �()), �(=), … , �(F) V Reject (1) if � ≤ ()) F V Reject (2) if � ≤ (=) FS) V Reject (3) if � ≤ (W) FS= ... � � V � � � Stop when � > � � − � + 1 (X) FSXZ) 1 0.003 0.05 0.01 0.01 2 0.005 0.05 0.01 0.0125 Holm-Bonferroni method 3 0.012 0.05 0.01 0.017 controls FWER 4 0.04 0.05 0.01 0.025 5 0.058 0.05 0.01 0.05 14 Holm-Bonferroni method No correction Bonferroni HB � 0.05 5×10ST 5×10ST ��� 0.058 0 0 ��� 0 0.87 0.87 30 smallest p-values (out of 1000) Holm-Bonferroni Bonferroni H true H false Total Holm-Bonferroni 0 0 Significant 0 4 4 Not 970 26 996 significant Total 970 30 1000 15 False discovery rate �� ��� = � False discovery rate False positive rate False discovery rate �� �� �� �� ��� = = ��� = = �( �� + �� � �� + �� The fraction of truly non-significant events The fraction of discoveries that are false we falsely marked as significant 56 56 ��� = = 0.058 ��� = = 0.65 970 86 No correction H0 true H0 false Total Significant �� = 56 �� = 30 86 Not significant �� = 914 �� = 0 914 Total 970 30 1000 17 BenJamini-Hochberg method Sort p-values �()), �(=), … , �(F) Find the largest �, such that � � ≤ � (X) � Reject all null hypotheses for � = 1, … , � � � � � � � � � � − � + 1 � 1 0.003 0.05 0.01 0.01 0.01 2 0.005 0.05 0.01 0.0125 0.02 Benjamini-Hochberg 3 0.012 0.05 0.01 0.017 0.03 method controls FDR 4 0.038 0.05 0.01 0.025 0.04 5 0.058 0.05 0.01 0.05 0.05 18 BenJamini-Hochberg method No correction Bonferroni HB BH � 0.05 5×10ST 3.7×10ST 0.0011 ��� 0.058 0 0 0.0021 ��� 0 0.87 0.87 0.30 ��� 0.65 0 0 0.087 Benjamini-Hochberg Benjamini-Hochberg Bonferroni H true H false Total Holm-Bonferroni 0 0 Significant 2 21 23 Not 968 9 977 significant Total 970 30 1000 19 Controlling FWER and FDR Holm-Bonferroni Benjamini-HochBerg controls FWER controls FDR ���� = Pr (�� ≥ 1) �� ��� = �� + �� ��� is a random variable Controlling FWER - guaranteed Controlling FDR - guaranteed ���� ≤ �′ �[���] ≤ � 20 BenJamini-Hochberg procedure controls FDR Controlling FDR �[���] ≤ � � ��� can be approXimated by the mean over many eXperiments Bootstrap: generate test data 10,000 times, perform 1000 t-tests for each set and find FDR for BH procedure 21 AdJusted p-values p-values can be “adjusted”, � so they compare directly X with �, and not � F adjusted p-values Problem: adjusted p-value � does not eXpress any � probability � Despite their popularity, I raw p-values recommend against using adjusted p-values 22 How to do this in R # Read generated data > d = read.table("http://tiny.cc/two_hypotheses", header=TRUE) > p = d$p # Holm-Bonferroni procedure > p.adj = p.adjust(p, "holm") > p[which(p.adj < 0.05)] [1] 1.476263e-05 2.662440e-05 3.029839e-05 # Benjamini-Hochberg procedure > p.adj = p.adjust(p, "BH") > p[which(p.adj < 0.05)] [1] 1.038835e-03 6.670798e-04 1.050547e-03 1.476263e-05 5.271367e-04 [6] 3.503370e-04 9.664789e-04 1.068863e-03 7.995860e-04 5.404476e-04 [11] 9.681321e-04 1.580069e-04 1.732747e-04 3.159954e-04 2.662440e-05 [16] 4.709732e-04 1.517964e-04 2.873971e-04 3.258726e-04 4.087615e-04 [21] 3.029839e-05 9.320438e-04 1.713309e-04 2.863402e-04 4.082322e-04 23 Estimating false discovery rate Control and estimate Controlling FDR Estimating FDR 1. FiX acceptable FDR limit, �, For each p-value, �_, form a point beforehand estimate of FDR, 2. Find a thresholding rule, so that ���`(�_) �[���] ≤ � 25 P-value distribution Data set 2 100% null 80% null, 20% alternative 26 P-value distribution Good Bad! 27 Definition of �( Proportion of null tests 80% null, 20% alternative # non−significant tests � = ( # all tests H1 H0 �( 28 Storey method Estimate �� Use the histogram for � > � to estimate �n((�) Do it for all 0 < � < 1 and then find the best �n( � Storey, J.D., 2002, JR Statist Soc B, 64, 479 29 Point estimate of FDR Point estimate, ���(�) 80% null, 20% alternative Arbitrary limit �, every �_ < � is significant. No. of significant tests is �(�) = # �_ < � No. of false positives is True positives ��(�) = ��(� False positives Hence, �� � � ��� � = ( ( �(�) � 30 Storey method Point estimate of FDR This is the so-called q-value: � �_ = min ��� � wxyz If monotonic � �_ = ��� �_ 31 How to do this in R > library(qvalue) # Read data set 1 > pvalues = read.table('http://tiny.cc/multi_FDR', header=TRUE) > p = pvalues$p # Benjamini-Hochberg limit > p.adj = p.adjust(p, method='BH') > sum(p.adj <= 0.05) [1] 216 # q-values > qobj = qvalue(p) > q = qobj$qv > summary(qobj) pi0: 0.8189884 Cumulative number of significant calls: <1e-04 <0.001 <0.01 <0.025 <0.05 <0.1 <1 p-value 40 202 611 955 1373 2138 10000 q-value 0 1 1 96 276 483 10000 local FDR 0 1 3 50 141 278 5915 > plot(qobj) > hist(qobj) 32 33 Interpretation of q-value � ≤ 0.0273 No. ID p-value q-value There are 106 tests with ... ... ... ... Expect 2.73% of false positives among 100 9249 0.000328 0.0266 these tests 101 8157 0.000328 0.0266 102 8228 0.000335 0.0269 Expect ~3 false positives if you set a limit 103 8291 0.000338 0.0269 of � ≤ 0.0273 or � ≤ 0.00353 104 8254 0.000347 0.0272 105 8875 0.000348 0.0272 106 8055 0.000353 0.0273 q-value tells you how many false 107 8235 0.000375 0.0284 positives you should expect after 108 8148 0.000376 0.0284 choosing a significance limit 109 8236 0.000381 0.0284 110 8040 0.000382 0.0284 ... ... ... ... 34 Q-values vs BenJamini-Hochberg When �n( = 1, both methods give the same result. BH adjusted �n( p-value For the same FDR, Storey’s method provides more q-value significant p-values. Hence, it is more powerful, especially for small �n(. But this depends on how good the estimate of �n( is. �n( - estimate of the proportion of null (non-significant) tests 35 Which multiple-test correction should I use? Benjamini-Hochberg No correction Bonferroni Storey False positives False negatives False positive False negative “Discover” effect where there is no effect Missed discovery Can be tested in follow-up eXperiments Once you’ve missed it, it’s gone Not hugely important in small samples Impossible to manage in large samples 36 Multiple test procedures: summary Method Controls Advantages Disadvantages Recommendation No FPR False negatives not Can result in Small samples, when correction inflated �� ≫ �� the cost of FN is high Bonferroni FWER None Lots of false Do not use negatives Holm- FWER Slightly better than Lots of false Appropriate only Bonferroni Bonferroni negatives when you want to guard against any false positives Benjamini- FDR Good trade-off On average, � of Better in large Hochberg between false your positives will samples positives and be false negatives Storey -- More powerful than Depends on a The best method, BH, in particular for good estimate of gives more insight small �n( �n( into FDR 37 Hand-outs available at http://tiny.cc/statlec BenJamini-Hochberg method Theoretical null distribution: uniform p-values � � = (X) � Generated data null distribution 39 BenJamini-Hochberg method Test data Null data 970 H0 1000 H0 20 H1 p-values � 100% FDR � � � 5% FDR � 40 BenJamini-Hochberg method Test data Null data 970 H0 1000 H0 20 H1 � � value value - - p p � � � 41.
Load more

Recommended publications
  • Underestimation of Type I Errors in Scientific Analysis
    Open Access Journal of Mathematical and Theoretical Physics Research Article Open Access Underestimation of type I errors in scientific analysis Abstract Volume 2 Issue 1 - 2019 Statistical significance is the hallmark, the main empirical thesis and the main argument Roman Anton to quantitatively verify or falsify scientific hypothesis in science. The correct use of Department of Theoretical Sciences, The University of Truth and statistics assures and increases the reproducibility of the results and findings, and Common Sense, Germany there is recently a widely wrong use of statistics and a reproducibility crisis in science. Today, most scientists can’t replicate the results of their peers and peer-review hinders Correspondence: Roman Anton, Department of Theoretical them to reveal it. Besides an estimated 80% of general misconduct in the practices, Sciences, The University of Truth and Common Sense, Germany, and above 90% of non-reproducible results, there is also a high proportion of generally Email false statistical assessment using statistical significance tests like the otherwise valuable t-test. The 5% confidence interval, the widely used significance threshold Received: October 29, 2018 | Published: January 04, 2019 of the t-test, is regularly and very frequently used without assuming the natural rate of false positives within the test system, referred to as the stochastic alpha or type I error. Due to the global need of a clear-cut clarification, this statistical research paper didactically shows for the scientific fields that are using the t-test, how to prevent the intrinsically occurring pitfalls of p-values and advises random testing of alpha, beta, SPM and data characteristics to determine the right p-level, statistics, and testing.
    [Show full text]
  • The Reproducibility of Research and the Misinterpretation of P Values
    bioRxiv preprint doi: https://doi.org/10.1101/144337; this version posted August 2, 2017. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC 4.0 International license. 1 The reproducibility of research and the misinterpretation of P values David Colquhoun University College London Email: [email protected] Revision 6 Abstract We wish to answer this question If you observe a “significant” P value after doing a single unbiased experiment, what is the probability that your result is a false positive?. The weak evidence provided by P values between 0.01 and 0.05 is explored by exact calculations of false positive rates. When you observe P = 0.05, the odds in favour of there being a real effect (given by the likelihood ratio) are about 3:1. This is far weaker evidence than the odds of 19 to 1 that might, wrongly, be inferred from the P value. And if you want to limit the false positive rate to 5 %, you would have to assume that you were 87% sure that there was a real effect before the experiment was done. If you observe P = 0.001 in a well-powered experiment, it gives a likelihood ratio of almost 100:1 odds on there being a real effect. That would usually be regarded as conclusive, But the false positive rate would still be 8% if the prior probability of a real effect was only 0.1.
    [Show full text]
  • From P-Value to FDR
    From P-value To FDR Jie Yang, Ph.D. Associate Professor Department of Family, Population and Preventive Medicine Director Biostatistical Consulting Core In collaboration with Clinical Translational Science Center (CTSC) and the Biostatistics and Bioinformatics Shared Resource (BB-SR), Stony Brook Cancer Center (SBCC). OUTLINE P-values - What is p-value? - How important is a p-value? - Misinterpretation of p-values Multiple Testing Adjustment - Why, How, When? - Bonferroni: What and How? - FDR: What and How? STATISTICAL MODELS Statistical model is a mathematical representation of data variability, ideally catching all sources of such variability. All methods of statistical inference have assumptions about • How data were collected • How data were analyzed • How the analysis results were selected for presentation Assumptions are often simple to express mathematically, but difficult to satisfy and verify in practice. Hypothesis test is the predominant approach to statistical inference on effect sizes which describe the magnitude of a quantitative relationship between variables (such as standardized differences in means, odds ratios, correlations etc). BASIC STEPS IN HYPOTHESIS TEST 1. State null (H0) and alternative (H1) hypotheses 2. Choose a significance level, α (usually 0.05) 3. Based on the sample, calculate the test statistic and calculate p-value based on a theoretical distribution of the test statistic 4. Compare p-value with the significance level α 5. Make a decision, and state the conclusion HISTORY OF P-VALUES P-values have been in use for nearly a century. The p-value was first formally introduced by Karl Pearson, in his Pearson's chi-squared test and popularized by Ronald Fisher.
    [Show full text]
  • Statistical Significance for Genome-Wide Experiments
    Statistical Significance for Genome-Wide Experiments John D. Storey∗ and Robert Tibshirani† January 2003 Abstract: With the increase in genome-wide experiments and the sequencing of multi- ple genomes, the analysis of large data sets has become commonplace in biology. It is often the case that thousands of features in a genome-wide data set are tested against some null hypothesis, where many features are expected to be significant. Here we propose an approach to statistical significance in the analysis of genome-wide data sets, based on the concept of the false discovery rate. This approach offers a sensible balance between the number of true findings and the number of false positives that is automatically calibrated and easily interpreted. In doing so, a measure of statistical significance called the q-value is associated with each tested feature in addition to the traditional p-value. Our approach avoids a flood of false positive results, while offering a more liberal criterion than what has been used in genome scans for linkage. Keywords: false discovery rates, genomics, multiple hypothesis testing, p-values, q-values ∗Department of Statistics, University of California, Berkeley CA 94720. Email: [email protected]. †Department of Health Research & Policy and Department of Statistics, Stanford University, Stanford CA 94305. Email: [email protected]. 1 Introduction Some of the earliest genome-wide analyses involved testing for linkage at loci spanning a large portion of the genome. Since a separate statistical test is performed at each locus, traditional p- value cut-offs of 0.01 or 0.05 had to be made stricter to avoid an abundance of false positive results.
    [Show full text]
  • Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy
    Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Chapter 11 Interpreting results and drawing conclusions Patrick Bossuyt, Clare Davenport, Jon Deeks, Chris Hyde, Mariska Leeflang, Rob Scholten. Version 0.9 Released December 13th 2013. ©The Cochrane Collaboration Please cite this version as: Bossuyt P, Davenport C, Deeks J, Hyde C, Leeflang M, Scholten R. Chapter 11:Interpreting results and drawing conclusions. In: Deeks JJ, Bossuyt PM, Gatsonis C (editors), Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 0.9. The Cochrane Collaboration, 2013. Available from: http://srdta.cochrane.org/. Saved date and time 13/12/2013 10:32 Jon Deeks 1 | P a g e Contents 11.1 Key points ................................................................................................................ 3 11.2 Introduction ............................................................................................................. 3 11.3 Summary of main results ......................................................................................... 4 11.4 Summarising statistical findings .............................................................................. 4 11.4.1 Paired summary statistics ..................................................................................... 5 11.4.2 Global measures of test accuracy ....................................................................... 10 11.4.3 Interpretation of summary statistics comparing index tests ............................... 12 11.4.4 Expressing
    [Show full text]
  • Redefine Statistical Significance We Propose to Change the Default P-Value Threshold for Statistical Significance from 0.05 to 0.005 for Claims of New Discoveries
    comment Redefine statistical significance We propose to change the default P-value threshold for statistical significance from 0.05 to 0.005 for claims of new discoveries. Daniel J. Benjamin, James O. Berger, Magnus Johannesson, Brian A. Nosek, E.-J. Wagenmakers, Richard Berk, Kenneth A. Bollen, Björn Brembs, Lawrence Brown, Colin Camerer, David Cesarini, Christopher D. Chambers, Merlise Clyde, Thomas D. Cook, Paul De Boeck, Zoltan Dienes, Anna Dreber, Kenny Easwaran, Charles Efferson, Ernst Fehr, Fiona Fidler, Andy P. Field, Malcolm Forster, Edward I. George, Richard Gonzalez, Steven Goodman, Edwin Green, Donald P. Green, Anthony Greenwald, Jarrod D. Hadfield, Larry V. Hedges, Leonhard Held, Teck Hua Ho, Herbert Hoijtink, Daniel J. Hruschka, Kosuke Imai, Guido Imbens, John P. A. Ioannidis, Minjeong Jeon, James Holland Jones, Michael Kirchler, David Laibson, John List, Roderick Little, Arthur Lupia, Edouard Machery, Scott E. Maxwell, Michael McCarthy, Don Moore, Stephen L. Morgan, Marcus Munafó, Shinichi Nakagawa, Brendan Nyhan, Timothy H. Parker, Luis Pericchi, Marco Perugini, Jeff Rouder, Judith Rousseau, Victoria Savalei, Felix D. Schönbrodt, Thomas Sellke, Betsy Sinclair, Dustin Tingley, Trisha Van Zandt, Simine Vazire, Duncan J. Watts, Christopher Winship, Robert L. Wolpert, Yu Xie, Cristobal Young, Jonathan Zinman and Valen E. Johnson he lack of reproducibility of scientific not address the appropriate threshold for probabilities. By Bayes’ rule, this ratio may studies has caused growing concern confirmatory or contradictory replications be written as: over the credibility of claims of new of existing claims. We also do not advocate T Pr Hx discoveries based on ‘statistically significant’ changes to discovery thresholds in fields ()1obs findings.
    [Show full text]
  • Hypothesis Testing
    Hypothesis Testing Wolfgang Huber, EMBL Karl Popper (1902-1994) Logical asymmetry between verification and falsifiability. ! No number of positive outcomes at the level of experimental testing can confirm a scientific theory, but a single counterexample is logically decisive: it shows the theory is false. The four steps of hypothesis testing Step 1: Set up a model of reality: null hypothesis, H0 Step 2: Do an experiment, collect data Step 3: Compute the probability of the data in this model Step 4: Make a decision: reject model if the computed probability is deemed to small H0: a model of reality that lets us make specific predictions of how the data should look like. The model is stated using the mathematical theory of probability. Examples of null hypotheses: • The coin is fair • The new drug is no better or worse than a placebo • The observed CellTitreGlo signal for my RNAi-treated cells is no different from that of the negative controls Example Toss a coin a certain number of times ⇒ If the coin is fair, then heads should appear half of the time (roughly). ! But what is “roughly”? We use combinatorics / probability theory to quantify this. ! For example, in 12 tosses with success rate p, the probability of seeing exactly 8 heads is Binomial Distribution H0 here: p = 0.5. Distribution of number of heads: 0.20 0.15 p 0.10 0.05 0.00 0 1 2 3 4 5 6 7 8 9 10 11 12 P(Heads ≤ 2) = 0.0193 n P(Heads ≥ 10) = 0.0193 Significance Level If H0 is true and the coin is fair (p=0.5), it is improbable to observe extreme events such as more than 9 heads 0.0193 = P(Heads ≥ 10 | H0 ) = “p-value” If we observe 10 heads in a trial, the null hypothesis is likely to be false.
    [Show full text]
  • 8. Multiple Test Corrections
    P-values and statistical tests 8. Multiple test corrections Marek Gierliński Division of Computational Biology Hand-outs available at http://is.gd/statlec 2 Gene expression experiment Differential expression: compare gene expression in two conditions (e.g. by t-test) gene_id p.value 1 GENE00001 0.040503700 1 in 20 chance of a false positive 2 GENE00002 0.086340732 3 GENE00003 0.552768467 4 GENE00004 0.379025917 5 GENE00005 0.990102618 6 GENE00006 0.182729903 7 GENE00007 0.923285031 8 GENE00008 0.938615285 9 GENE00009 0.431912336 10 GENE00010 0.822697032 10,000 genes 11 GENE00011 0.004911421 12 GENE00012 0.873463918 13 GENE00013 0.481156679 ~500 genes are “significant” by chance 14 GENE00014 0.442047456 15 GENE00015 0.794117108 16 GENE00016 0.214535451 17 GENE00017 0.231943488 18 GENE00018 0.980911106 19 GENE00019 0.422162464 20 GENE00020 0.915841637 ... 3 Even unlikely result will eventually happen if you repeat your test many times 4 Lets perform a test ! times False positives True positives Reality No Effect Total effect Number of Significant "# $# % discoveries Not $& "& ! − % significant Test result Total !( !) ! Number of tests True negatives False negatives 5 Family-wise error rate !"#$ = Pr(!) ≥ 1) 6 Probability of winning at least once Play lottery Probability of winning is ! Events Probability ! ! " 1 − ! Events Probability !! !×! !" !×(1 − !) "! (1 − !)×! "" 1 − ! ' Probability of not winning at all Probability of winning at least once 1 − 1 − ! ' 7 False positive probability H0: no effect Set ! = 0.05 One test Probability of having a false positive &' = ! Two independent tests Probability of having at least one false positive in either test ( &( = 1 − 1 − ! + independent tests Probability of having at least one false positive in any test , &, = 1 − 1 − ! 8 Family-wise error rate (FWER) Probability of having at least one false positive among ! tests; " = 0.05 ( '( = 1 − 1 − " Jelly beans test, ! = 20, " = 0.05 -.
    [Show full text]
  • Hypothesis Testing (1/22/13)
    STA613/CBB540: Statistical methods in computational biology Hypothesis Testing (1/22/13) Lecturer: Barbara Engelhardt Scribe: Lisa Cervia 1. Hypothesis Testing 1.1. Classical. In classical hypothesis testing we are concerned with evaluating the probability of the data given that a hypothesis is true: P (DjH = t) In Bayesian hypothesis testing, we instead consider the probability that the hypothesis is true given a set of data: P (H = tjD) Throughout both paradigms, we compare the null hypothesis, or one specific model of the data, with an alternative hypothesis, a second possible model of the data. We are interested in detecting data points for which the alternative model is a significantly better at explaining the data than the null model: Null hypothesis: H0 Alternative hypothesis: HA So, as an example, consider the hypothesis that, for two nucleotide sequences of length n, that they are somehow are homologous at a higher rate than expected by chance (assuming the probability 1 of a nucleotide match in the null hypothesis is 4 .) 1 Null, H : X ∼ Binomial(n; p = ) 0 4 1 One sided test, H : X ∼ Binomial(n; p < ) A 4 1 Two sided test, H : X ∼ Binomial(n; p 6= ). A 4 1.2. Test Statistic. Let t(x) be the threshold of a test statistic evaluated on m data points, or features, where x = fx1; :::; xmg, and let m be the number of tests that are performed. For those values that are more extreme than the threshold t(x), we are going to call those feature significant, and for them we will say that we reject the null hypothesis in favor of the alternative hypothesis.
    [Show full text]
  • False Discovery Rate Control Is a Recommended Alternative to Bonferroni-Type Adjustments in Health Studies Mark E
    Author's personal copy Journal of Clinical Epidemiology 67 (2014) 850e857 REVIEW ARTICLES False discovery rate control is a recommended alternative to Bonferroni-type adjustments in health studies Mark E. Glickmana,b,*, Sowmya R. Raoa,c, Mark R. Schultza aCenter for Health care Organization and Implementation Research, Bedford VA Medical Center, 200 Springs Road (152), Bedford, MA 01730, USA bDepartment of Health Policy and Management, Boston University School of Public Health, 715 Albany Street, Talbot Building, Boston, MA 02118, USA cDepartment of Quantitative Health Sciences, University of Massachusetts Medical School, 55 N. Lake Avenue, Worcester, MA 01655, USA Accepted 12 March 2014; Published online 13 May 2014 Abstract Objectives: Procedures for controlling the false positive rate when performing many hypothesis tests are commonplace in health and medical studies. Such procedures, most notably the Bonferroni adjustment, suffer from the problem that error rate control cannot be local- ized to individual tests, and that these procedures do not distinguish between exploratory and/or data-driven testing vs. hypothesis-driven testing. Instead, procedures derived from limiting false discovery rates may be a more appealing method to control error rates in multiple tests. Study Design and Setting: Controlling the false positive rate can lead to philosophical inconsistencies that can negatively impact the practice of reporting statistically significant findings. We demonstrate that the false discovery rate approach can overcome these inconsis- tencies and illustrate its benefit through an application to two recent health studies. Results: The false discovery rate approach is more powerful than methods like the Bonferroni procedure that control false positive rates. Controlling the false discovery rate in a study that arguably consisted of scientifically driven hypotheses found nearly as many sig- nificant results as without any adjustment, whereas the Bonferroni procedure found no significant results.
    [Show full text]
  • How Should We Use Hypothesis Tests to Guide the Next Experiment?
    Many tests, many strata: How should we use hypothesis tests to guide the next experiment? Jake Bowers Nuole Chen University of Illinois @ Urbana-Champaign University of Illinois @ Urbana-Champaign Departments of Political Science & Department of Political Science Statistics Associate Fellow, Office of Evaluation Fellow, Office of Evaluation Sciences, GSA Sciences, GSA Methods Director, EGAP https://publish.illinois.edu/nchen3 Fellow, CASBS http://jakebowers.org 25 January 2019 The Plan Motivation and Application: An experiment with many blocks raises questions about where to focus the next experiment Background: Hypothesis testing and error for many tests Adjustment based methods for controlling error rate. Limitations and Advantages of Different Approaches 1/23 Motivation and Application: An experiment with many blocks raises questions about where to focus the next experiment A (Fake) Field Experiment ”...We successfully randomized the college application communication to people within each of the 100 HUD buildings. The estimated effect of the new policy is 5 (p < .001).” ● 120 ● ● ● ● ● 100 ● ● ● ● ● ● ● ● ● ● ● ● ● ● 80 ● ● ● ● ● ● 60 40 20 0 2/23 0 1 people in just a few buildings? Policy Expert: ...Great! But this seems small. Could it be mainly effecting A (Fake) Field Experiment 60 50 80 50 40 40 60 30 30 40 20 20 20 10 10 0 1 0 1 0 1 ● ● 70 70 ● 80 ● 60 60 50 50 60 40 40 40 30 30 20 20 20 10 10 0 0 0 1 0 1 0 1 ● ● 80 100 80 ● 80 60 60 ● 60 40 40 40 20 20 20 0 0 1 0 1 0 1 3/23 Jake: This is a natural and important question both for the development of new theory and better policy.
    [Show full text]
  • Multiple Comparisons: Problem and Solutions
    Multiple comparisons: problem and solutions Gareth Barnes Wellcome Centre for Human Neuroimaging University College London SPM M/EEG Course By the end of this talk Understand what the multiple comparisons problem is. Be familiar with some common approaches. Be able to explain Random field theory. Contrast c General Pre- Statistical Linear processing Inference Model Is this real ? Need to avoid cherry-picking. i.e. need an objective threshold. Need to adjust this threshold depending on how many independent tests you do. (e.g. if you do 20 tests with a false positive rate of 1/20 .. then expect one peak just due to chance). T test on a single voxel / time point T value Test only this 1.66 Task A T distribution Task B P=0.05 u Threshold Multiple tests in space u u u u u If we have 100,000 voxels, u α=0.05 5,000 false positive voxels. This is clearly undesirable; to correct t for this we can define a null hypothesis t t for a collection of tests. t t t signal Use of ‘uncorrected’ p-value, α =0.1 11.3% 11.3% 12.5% 10.8% 11.5% 10.0% 10.7% 11.2% 10.2% 9.5% Percentage of Null Pixels that are False Positives Bonferroni correction Set the test-wise error rate (α) to be a the ideal Family-Wise Error rate (FWER) (αFWE) divided by the number of tests. e.g. for five tests choose p<0.01 for each test to control family at p<0.05 This correction does not require the tests to be independent but becomes very stringent if they are not.
    [Show full text]