DOCSLIB.ORG

Aligned Rank Transform for Nonparametric Factorial Analyses Using Only ANOVA Procedures Jacob O

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Aligned Rank Transform for Nonparametric Factorial Analyses Using Only ANOVA Procedures Jacob O CHI 2011 • Session: Research Methods May 7–12, 2011 • Vancouver, BC, Canada The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only ANOVA Procedures Jacob O. Wobbrock,1 Leah Findlater,1 Darren Gergle,2 James J. Higgins3 1The Information School 2School of Communication 3Department of Statistics DUB Group Northwestern University Kansas State University University of Washington Evanston, IL 60208 USA Manhattan, KS 66506 USA Seattle, WA 98195 USA [email protected] [email protected] wobbrock, [email protected] ABSTRACT some methods for handling nonparametric factorial data Nonparametric data from multi-factor experiments arise exist (see Table 1) [12], most are not well known to HCI often in human-computer interaction (HCI). Examples may researchers, require advanced statistical knowledge, and are include error counts, Likert responses, and preference not included in common statistics packages.1 tallies. But because multiple factors are involved, common Table 1. Some possible analyses for nonparametric data. nonparametric tests (e.g., Friedman) are inadequate, as they Method Limitations are unable to examine interaction effects. While some statistical techniques exist to handle such data, these General Linear Models Can perform factorial parametric analyses, (GLM) but cannot perform nonparametric analyses. techniques are not widely available and are complex. To address these concerns, we present the Aligned Rank Mann-Whitney U, Can perform nonparametric analyses, but Kruskal-Wallis cannot handle repeated measures or analyze Transform (ART) for nonparametric factorial data analysis multiple factors or interactions. in HCI. The ART relies on a preprocessing step that “aligns” Wilcoxon, Friedman Can perform nonparametric analyses and data before applying averaged ranks, after which point handle repeated measures, but cannot common ANOVA procedures can be used, making the ART analyze multiple factors or interactions. accessible to anyone familiar with the F-test. Unlike most χ2, Logistic Regression, Can perform factorial nonparametric articles on the ART, which only address two factors, we Generalized Linear analyses, but cannot handle repeated generalize the ART to N factors. We also provide ARTool Models (GZLM) measures. and ARTweb, desktop and Web-based programs for aligning Generalized Linear Can perform factorial nonparametric analyses and ranking data. Our re-examination of some published Mixed Models (GLMM), and handle repeated measures, but are not HCI results exhibits advantages of the ART. Generalized Estimating widely available and are complex. Equations (GEE) Author Keywords: Statistics, analysis of variance, ANOVA, factorial analysis, nonparametric data, F-test. Kaptein et al.’s [7] Can perform factorial nonparametric analyses nonparametric method and handle repeated measures, but requires ACM Classification Keywords: H.5.2 [Information different mathematics and software modules interfaces and presentation]: User interfaces – for each type of experiment design. ([7] evaluation/methodology, theory and methods. focused only on 2×2 mixed designs.) General Terms: Experimentation, Measurement, Theory. Aligned Rank Transform Can perform factorial nonparametric analyses (ART) and handle repeated measures. Requires INTRODUCTION only an ANOVA after data alignment and Studies in human-computer interaction (HCI) often ranking, provided for by ARTool or ARTweb. generate nonparametric data from multiple independent variables. Examples of such data may include error counts, A remedy to the paucity of nonparametric factorial analyses Likert responses, or preference tallies. Often complicating would be a procedure that retains the familiarity and this picture are correlated data arising from repeated interpretability of the familiar parametric F-test. We present measures. Two analysis approaches for data like these just such an analysis called the Aligned Rank Transform appear regularly in the HCI literature. The first simply uses (ART). The ART relies on an alignment and ranking step before using F-tests. We offer two equivalent tools to do the a parametric F-test, which risks violating ANOVA assumptions and inflating Type I error rates. The second alignment and ranking, one for the desktop (ARTool) and uses common one-way nonparametric tests (e.g., Friedman), one on the Web (ARTweb). Unlike the advanced methods foregoing the examination of interaction effects. Although 1 A useful page from statistics consulting at UCLA for choosing the right Permission to make digital or hard copies of all or part of this work for statistical test has a conspicuous omission marked with “???”, which personal or classroom use is granted without fee provided that copies are appears in the slot for analyzing factorial ordinal data. See not made or distributed for profit or commercial advantage and that copies http://www.ats.ucla.edu/stat/mult_pkg/whatstat/default.htm. Similarly, bear this notice and the full citation on the first page. To copy otherwise, another useful page giving the rationale for myriad statistical tests fails to or republish, to post on servers or to redistribute to lists, requires prior describe any nonparametric factorial analysis for repeated measures data. specific permission and/or a fee. See http://www.ats.ucla.edu/stat/spss/whatstat/whatstat.htm. Both pages CHI 2011, May 7–12, 2011, Vancouver, BC, Canada. were accessed January 12, 2011. Copyright 2011 ACM 978-1-4503-0267-8/11/05....$10.00. 143 CHI 2011 • Session: Research Methods May 7–12, 2011 • Vancouver, BC, Canada shown in Table 1, researchers only familiar with ANOVA with matching levels of independent variables (X1 and X2). can use, interpret, and report results from the ART. Thus, the cell mean in row 1 is the mean Y of s1 and s5. We describe the ART and contribute: (1) its generalization Table 2. Example calculation of cell means. from two factors to N factors, (2) the ARTool and ARTweb Subject X1 X2 Y cell mean programs for easy alignment and ranking, and (3) a re- s1 a a 12 (12+19)/2 examination of some published HCI data. s2 a b 7 (7+16)/2 s3 b a 14 (14+14)/2 THE ALIGNED RANK TRANSFORM FOR N FACTORS s4 b b 8 (8+10)/2 This section describes the Aligned Rank Transform, s5 a a 19 (12+19)/2 s6 a b 16 (7+16)/2 generalizing it to an arbitrary number of factors. The ART is s7 b a 14 (14+14)/2 for use in circumstances similar to the parametric ANOVA, s8 b b 10 (8+10)/2 except that the response variable may be continuous or ordinal, and is not required to be normally distributed. Step 2. Compute estimated effects for all main and interaction effects. Let A, B, C, and D be factors with Background levels Ai, i = 1..a; Bj, j = 1..b; Ck, k = 1..c; Dℓ, ℓ = 1..d. Rank transformations have appeared in statistics for years Let be the mean response Y for rows where factor A is [12]. Conover and Iman’s [1] Rank Transform (RT) applies Ai i ranks, averaged in the case of ties, over a data set, and then at level i. Let ABi j be the mean response Yij for rows where uses the parametric F-test on the ranks, resulting in a factor A is at level i and factor B is at level j. And so on. Let nonparametric factorial procedure. However, it was µ be the grand mean of Y over all rows. subsequently discovered that this process produces inaccurate results for interaction effects [5,11], making the One-way effects. The estimated main effect for a factor A RT method unsuitable for factorial designs. with response Yi is The Aligned Rank Transform (ART) [2,10] corrects this Ai . problem, providing accurate nonparametric treatment for Two-way effects. The estimated effect for an A*B both main and interaction effects [4,5,11]. It relies on a interaction with response Y is preprocessing step that first “aligns” the data for each effect ij (main or interaction) before assigning ranks, averaged in ABAB . i j i j the case of ties. Data alignment is an established process in statistics [6] by which effects are estimated as marginal Three-way effects. The estimated effect for an A*B*C means and then “stripped” from the response variable so interaction with response Yijk is that all effects but one are removed. ABCABACBCABC . ijk ij ik jk i j k Let’s consider an example: In a two-factor experiment with Four-way effects. The estimated effect for an A*B*C*D effects A, B, and A*B, and response Y, testing for the interaction with response Y is significance of effect A means first stripping from Y ijkℓ ABCDABCABDACDBCD estimates of effects B and A*B, leaving only a possible i j k i j k i j i k j k effect of A behind. This alignment results in YA′, whose ABACADBCBDCD i j i k i j k j k values are then ranked, producing YA″. Lastly, a full- ABCD . factorial ANOVA is run with YA″ as the response and model i j k terms A, B, and A*B, but importantly, only the effect of A is N-way effects. The generalized estimated effect for an N- examined in the ANOVA table; B and A*B are ignored. This way interaction is process is then repeated for the effects of B and A*B, i.e., on aligned ranks YB″ and YA*B″, respectively. Thus, to use N way the ART, responses Y from a study must be aligned and N-1 way N-2 way N-3 way N-4 way ranked for each effect of interest. This is a tedious process to do by hand, but ARTool or ARTweb make it easy. ... N-h way // if h is odd, or ART Procedure for N Factors We present the ART procedure in five steps: N-h way // if h is even Step 1. Compute residuals. For each raw response Y, ..
Load more

Recommended publications
  • 14. Non-Parametric Tests You Know the Story. You've Tried for Hours To
    14. Non-Parametric Tests You know the story. You’ve tried for hours to normalise your data. But you can’t. And even if you can, your variances turn out to be unequal anyway. But do not dismay. There are statistical tests that you can run with non-normal, heteroscedastic data. All of the tests that we have seen so far (e.g. t-tests, ANOVA, etc.) rely on comparisons with some probability distribution such as a t-distribution or f-distribution. These are known as “parametric statistics” as they make inferences about population parameters. However, there are other statistical tests that do not have to meet the same assumptions, known as “non-parametric tests.” Many of these tests are based on ranks and/or are comparisons of sample medians rather than means (as means of non-normal data are not meaningful). Non-parametric tests are often not as powerful as parametric tests, so you may find that your p-values are slightly larger. However, if your data is highly non-normal, non-parametric test may detect an effect that is missed in the equivalent parametric test due to falsely large variances caused by the skew of the data. There is, of course, no problem in analyzing normally distributed data with non-parametric statistics also. Non-parametric tests are equally valid (i.e. equally defendable) and most have been around as long as their parametric counterparts. So, if your data does not meet the assumption of normality or homoscedasticity and if it is impossible (or inappropriate) to transform it, you can use non-parametric tests.
    [Show full text]
  • Statistical Analysis in JASP
    Copyright © 2018 by Mark A Goss-Sampson. All rights reserved. This book or any portion thereof may not be reproduced or used in any manner whatsoever without the express written permission of the author except for the purposes of research, education or private study. CONTENTS PREFACE .................................................................................................................................................. 1 USING THE JASP INTERFACE .................................................................................................................... 2 DESCRIPTIVE STATISTICS ......................................................................................................................... 8 EXPLORING DATA INTEGRITY ................................................................................................................ 15 ONE SAMPLE T-TEST ............................................................................................................................. 22 BINOMIAL TEST ..................................................................................................................................... 25 MULTINOMIAL TEST .............................................................................................................................. 28 CHI-SQUARE ‘GOODNESS-OF-FIT’ TEST............................................................................................. 30 MULTINOMIAL AND Χ2 ‘GOODNESS-OF-FIT’ TEST. ..........................................................................
    [Show full text]
  • Friedman Test
    community project encouraging academics to share statistics support resources All stcp resources are released under a Creative Commons licence stcp-marquier-FriedmanR The following resources are associated: The R dataset ‘Video.csv’ and R script ‘Friedman.R’ Friedman test in R (Non-parametric equivalent to repeated measures ANOVA) Dependent variable: Continuous (scale) but not normally distributed or ordinal Independent variable: Categorical (Time/ Condition) Common Applications: Used when several measurements of the same dependent variable are taken at different time points or under different conditions for each subject and the assumptions of repeated measures ANOVA have not been met. It can also be used to compare ranked outcomes. Data: The dataset ‘Video’ contains some results from a study comparing videos made to aid understanding of a particular medical condition. Participants watched three videos (A, B, C) and one product demonstration (D) and were asked several Likert style questions about each. These were summed to give an overall score for each e.g. TotalAGen below is the total score of the ordinal questions for video A. Friedman ranks each participants responses The Friedman test ranks each person’s score from lowest to highest (as if participants had been asked to rank the methods from least favourite to favourite) and bases the test on the sum of ranks for each column. For example, person 1 gave C the lowest Total score of 13 and A the highest so SPSS would rank these as 1 and 4 respectively. As the raw data is ranked to carry out the test, the Friedman test can also be used for data which is already ranked e.g.
    [Show full text]
  • Compare Multiple Related Samples
    Compare Multiple Related Samples The command compares multiple related samples using the Friedman test (nonparametric alternative to the one-way ANOVA with repeated measures) and calculates the Kendall's coefficient of concordance (also known as Kendall’s W). Kendall's W makes no assumptions about the underlying probability distribution and allows to handle any number of outcomes, unlike the standard Pearson correlation coefficient. Friedman test is similar to the Kruskal-Wallis one-way analysis of variance with the difference that Friedman test is an alternative to the repeated measures ANOVA with balanced design. How To For unstacked data (each column is a sample): o Run the STATISTICS->NONPARAMETRIC STATISTICS -> COMPARE MULTIPLE RELATED SAMPLES [FRIEDMAN ANOVA, CONCORDANCE] command. o Select variables to compare. For stacked data (with a group variable): o Run the STATISTICS->NONPARAMETRIC STATISTICS -> COMPARE MULTIPLE RELATED SAMPLES (WITH GROUP VARIABLE) command. o Select a variable with observations (VARIABLE) and a text or numeric variable with the group names (GROUPS). RESULTS The report includes Friedman ANOVA and Kendall’s W test results. THE FRIEDMAN ANOVA tests the null hypothesis that the samples are from identical populations. If the p-value is less than the selected 훼 level the null-hypothesis is rejected. If there are no ties, Friedman test statistic Ft is defined as: 푘 12 퐹 = [ ∑ 푅2] − 3푛(푘 + 1) 푡 푛푘(푘 + 1) 푖 푖=1 where n is the number of rows, or subjects; k is the number of columns or conditions, and Ri is the sum of the ranks of ith column. If ranking results in any ties, the Friedman test statistic Ft is defined as: 푅2 푛(푘 − 1) [∑푘 푖 − 퐶 ] 푖=1 푛 퐹 퐹푡 = 2 ∑ 푟푖푗 − 퐶퐹 where n is the number rows, or subjects, k is the number of columns, and Ri is the sum of the ranks from column, or condition I; CF is the ties correction (Corder et al., 2009).
    [Show full text]
  • Repeated-Measures ANOVA & Friedman Test Using STATCAL
    Repeated-Measures ANOVA & Friedman Test Using STATCAL (R) & SPSS Prana Ugiana Gio Download STATCAL in www.statcal.com i CONTENT 1.1 Example of Case 1.2 Explanation of Some Book About Repeated-Measures ANOVA 1.3 Repeated-Measures ANOVA & Friedman Test 1.4 Normality Assumption and Assumption of Equality of Variances (Sphericity) 1.5 Descriptive Statistics Based On SPSS dan STATCAL (R) 1.6 Normality Assumption Test Using Kolmogorov-Smirnov Test Based on SPSS & STATCAL (R) 1.7 Assumption Test of Equality of Variances Using Mauchly Test Based on SPSS & STATCAL (R) 1.8 Repeated-Measures ANOVA Based on SPSS & STATCAL (R) 1.9 Multiple Comparison Test Using Boferroni Test Based on SPSS & STATCAL (R) 1.10 Friedman Test Based on SPSS & STATCAL (R) 1.11 Multiple Comparison Test Using Wilcoxon Test Based on SPSS & STATCAL (R) ii 1.1 Example of Case For example given data of weight of 11 persons before and after consuming medicine of diet for one week, two weeks, three weeks and four week (Table 1.1.1). Tabel 1.1.1 Data of Weight of 11 Persons Weight Name Before One Week Two Weeks Three Weeks Four Weeks A 89.43 85.54 80.45 78.65 75.45 B 85.33 82.34 79.43 76.55 71.35 C 90.86 87.54 85.45 80.54 76.53 D 91.53 87.43 83.43 80.44 77.64 E 90.43 84.45 81.34 78.64 75.43 F 90.52 86.54 85.47 81.44 78.64 G 87.44 83.34 80.54 78.43 77.43 H 89.53 86.45 84.54 81.35 78.43 I 91.34 88.78 85.47 82.43 78.76 J 88.64 84.36 80.66 78.65 77.43 K 89.51 85.68 82.68 79.71 76.5 Average 89.51 85.68 82.68 79.71 76.69 Based on Table 1.1.1: The person whose name is A has initial weight 89,43, after consuming medicine of diet for one week 85,54, two weeks 80,45, three weeks 78,65 and four weeks 75,45.
    [Show full text]
  • Chapter 4. Data Analysis
    4. DATA ANALYSIS Data analysis begins in the monitoring program group of observations selected from the target design phase. Those responsible for monitoring population. In the case of water quality should identify the goals and objectives for monitoring, descriptive statistics of samples are monitoring and the methods to be used for used almost invariably to formulate conclusions or analyzing the collected data. Monitoring statistical inferences regarding populations objectives should be specific statements of (MacDonald et al., 1991; Mendenhall, 1971; measurable results to be achieved within a stated Remington and Schork, 1970; Sokal and Rohlf, time period (Ponce, 1980b). Chapter 2 provides 1981). A point estimate is a single number an overview of commonly encountered monitoring representing the descriptive statistic that is objectives. Once goals and objectives have been computed from the sample or group of clearly established, data analysis approaches can observations (Freund, 1973). For example, the be explored. mean total suspended solids concentration during baseflow is 35 mg/L. Point estimates such as the Typical data analysis procedures usually begin mean (as in this example), median, mode, or with screening and graphical methods, followed geometric mean from a sample describe the central by evaluating statistical assumptions, computing tendency or location of the sample. The standard summary statistics, and comparing groups of data. deviation and interquartile range could likewise be The analyst should take care in addressing the used as point estimates of spread or variability. issues identified in the information expectations report (Section 2.2). By selecting and applying The use of point estimates is warranted in some suitable methods, the data analyst responsible for cases, but in nonpoint source analyses point evaluating the data can prevent the “data estimates of central tendency should be coupled rich)information poor syndrome” (Ward 1996; with an interval estimate because of the large Ward et al., 1986).
    [Show full text]
  • 9 Blocked Designs
    9 Blocked Designs 9.1 Friedman's Test 9.1.1 Application 1: Randomized Complete Block Designs • Assume there are k treatments of interest in an experiment. In Section 8, we considered the k-sample Extension of the Median Test and the Kruskal-Wallis Test to test for any differences in the k treatment medians. • Suppose the experimenter is still concerned with studying the effects of a single factor on a response of interest, but variability from another factor that is not of interest is expected. { Suppose a researcher wants to study the effect of 4 fertilizers on the yield of cotton. The researcher also knows that the soil conditions at the 8 areas for performing an experiment are highly variable. Thus, the researcher wants to design an experiment to detect any differences among the 4 fertilizers on the cotton yield in the presence a \nuisance variable" not of interest (the 8 areas). • Because experimental units can vary greatly with respect to physical characteristics that can also influence the response, the responses from experimental units that receive the same treatment can also vary greatly. • If it is not controlled or accounted for in the data analysis, it can can greatly inflate the experimental variability making it difficult to detect real differences among the k treatments of interest (large Type II error). • If this source of variability can be separated from the treatment effects and the random experimental error, then the sensitivity of the experiment to detect real differences between treatments in increased (i.e., lower the Type II error). • Therefore, the goal is to choose an experimental design in which it is possible to control the effects of a variable not of interest by bringing experimental units that are similar into a group called a \block".
    [Show full text]
  • Basic Statistical Analysis Aov(), Anova(), Lm(), Glm(): Linear And
    basic statistical analysis R reference card, by Jonathan Baron aov(), anova(), lm(), glm(): linear and non- Parentheses are for functions, brackets are for indi- linear models, anova cating the position of items in a vector or matrix. t.test(): t test (Here, items with numbers like x1 are user-supplied variables.) prop.test(), binom.test(): sign test chisq.test(x1): chi-square test on matrix x1 Miscellaneous fisher.test(): Fisher exact test q(): quit cor(a): show correlations <-: assign cor.test(a,b): test correlation INSTALL package1: install package1 friedman.test(): Friedman test m1[,2]: column 2 of matrix m1 m1[,2:5] or m1[,c(2,3,4,5)]: columns 2–5 some statistics in mva package m1$a1: variable a1 in data frame m1 prcomp(): principal components NA: missing data kmeans(): kmeans cluster analysis is.na: true if data missing factanal(): factor analysis library(mva): load (e.g.) the mva package cancor(): canonical correlation Help Graphics help(command1): get help with command1 (NOTE: plot(), barplot(), boxplot(), stem(), hist(): USE THIS FOR MORE DETAIL THAN THIS basic plots CARD CAN PROVIDE.) matplot(): matrix plot help.start(): start browser help pairs(matrix): scatterplots help(package=mva): help with (e.g.) package mva coplot(): conditional plot apropos("topic1"): commands relevant to topic1 stripchart(): strip chart example(command1): examples of command1 qqplot(): quantile-quantile plot Input and output qqnorm(), qqline(): fit normal distribution source("file1"): run the commands in file1. read.table("file1"): read in data from file1
    [Show full text]
  • 12.9 Friedman Rank Test: Nonparametric Analysis for The
    M12_BERE8380_12_SE_C12.9.qxd 2/21/11 3:56 PM Page 1 12.9 Friedman Rank Test: Nonparametric Analysis for the Randomized Block Design 1 12.9 Friedman Rank Test: Nonparametric Analysis for the Randomized Block Design When analyzing a randomized block design, sometimes the data consist of only the ranks within each block. Other times, you cannot assume that the data from each of the c groups are from normally distributed populations. In these situations, you can use the Friedman rank test. You use the Friedman rank test to determine whether c groups have been selected from populations having equal medians. That is, you test = = Á = H0: M.1 M.2 M.c against the alternative = Á H1: Not all M.j are equal (where j 1, 2, , c). To conduct the test, you replace the data values in each of the r independent blocks with the corresponding ranks, so that you assign rank 1 to the smallest value in the block and rank c to the largest. If any values in a block are tied, you assign them the mean of the ranks that they would otherwise have been assigned. Thus, Rij is the rank (from 1 to c) associated with the jth group in the ith block. Equation (12.13) defines the test statistic for the Friedman rank test. FRIEDMAN RANK TEST FOR DIFFERENCES AMONG C MEDIANS c = 12 2 - + FR + a R.j 3r(c 1) (12.13) rc(c 1) j=1 where 2 = = Á R.j square of the total of the ranks for group j ( j 1, 2, , c) r = number of blocks c = number of groups As the number of blocks gets large (i.e., greater than 5), you can approximate the test sta- - tistic FR by using the chi-square distribution with c 1 degrees of freedom.
    [Show full text]
  • Sample Size Calculation with Gpower
    Sample Size Calculation with GPower Dr. Mark Williamson, Statistician Biostatistics, Epidemiology, and Research Design Core DaCCoTA Purpose ◦ This Module was created to provide instruction and examples on sample size calculations for a variety of statistical tests on behalf of BERDC ◦ The software used is GPower, the premiere free software for sample size calculation that can be used in Mac or Windows https://dl2.macupdate.com/images/icons256/24037.png Background ◦ The Biostatistics, Epidemiology, and Research Design Core (BERDC) is a component of the DaCCoTA program ◦ Dakota Cancer Collaborative on Translational Activity has as its goal to bring together researchers and clinicians with diverse experience from across the region to develop unique and innovative means of combating cancer in North and South Dakota ◦ If you use this Module for research, please reference the DaCCoTA project The Why of Sample Size Calculation ◦ In designing an experiment, a key question is: How many animals/subjects do I need for my experiment? ◦ Too small of a sample size can under-detect the effect of interest in your experiment ◦ Too large of a sample size may lead to unnecessary wasting of resources and animals ◦ Like Goldilocks, we want our sample size to be ‘just right’ ◦ The answer: Sample Size Calculation ◦ Goal: We strive to have enough samples to reasonably detect an effect if it really is there without wasting limited resources on too many samples. https://upload.wikimedia.org/wikipedia/commons/thumb/e/ef/The_Three_Bears_- _Project_Gutenberg_eText_17034.jpg/1200px-The_Three_Bears_-_Project_Gutenberg_eText_17034.jpg
    [Show full text]
  • BMC Immunology Biomed Central
    BMC Immunology BioMed Central Commentary Open Access A guide to modern statistical analysis of immunological data Bernd Genser*1,2, Philip J Cooper3,4, Maria Yazdanbakhsh5, Mauricio L Barreto2 and Laura C Rodrigues2 Address: 1Department of Epidemiology and Population Health, London School of Hygiene and Tropical Medicine, London, UK, 2Instituto de Saúde Coletiva, Federal University of Bahia, Salvador, Brazil, 3Centre for Infection, St George's University of London, London, UK, 4Instituto de Microbiologia, Universidad San Francisco de Quito, Quito, Ecuador and 5Department of Parasitology, Leiden University Medical Center, Leiden, The Netherlands Email: Bernd Genser* - [email protected]; Philip J Cooper - [email protected]; Maria Yazdanbakhsh - [email protected]; Mauricio L Barreto - [email protected]; Laura C Rodrigues - [email protected] * Corresponding author Published: 26 October 2007 Received: 22 February 2007 Accepted: 26 October 2007 BMC Immunology 2007, 8:27 doi:10.1186/1471-2172-8-27 This article is available from: http://www.biomedcentral.com/1471-2172/8/27 © 2007 Genser et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Background: The number of subjects that can be recruited in immunological studies and the number of immunological parameters that can be measured has increased rapidly over the past decade and is likely to continue to expand. Large and complex immunological datasets can now be used to investigate complex scientific questions, but to make the most of the potential in such data and to get the right answers sophisticated statistical approaches are necessary.
    [Show full text]
  • Exact P-Values for Pairwise Comparison of Friedman Rank Sums
    Eisinga et al. BMC Bioinformatics (2017) 18:68 DOI 10.1186/s12859-017-1486-2 METHODOLOGY ARTICLE Open Access Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers Rob Eisinga1* , Tom Heskes2, Ben Pelzer1 and Manfred Te Grotenhuis1 Abstract Background: The Friedman rank sum test is a widely-used nonparametric method in computational biology. In addition to examining the overall null hypothesis of no significant difference among any of the rank sums, it is typically of interest to conduct pairwise comparison tests. Current approaches to such tests rely on large-sample approximations, due to the numerical complexity of computing the exact distribution. These approximate methods lead to inaccurate estimates in the tail of the distribution, which is most relevant for p-value calculation. Results: We propose an efficient, combinatorial exact approach for calculating the probability mass distribution of the rank sum difference statistic for pairwise comparison of Friedman rank sums, and compare exact results with recommended asymptotic approximations. Whereas the chi-squared approximation performs inferiorly to exact computation overall, others, particularly the normal, perform well, except for the extreme tail. Hence exact calculation offers an improvement when small p-values occur following multiple testing correction. Exact inference also enhances the identification of significant differences whenever the observed values are close to the approximate critical value. We illustrate the proposed method in the context of biological machine learning, were Friedman rank sum difference tests are commonly used for the comparison of classifiers over multiple datasets. Conclusions: We provide a computationally fast method to determine the exact p-value of the absolute rank sum difference of a pair of Friedman rank sums, making asymptotic tests obsolete.
    [Show full text]