DOCSLIB.ORG

How to Properly Compare Deep Neural Models

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
How to Properly Compare Deep Neural Models Deep Dominance - How to Properly Compare Deep Neural Models Rotem Dror Segev Shlomov Roi Reichart Faculty of Industrial Engineering and Management, Technion, IIT rtmdrr|segevs|[email protected] Abstract was deterministic and the number of configura- tions a model could have was rather small – deci- Comparing between Deep Neural Network sions about model design were usually limited to (DNN) models based on their performance on unseen data is crucial for the progress of feature selection and the selection of one of a few the NLP field. However, these models have loss functions. Consequently, when one model a large number of hyper-parameters and, be- performed better than another on unseen data it ing non-convex, their convergence point de- was safe to argue that the winning model was gen- pends on the random values chosen at ini- erally better, especially when the results were sta- tialization and during training. Proper DNN tistically significant (Dror et al., 2018), and when comparison hence requires a comparison be- the effect of multiple hypothesis testing was taken tween their empirical score distributions on unseen data, rather than between single eval- into account in cases of evaluation with multiple uation scores as is standard for more simple, datasets (Dror et al., 2017). convex models. In this paper, we propose to With the recent emergence of Deep Neural Net- adapt to this problem a recently proposed test works (DNNs), data-driven performance compar- for the Almost Stochastic Dominance relation ison has become much more complicated. While between two distributions. We define the cri- models such as LSTM (Hochreiter and Schmid- teria for a high quality comparison method huber, 1997), Bi-LSTM (Schuster and Paliwal, between DNNs, and show, both theoretically 1997) and the transformer (Vaswani et al., 2017) and through analysis of extensive experimen- tal results with leading DNN models for se- improved the state-of-the-art in many NLP tasks quence tagging tasks, that the proposed test (e.g. (Dozat and Manning, 2017; Hershcovich meets all criteria while previously proposed et al., 2017; Yadav and Bethard, 2018)), it is methods fail to do so. We hope the test we much more difficult to compare the performance propose here will set a new working practice of algorithms that are based on these models. 1 in the NLP community. This is because the loss functions of these mod- 1 Introduction els are non-convex (Dauphin et al., 2014), mak- ing the solution to which they converge (a lo- A large portion of the research activity in Natural cal minimum or a saddle point) sensitive to ran- Language Processing (NLP) is devoted to the de- dom weight initialization and the order of train- velopment of new algorithms for existing or new ing examples. Moreover, as these complex mod- tasks. To evaluate the quality of a new method, its els are not fully understood, their training is often performance on unseen datasets is compared to the enhanced by heuristics such as random dropouts performance of existing methods. The progress of (Srivastava et al., 2014) that introduces another the field hence crucially depends on our ability to level of non-determinism to the training process. draw conclusions from such comparisons. Finally, the increased model complexity results in In the past, most supervised NLP models have a much larger number of configurations, governed been linear (or log-linear), convex and relatively by a large space of hyper-parameters for model simple (e.g. (Toutanova et al., 2003; Finkel et al., properties such as the number of layers and the 2008; Ritter et al., 2011)). Hence, their training number of neurons in each layer. 1Our code is available at: https://github.com/ With so many degrees of freedom governed by rtmdrr/deepComparison random and arbitrary values, when comparing two 2773 Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2773–2785 Florence, Italy, July 28 - August 2, 2019. c 2019 Association for Computational Linguistics DNNs it is not possible to consider a single test-set (c), since it does not provide information beyond evaluation score for each model. If we do that, we its decision if one of the distributions is stochas- might compare just the best models that someone tically dominant over the other or not, and as we happened to train rather than the methods them- show in x5 its power (criterion (b)) is very limited. selves. Instead, it is necessary to compare between A new comparison tool (x 4): We propose a the score distributions generated by different runs solution that meets our three criteria. Particu- of each of the models. Unfortunately, this compar- larly, we adapt to our problem the recently pre- ison task, which is fundamental to the progress of sented concept of Almost Stochastic Order (ASO) the field, has not received a systematic treatment between two distributions (Alvarez-Esteban´ et al., thus far. Our goal is to close this gap and propose a 2017),2 and the statistical significance test for this simple and effective comparison tool between two property, which makes very modest assumptions DNNs based on their test set score distributions. about the participating distributions (criterion (a)). Particularly, we make four contributions: Further, in line with criterion (c), the test returns a variable 2 [0; 1], that quantifies the degree to Defining a DNN comparison framework: We which one algorithm is stochastically larger than define three criteria that a DNN comparison tool the other, with = 0 reflecting stochastic order. should meet: (a) Since we observe only a sam- We further show that the test is designed to be very ple from the population score distribution of each powerful (criterion (b)), which is possible because model, the decision should be significant under the decision on the superior algorithm is comple- well justified statistical assumptions. This assures mented by the confidence score. that future runs of the superior model are likely Extensive experimental analysis (x 5): We re- to get higher scores than future runs of the infe- visit the extensive experimental setup of Reimers rior model; (b) The decision mechanism should and Gurevych(2017a,b), who performed 510 be powerful, being able to make decisions in most comparisons between strong DNN-based se- possible decision tasks; and, finally, (c) Since both quence tagging models. In each of their experi- models depend on random decisions, it is likely ments they compared two models – either different that none of them is promised to be superior over models or two variants of the same model differing the other in all cases (e.g. with all possible random in some of their hyper-parameters – and reported seeds). A powerful comparison tool should hence the score distributions of each model across vari- augment its decision with a confidence score, re- ous random seeds and hyper-parameter configura- flecting the probability that the superior model will tions. Our analysis reveals that while our test can indeed produce a better output. declare one of the algorithms superior in 100% of Analysis of existing solutions (x 3,5): The the cases, the COS approach can do that in 49.01% comparison problem we address has been high- of the cases, and the SO approach in a mere 0.98%. lighted by Reimers and Gurevych(2017b, 2018), In addition to being powerful, the decisions and who established its importance in an extensive the confidence scores of our proposed test are also experimentation with neural sequence models well aligned with the tests proposed in previous (Reimers and Gurevych, 2017a), and proposed literature: when the previous methods are chal- two main solutions (x3). One solution, which we lenged, our method still makes a decision but it refer to as the collection of statistics (COS) solu- also indicates the smaller gap between the algo- tion, is based on the analysis of statistics of the em- rithms. We hope that this work will establish a pirical score distribution of the two algorithms – standard for the comparison between DNNs. such as their mean, median and standard deviation (std), as well as their minimum and maximum val- 2 Performance Variance in DNNs ues. Unfortunately, this solution does not respect In this section we discuss the source of non- criterion (a) as it does not deal with significance, determinism in DNNs, focusing on hyper- and as we demonstrate in x5 its power (criterion parameter configurations and random choices. (b)) is also limited. Their second solution is based on significance testing for Stochastic Order (SO) Hyper-parameter Configurations DNNs are (Lehmann, 1955), a strict criterion that is hardly complex models governed by a variety of hyper- met in reality. While this solution respects crite- 2We use the terms Almost Stochastic Order and Almost rion (a), it is not designed to deal with criterion Stochastic Dominance interchangeably in this paper. 2774 parameters. A formal definition of a hyper- As discussed in x1, being non-convex, the con- parameter, differentiating it from a standard pa- vergent point of DNNs is deeply affected by these rameter, is usually a parameter whose value is random effects. Unfortunately, exploring all pos- set before the learning process begins. We can sible random seeds is impossible both because roughly say that hyper-parameters determine the they form an uncountable set and because their structure of the model and particular algorithmic values are uninterpretable and it is hence even hard decisions related, e.g., to its optimization. Some to decide on the relevant search space for their val- popular structure-related hyper-parameters in the ues. This dictates the need for reporting model re- DNN literature are the number of layers, layer sults with multiple random choices.
Load more

Recommended publications
  • Strength in Numbers: the Rising of Academic Statistics Departments In
    Agresti · Meng Agresti Eds. Alan Agresti · Xiao-Li Meng Editors Strength in Numbers: The Rising of Academic Statistics DepartmentsStatistics in the U.S. Rising of Academic The in Numbers: Strength Statistics Departments in the U.S. Strength in Numbers: The Rising of Academic Statistics Departments in the U.S. Alan Agresti • Xiao-Li Meng Editors Strength in Numbers: The Rising of Academic Statistics Departments in the U.S. 123 Editors Alan Agresti Xiao-Li Meng Department of Statistics Department of Statistics University of Florida Harvard University Gainesville, FL Cambridge, MA USA USA ISBN 978-1-4614-3648-5 ISBN 978-1-4614-3649-2 (eBook) DOI 10.1007/978-1-4614-3649-2 Springer New York Heidelberg Dordrecht London Library of Congress Control Number: 2012942702 Ó Springer Science+Business Media New York 2013 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer.
    [Show full text]
  • Erich Leo Lehmann, 19172009
    J. R. Statist. Soc. A (2010) 173, Part 3, pp. 683–689 Obituaries Erich Leo Lehmann, 1917–2009 Erich L. Lehmann, Professor Emeritus at the University of California, Berkeley, passed away on September 12th, 2009, aged 91 years. Erich was one of the engines that drove much of the development of theoretical and mainstream statistics during the second half of the 20th century. At the same time he kept himself aware of developments in applied statistics and prob- ability. He knew both the subject matter and the individuals developing our subject. He was a member of the powerful team of individuals that Jerzy Neyman built up at Berkeley in the 1950s. These included David Blackwell, Joe Hodges, Lucien Le Cam, Michel Loève, Henry Scheffé and Elizabeth Scott. Erich co-authored articles with each, except for the probabilist Loève. Erich was born in Strasbourg, France, on November 20th, 1917. His family moved to Frank- furt where they lived until 1933. When the Nazis came into power the family fled to Switzerland where Erich went to high school. In 1938, following his father’s advice, he went to study math- ematics at Trinity College, Cambridge. Erich remarked that he was ‘always the best student in mathematics’, but he did not enjoy the accompanying astronomy and physics. Of the latter he said ‘I hated it’ (in ‘A conversation with Erich L. Lehmann’, with Morris DeGroot, in volume 1 of Statistical Science (1986)). In 1940, again influenced by his father, Erich went to New York and then following a suggestion by R. Courant he crossed the country to study at Berkeley.
    [Show full text]
  • Peeking at A/B Tests Why It Ma‚Ers, and What to Do About It
    KDD 2017 Applied Data Science Paper KDD’17, August 13–17, 2017, Halifax, NS, Canada Peeking at A/B Tests Why it maers, and what to do about it Ramesh Johari∗ Pete Koomen† Stanford University Optimizely, Inc. [email protected] Leonid Pekelis‡ David Walsh§ Optimizely, Inc. Stanford University [email protected] [email protected] ABSTRACT obtain a very simple “user interface”, because these measures isolate is paper reports on novel statistical methodology, which has been the task of analyzing experiments from the details of their design deployed by the commercial A/B testing platform Optimizely to and implementation. communicate experimental results to their customers. Our method- Crucially, the inferential validity of these p-values and con- ology addresses the issue that traditional p-values and condence dence intervals requires the separation between the design and intervals give unreliable inference. is is because users of A/B analysis of experiments to be strictly maintained. In particular, the testing soware are known to continuously monitor these measures sample size must be xed in advance. Compare this to A/B testing as the experiment is running. We provide always valid p-values practice, where users oen continuously monitor the p-values and and condence intervals that are provably robust to this eect. condence intervals reported in order to re-adjust the sample size Not only does this make it safe for a user to continuously monitor, of an experiment dynamically [14]. Figure 1 shows a typical A/B but it empowers her to detect true eects more eciently. is testing dashboard that enables such behavior.
    [Show full text]
  • Erich L. Lehmann, the Lehmann Symposia, and November 20Th 1917 3
    IMS Collections Vol. 0 (2009) 1{7 c Institute of Mathematical Statistics, 2009 1 1 arXiv: math.PR/0000021 2 2 3 3 4 Erich L. Lehmann, The Lehmann 4 5 5 th 6 Symposia, and November 20 1917 6 7 7 8 Javier Rojo?? 8 9 9 Rice University 10 10 11 11 12 12 13 The Lehmann Symposia originated as a result of a conversation I had in the 13 14 year 2001 with the, then, Director of the Centro de Investigaci´onen Matem´aticas 14 15 (CIMAT), Victor P´erez-Abreu.We both felt that there was an urgent need to 15 16 bring back into focus theoretical statistics and our proposed solution was a series 16 17 of Symposia that could serve as a forum for some of the exciting theoretical work 17 18 being done in statistics. The First Lehmann Symposium took place at CIMAT in 18 19 May of 2002. Most of the participants were Mexican colleagues. The program can 19 20 be seen at the site http://www.stat.rice.edu/lehmann/1st-Lehmann.html. The 20 21 second Lehmann Symposium { http://www.stat.rice.edu/lehmann/ { was held 21 22 in May of 2004 at the School of Engineering at Rice University. Initially, the venues 22 23 for the Symposia would alternate between CIMAT and Rice University. However, 23 24 for various reasons, some being financial, it was decided to hold the 3rd Lehmann 24 25 Symposium in the United States. 25 26 The original plans for the Third Lehmann Symposium were to hold the sym- 26 27 posium at the Mathematical Sciences Research Institute (MSRI) in Berkeley dur- 27 28 ing the month of November of 2007.
    [Show full text]
  • Νο 38-Statistical Periscope No.38
    Νέα του ΕΣΙ, Νο. 38, Μάρτιος 2010 GSI News, No.38, March 2010 ΣΤΑΤΙΣΤΙΚΟ STATISTICAL ΠΕΡΙΣΚΟΠΙΟ PERISCOPE Από τον Editor Από τον Πρόεδρο Το θέμα της Στατιστικής Υπηρεσίας είναι γνωστό Όλα είναι έτοιμα για το 23ο Πανελλήνιο Συνέδριο σε όλους. Μετά από εκτενή διαβούλευση ψηφίστηκε Στατιστικής με θέμα «Στατιστική και Διαδίκτυο» το στις 3 Μαρτίου 2010 από τη Βουλή των Ελλήνων οποίο θα διοργανωθεί στη Βέροια από 7 έως 11 η σύσταση της Ελληνικής Στατιστικής Αρχής η Απριλίου 2010 από το Τμήμα Μαθηματικών του οποία θα αποτελεί την εθνική στατιστική υπηρεσία Αριστοτελείου Πανεπιστημίου Θεσσαλονίκης. σύμφωνα με τους κανονισμούς του Ευρωπαϊκού Προσκεκλημένοι ομιλητές του συνεδρίου είναι ο Κοινοβουλίου και θα διοικείται από 7μελές Καθηγητής Γιώργος Μητακίδης του Διοικητικό Συμβούλιο. Τέσσερα από τα μέλη του Πανεπιστημίου Πατρών και ο Καθηγητής Victor ΔΣ, ένα από τα οποία ορίζεται ως Πρόεδρος και ένα Ivanov του Joint Institute for Nuclear Research, ως Αντιπρόεδρος, επιλέγονται από τη Διάσκεψη Ρωσία. Οι δηλώσεις συμμετοχής με παρουσίαση των Προέδρων της Βουλής, με εισήγηση του εργασίας έχουν ήδη ξεπεράσει τις 90 εκ των οποίων Υπουργού Οικονομικών και ύστερα από δημόσια 12 είναι poster. προκήρυξη. Η καταληκτική ημερομηνία για Υπενθυμίζεται ότι διοργανώνεται και φέτος ο υποβολή υποψηφιοτήτων είναι η Παρασκευή 26 διαγωνισμός για το Βραβείο Νέου Στατιστικού για Μαρτίου 2010. Οι σχετικές πληροφορίες το 2010. Το βραβείο, το οποίο φέτος διεκδικούν βρίσκονται στην ιστοσελίδα τέσσερεις υποψήφιοι, απευθύνεται σε www.opengov.gr/home/?p=579. μεταπτυχιακούς φοιτητές και νέους διδάκτορες και Επιπρόσθετα, με τον ίδιο νόμο, αποφασίστηκε η συνοδεύεται από χρηματικό έπαθλο. σύσταση Συμβουλίου του Ελληνικού Στατιστικού Συστήματος (ΣΥ.ΕΛ.Σ.Σ) το οποίο γνωμοδοτεί Η τελετή έναρξης όπου θα μιλήσει ο Κος στην ΕΛ.ΣΤΑΤ για θέματα που την αφορούν.
    [Show full text]
  • Germans Settling North America : Going Dutch – Gone American
    Gellinek Going Dutch – Gone American Christian Gellinek Going Dutch – Gone American Germans Settling North America Aschendorff Münster Printed with the kind support of Carl-Toepfer-Stiftung, Hamburg, Germany © 2003 Aschendorff Verlag GmbH & Co. KG, Münster Das Werk ist urheberrechtlich geschützt. Die dadurch begründeten Rechte, insbesondere die der Überset- zung, des Nachdrucks, der Entnahme von Abbildungen, der Funksendung, der Wiedergabe auf foto- mechanischem oder ähnlichem Wege und der Speicherung in Datenverarbeitungsanlagen bleiben, auch bei nur auszugsweiser Verwertung, vorbehalten. Die Vergütungsansprüche des § 54, Abs. 2, UrhG, werden durch die Verwertungsgesellschaft Wort wahrgenommen. Druck: Druckhaus Aschendorff, Münster, 2003 Gedruckt auf säurefreiem, alterungsbeständigem Papier ∞ ISBN 3-402-05182-6 This Book is dedicated to my teacher of Comparative Anthropology at Yale Law School from 1961 to 1963 F. S. C. Northrop (1893–1992) Sterling Professor of Philosophy and Law, author of the benchmark for comparative philosophy, Philosophical Anthropology and Practical Politics This Book has two mottoes which bifurcate as the topic =s divining rod The first motto is by GERTRUDE STEIN [1874–1946], a Pennsylvania-born woman of letters, raised in California, and expatriate resident of Europe after 1903: AIn the United States there is more space where nobody is than where anybody is. That is what makes America what it is.@1 The second motto has to do with the German immigration. It is borrowed from a book by THEODOR FONTANE [1819–1898], a Brandenburg-born writer, and a critic of Prussia. An old German woman, whose grandchildren have emigrated to Anmerica is speaking in her dialect of Low German: [ADröwen in Amirika.
    [Show full text]
  • ERICH LEO LEHMANN—A GLIMPSE INTO HIS LIFE and WORK 3 Asked Hsu If He Could Provide Another Thesis Topic for Erich
    The Annals of Statistics 2011, Vol. 39, No. 5, 2244–2265 DOI: 10.1214/11-AOS927 c Institute of Mathematical Statistics, 2011 ERICH LEO LEHMANN—A GLIMPSE INTO HIS LIFE AND WORK By Javier Rojo1 Rice University Through the use of a system-building approach, an approach that includes finding common ground for the various philosophical paradigms within statistics, Erich L. Lehmann is responsible for much of the synthesis of classical statistical knowledge that developed from the Neyman–Pearson–Wald school. A biographical sketch and a brief summary of some of his many contributions are presented here. His complete bibliography is also included and the references present many other sources of information on his life and his work. 1. Biographical sketch. Erich L. Lehmann was born in Strasbourg on November 20th, 1917. He passed away in Berkeley, California on the morning of September 12th, 2009. His family left Germany in 1933, as the Nazis came to power, to settle in Switzerland. He spent five years in Z¨urich and two years in Trinity College in Cambridge studying mathematics. Under the United States French immigration quota—Strasbourg was, by then, part of France as a consequence of the Versailles treaty—he arrived in New York at the end of 1940. Edmund Landau, the famous number theorist, was an acquaintance of the Lehmann family and had suggested Trinity College as the place Erich should go to study mathematics. Landau died in 1938 from a heart attack, but his wife wrote a letter of introduction for Erich to take to Landau’s G¨ottingen colleague Richard Courant who was now in New York developing what became the Courant Institute.
    [Show full text]
  • CURRICULUM VITAE Javier Rojo, Ph.D
    CURRICULUM VITAE Javier Rojo, Ph.D. Korvis Professor of Statistics Address: Dr. Javier Rojo Korvis Professor of Statistics Director, RUSIS @ Oregon State Adjunct Professor, MD Anderson Cancer Center Adjunct Professor, Civil and Environmental Engineering Rice University Weniger Hall 239 Oregon State University Corvallis, Oregon 97331 Telephone: (775) 303-7437 Fax: E-mail: [email protected], [email protected] EDUCATION 1973 B.S. Mathematics -- University of Texas at El Paso Minor: Physics 1976 M.S. Mathematics -- University of Texas at El Paso 1978 M.S. Statistics -- Stanford University 1984 Ph.D. Statistics -- University of California, Berkeley Dissertation: On Lehmann's General Concept of Unbiasedness and the Existence of L-unbiased Estimators Thesis advisor: Erich L. Lehmann ACADEMIC APPOINTMENTS 1/1/17 -- Present Korvis Professor of Statistics Department of Statistics, Oregon State University 04/2015 – Present Appointed Adjunct Professor, Civil and Environmental Engineering, Rice University 11/2014 – Present Appointed to the National Scientific Advisory Committee, Statistical and Applied Mathematical Sciences Institute, SAMSI, Research Triangle Park 1/1/2014 – 12/31/2016 Seneca C. and May B. Weeks Endowed Chair of Statistics Chair, Department of Mathematics and Statistics 1 University of Nevada at Reno 1/2014 – Present Appointed to the Scientific Advisory Committee, Mathematical Biosciences Institute, Ohio State University 10/2006 -- 2013 Appointed Keck Faculty Affiliate, The Gulf Coast Consortia 11/2005 -- Present Adjunct
    [Show full text]
  • Asymptotic and Finite-Sample Properties of Estimators Based on Stochastic Gradients
    Asymptotic and finite-sample properties of estimators based on stochastic gradients The Harvard community has made this article openly available. Please share how this access benefits you. Your story matters Citation Toulis, Panos, and Edoardo M. Airoldi. 2017. “Asymptotic and Finite- Sample Properties of Estimators Based on Stochastic Gradients.” The Annals of Statistics 45 (4) (August): 1694–1727. doi:10.1214/16- aos1506. Published Version doi:10.1214/16-AOS1506 Citable link http://nrs.harvard.edu/urn-3:HUL.InstRepos:37304570 Terms of Use This article was downloaded from Harvard University’s DASH repository, and is made available under the terms and conditions applicable to Open Access Policy Articles, as set forth at http:// nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of- use#OAP Submitted to the Annals of Statistics arXiv: arXiv:1408.2923 ASYMPTOTIC AND FINITE-SAMPLE PROPERTIES OF ESTIMATORS BASED ON STOCHASTIC GRADIENTS By Panos Toulis and Edoardo M. Airoldi∗ University of Chicago and Harvard University Stochastic gradient descent procedures have gained popularity for parameter estimation from large data sets. However, their statis- tical properties are not well understood, in theory. And in practice, avoiding numerical instability requires careful tuning of key param- eters. Here, we introduce implicit stochastic gradient descent proce- dures, which involve parameter updates that are implicitly defined. Intuitively, implicit updates shrink standard stochastic gradient de- scent updates. The amount of shrinkage depends on the observed Fisher information matrix, which does not need to be explicitly com- puted; thus, implicit procedures increase stability without increas- ing the computational burden. Our theoretical analysis provides the first full characterization of the asymptotic behavior of both stan- dard and implicit stochastic gradient descent-based estimators, in- cluding finite-sample error bounds.
    [Show full text]
  • Commemorating the Retirement of R. Clifford Blair JMASM Editors
    Journal of Modern Applied Statistical Methods Volume 3 | Issue 2 Article 28 11-1-2004 Special Section: Commemorating the Retirement of R. Clifford Blair JMASM Editors Follow this and additional works at: http://digitalcommons.wayne.edu/jmasm Recommended Citation Editors, JMASM (2004) "Special Section: Commemorating the Retirement of R. Clifford Blair," Journal of Modern Applied Statistical Methods: Vol. 3 : Iss. 2 , Article 28. Available at: http://digitalcommons.wayne.edu/jmasm/vol3/iss2/28 This Full Issue is brought to you for free and open access by the Open Access Journals at DigitalCommons@WayneState. It has been accepted for inclusion in Journal of Modern Applied Statistical Methods by an authorized editor of DigitalCommons@WayneState. Journal of Modern Applied Statistical Methods Copyright © 2004 JMASM, Inc. November, 2004, Vol. 3, No 2, 518-566 1538 – 9472/04/$95.00 A Conversation With R. Clifford Blair On The Occasion Of His Retirement Shlomo S. Sawilowsky Evaluation and Research Wayne State University _____________________________________________________________________________________ An interview was conducted on 23 November 2003 with R. Clifford Blair on the occasion on his retirement from the University of South Florida. This article is based on that interview. Biographical sketches and images of members of his academic genealogy are provided. Keywords: R. Clifford Blair, nonparametric, Wilcoxon rank sum, rank transform, multivariate permutation tests, step-down multiple comparison test, comparative statistical power _____________________________________________________________________________________ “In the last 30 years there have been The people who lived in our area were important changes in the canons of good farmers who came from southern Georgia. When statistical practice or data analysis. Until the depression came along, they moved into the recently, and thanks to the work of J.
    [Show full text]
  • Selected Works in Probability and Statistics
    Selected Works in Probability and Statistics For further volumes: http://www.springer.com/series/8556 Javier Rojo Editor Selected Works of E. L. Lehmann Editor Javier Rojo Rice University Houston, TX 77005, USA [email protected] ISBN 978-1-4614-1411-7 e-ISBN 978-1-4614-1412-4 DOI 10.1007/978-1-4614-1412-4 Springer New York Dordrecht Heidelberg London Library of Congress Control Number: 2011941600 © Springer Science+Business Media, LLC 2012 All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com) Preface to the Series Springer’s Selected Works in Probability and Statistics series offers scientists and scholars the opportunity of assembling and commenting upon major classical works in statistics, and honors the work of distinguished scholars in probability and statistics. Each volume contains the original papers, original commentary by experts on the subject’s papers, and relevant biographies and bibliographies.
    [Show full text]
  • Personal Reminiscences About Erich L. Lehmann
    Statistical Modelling 2013; 13(5&6): 387–395 Personal reminiscences about Erich L. Lehmann Javier Rojo Statistics Department, Rice University, Houston, TX 77005 Abstract: Several works have been written about Erich L. Lehmann’s fundamental and influential work in statistics and about his life. This work recounts the author’s personal reminiscences about being a student of Erich L. Lehmann and later a colleague and a friend. Key words: History of statistics, Berkeley statistics department, Optimality, Symposia Several works have been written about Erich L. Lehmann’s personal and professional life. van Zwet (2011) narrates some of the author’s interactions with Erich Lehmann and describes Erich’s contributions, mostly in nonparametric statistics. Rojo (2011) provides a biographical sketch, a complete bibliography and a summary of Erich’s influential works. A wealth of information about Erich’s professional relationships, philosophy and life experiences can be found in a wonderful book by Lehmann (2008), and Erich’s various professional and personal facets are also disclosed in DeGroot (1986), Lehmann (1993, 1996) and Reid (1982). Rojo (2012) presents a selection of Erich’s works with expert commentary from several authors. Therefore, this work focuses on the author’s experiences as a student, colleague and friend of Erich L. Lehmann. These reminiscences cover the years from 1978, the time when the author first met Erich, to 2009 when Erich passed away. I had a difficult time deciding on what to share, not because I do not have things to say. Rather, because there are too many wonderful memories to choose from. Erich’s qualities as a person, mentor and friend are too many to share, and space is so limited, that a complete set of memories will have to wait for a future time.
    [Show full text]