DOCSLIB.ORG

Peeking at A/B Tests Why It Ma‚Ers, and What to Do About It

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Peeking at A/B Tests Why It Ma‚Ers, and What to Do About It KDD 2017 Applied Data Science Paper KDD’17, August 13–17, 2017, Halifax, NS, Canada Peeking at A/B Tests Why it maers, and what to do about it Ramesh Johari∗ Pete Koomen† Stanford University Optimizely, Inc. [email protected] Leonid Pekelis‡ David Walsh§ Optimizely, Inc. Stanford University [email protected] [email protected] ABSTRACT obtain a very simple “user interface”, because these measures isolate is paper reports on novel statistical methodology, which has been the task of analyzing experiments from the details of their design deployed by the commercial A/B testing platform Optimizely to and implementation. communicate experimental results to their customers. Our method- Crucially, the inferential validity of these p-values and con- ology addresses the issue that traditional p-values and condence dence intervals requires the separation between the design and intervals give unreliable inference. is is because users of A/B analysis of experiments to be strictly maintained. In particular, the testing soware are known to continuously monitor these measures sample size must be xed in advance. Compare this to A/B testing as the experiment is running. We provide always valid p-values practice, where users oen continuously monitor the p-values and and condence intervals that are provably robust to this eect. condence intervals reported in order to re-adjust the sample size Not only does this make it safe for a user to continuously monitor, of an experiment dynamically [14]. Figure 1 shows a typical A/B but it empowers her to detect true eects more eciently. is testing dashboard that enables such behavior. paper provides simulations and numerical studies on Optimizely’s is “peeking” behavior results because the opportunity cost of data, demonstrating an improvement in detection performance over longer experiments is large, so there is value to detecting true eects traditional methods. as quickly as possible, or giving up if it appears that no eect will be detected soon so that the user may test something else. Further, KEYWORDS most users lack good prior understanding of both their tolerance for longer experiments as well as the eect size they seek, frustrating A/B testing, sequential hypothesis testing, p-values, condence aempts to optimize the sample size in advance. Peeking early intervals at results to trade o maximum detection with minimum samples dynamically seems like a substantial benet of the real-time data 1 INTRODUCTION that modern A/B testing environments can provide. Web applications typically optimize their product oerings using Unfortunately, stopping experiments in an adaptive manner randomized controlled trials (RCTs); in industry parlance this is through continuous monitoring of the dashboard will severely fa- known as A/B testing. e rapid rise of A/B testing has led to the vorably bias the selection of experiments deemed signicant. Indeed, emergence of a number of widely used platforms that handle the very high false positive probabilities can be obtained—well in excess implementation of these experiments [10, 20]. e typical A/B test of the nominal desired false positive probability (typically set at 5%). compares the values of a parameter across two variations (con- As an example, even with 10,000 samples (quite common in online trol and treatment) to see if one variation oers an opportunity to A/B testing), we nd that the false positive probability can easily be improve their service, while the A/B testing platform communi- inated by 5-10x. at means that, throughout the industry, users cates results to the user via standard frequentist parameter testing have been drawing inferences that are not supported by their data. measures, i.e., p-values and condence intervals. In doing so, they Our paper presents the approach taken to address this challenge within the large-scale commercial A/B testing platform Optimizely. ∗ RJ is a technical advisor to Optimizely, Inc.; this work was completed as part of his We develop novel methodology to compute p-values and condence work with Optimizely. †PK is co-founder and Chief Technology Ocer of Optimizely, Inc. intervals; our measures, which we call always valid, allow users to ‡LP is a technical advisor to Optimizely, Inc.; this work was completed when he was continuously monitor the experiment and stop at a data-dependent an employee at Optimizely. time of their choosing, while maintaining control over false positive §is work was completed while DW was employed by Optimizely, Inc. probability at a desired pre-set level. is protects statistically naive Permission to make digital or hard copies of part or all of this work for personal or users, and lets all users leverage real-time data to trade o the classroom use is granted without fee provided that copies are not made or distributed detection power and sample size dynamically. As described in the for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. Copyrights for third-party components of this work must be honored. paper, our methods build on classical results in the sequential testing For all other uses, contact the owner/author(s). literature in statistics. e methods we describe were implemented KDD’17, August 13–17, 2017, Halifax, NS, Canada. in the Optimizely platform in January 2015 as Optimizely Stats © 2017 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-4887-4/17/08. DOI: hp://dx.doi.org/10.1145/3097983.3097992 Engine, and have been in use across all products including mobile, 1517 KDD 2017 Applied Data Science Paper KDD’17, August 13–17, 2017, Halifax, NS, Canada Figure 1: A typical results page in Optimizely’s A/B testing dashboard. e dashboard encourages users to continuously monitor their experiments, by providing updated results on experiments in real-time. web, and server-side testing; hundreds of thousands of experiments One-variation experiment. In a one-variation experiment, we have been run by thousands of customers since its launch. test a single variation (or treatment) against a known baseline. In In Section 2, we outline the basic A/B testing problem, as well particular, we suppose independent observations from an exponen- as the typical approach used today. In Section 3, we discuss why ¹ º1 iid∼ tial family X = Xn n=1 Fθ , where the parameter θ takes values continuous monitoring “breaks” the existing paradigm and leads in Θ ⊂ Rp . In this seing, we consider the problem of testing a sim- to invalid inference, and we propose a denition of always valid ple null hypothesis H0 : θ = θ0 against the composite alternative p-values and condence intervals that admit valid inference despite H1 : θ , θ0. Here θ0 is the known baseline of comparison. continuous monitoring of tests by users. In Section 4, we give the roughout the paper, we index probability distributions by the approach taken in the Optimizely platform to compute these mea- parameters; e.g., Pθ denotes the probability distribution on the data sures; in particular, they are derived from a novel generalization of induced by parameter θ. the mixture sequential probability ratio test (mSPRT) [16]. In Section Two-variation experiment. In a two-variation experiment or 5, we empirically demonstrate that our approach both allows users A/B test, we test two variations (e.g., treatment and control, or to control false positive probability, and improves the user’s ability A and B) against each other. Here we observe two independent to trade o between detection of real eects and the length of the i.i.d. sequences X and Y, corresponding to the observations on visi- experiment (in an appropriate sense). tors receiving experiences A and B respectively. In studying A/B e core of our solution is formulated for the basic A/B testing tests, we restrict the data model to the two most common cases en- problem with two variations (treatment and control). We conclude countered in practice: Bernoulli data with success probabilities µA in Section 6 by addressing challenges that arise for multivariate and µB (used to model binary outcomes such as clicks, conversions, testing, where users have many variations and metrics of interest etc.); and normal data with means µA and µB and known variance that they compare simultaneously. Multivariate testing immediately σ 2 (used to model continuous-valued outcomes such as time on gives rise to a severe multiple comparisons problem, where users site). In this seing, we consider the problem of testing the null can overinterpret signicant results unless appropriate corrections B A hypothesis H0 : θ := µ − µ = 0 against H1 : θ , 0. are applied [21]. In our deployment, always valid p-values are Decision rules. e experimenter needs to decide how long to combined with multiple hypothesis testing correction procedures to run the test, and whether to reject the null hypothesis when the test provide a robust inference platform for experimenters, supporting is done. We formalize this process through the notion of a decision both continuous monitoring and multivariate testing. rule. Formally, a decision rule is a pair ¹T; δº, where T is a stopping time that denotes the sample size at which the test is ended, and δ 2 PRELIMINARIES is a binary-valued decision dependent only on the observations up 1 In this section, we describe the typical approach for analyzing A/B to time T , where δ = 1 indicates that H0 is rejected. A stopping tests based on the frequentist theory of hypothesis testing, which time is any time that is dependent only on the data observed up to we refer to as xed-horizon testing. that time; therefore, this denition captures the crucial feature of decision-making in A/B tests that the terminal sample size may be 2.1 Experiments and decision rules data-dependent.
Load more

Recommended publications
  • Strength in Numbers: the Rising of Academic Statistics Departments In
    Agresti · Meng Agresti Eds. Alan Agresti · Xiao-Li Meng Editors Strength in Numbers: The Rising of Academic Statistics DepartmentsStatistics in the U.S. Rising of Academic The in Numbers: Strength Statistics Departments in the U.S. Strength in Numbers: The Rising of Academic Statistics Departments in the U.S. Alan Agresti • Xiao-Li Meng Editors Strength in Numbers: The Rising of Academic Statistics Departments in the U.S. 123 Editors Alan Agresti Xiao-Li Meng Department of Statistics Department of Statistics University of Florida Harvard University Gainesville, FL Cambridge, MA USA USA ISBN 978-1-4614-3648-5 ISBN 978-1-4614-3649-2 (eBook) DOI 10.1007/978-1-4614-3649-2 Springer New York Heidelberg Dordrecht London Library of Congress Control Number: 2012942702 Ó Springer Science+Business Media New York 2013 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer.
    [Show full text]
  • Erich Leo Lehmann, 19172009
    J. R. Statist. Soc. A (2010) 173, Part 3, pp. 683–689 Obituaries Erich Leo Lehmann, 1917–2009 Erich L. Lehmann, Professor Emeritus at the University of California, Berkeley, passed away on September 12th, 2009, aged 91 years. Erich was one of the engines that drove much of the development of theoretical and mainstream statistics during the second half of the 20th century. At the same time he kept himself aware of developments in applied statistics and prob- ability. He knew both the subject matter and the individuals developing our subject. He was a member of the powerful team of individuals that Jerzy Neyman built up at Berkeley in the 1950s. These included David Blackwell, Joe Hodges, Lucien Le Cam, Michel Loève, Henry Scheffé and Elizabeth Scott. Erich co-authored articles with each, except for the probabilist Loève. Erich was born in Strasbourg, France, on November 20th, 1917. His family moved to Frank- furt where they lived until 1933. When the Nazis came into power the family fled to Switzerland where Erich went to high school. In 1938, following his father’s advice, he went to study math- ematics at Trinity College, Cambridge. Erich remarked that he was ‘always the best student in mathematics’, but he did not enjoy the accompanying astronomy and physics. Of the latter he said ‘I hated it’ (in ‘A conversation with Erich L. Lehmann’, with Morris DeGroot, in volume 1 of Statistical Science (1986)). In 1940, again influenced by his father, Erich went to New York and then following a suggestion by R. Courant he crossed the country to study at Berkeley.
    [Show full text]
  • Erich L. Lehmann, the Lehmann Symposia, and November 20Th 1917 3
    IMS Collections Vol. 0 (2009) 1{7 c Institute of Mathematical Statistics, 2009 1 1 arXiv: math.PR/0000021 2 2 3 3 4 Erich L. Lehmann, The Lehmann 4 5 5 th 6 Symposia, and November 20 1917 6 7 7 8 Javier Rojo?? 8 9 9 Rice University 10 10 11 11 12 12 13 The Lehmann Symposia originated as a result of a conversation I had in the 13 14 year 2001 with the, then, Director of the Centro de Investigaci´onen Matem´aticas 14 15 (CIMAT), Victor P´erez-Abreu.We both felt that there was an urgent need to 15 16 bring back into focus theoretical statistics and our proposed solution was a series 16 17 of Symposia that could serve as a forum for some of the exciting theoretical work 17 18 being done in statistics. The First Lehmann Symposium took place at CIMAT in 18 19 May of 2002. Most of the participants were Mexican colleagues. The program can 19 20 be seen at the site http://www.stat.rice.edu/lehmann/1st-Lehmann.html. The 20 21 second Lehmann Symposium { http://www.stat.rice.edu/lehmann/ { was held 21 22 in May of 2004 at the School of Engineering at Rice University. Initially, the venues 22 23 for the Symposia would alternate between CIMAT and Rice University. However, 23 24 for various reasons, some being financial, it was decided to hold the 3rd Lehmann 24 25 Symposium in the United States. 25 26 The original plans for the Third Lehmann Symposium were to hold the sym- 26 27 posium at the Mathematical Sciences Research Institute (MSRI) in Berkeley dur- 27 28 ing the month of November of 2007.
    [Show full text]
  • Νο 38-Statistical Periscope No.38
    Νέα του ΕΣΙ, Νο. 38, Μάρτιος 2010 GSI News, No.38, March 2010 ΣΤΑΤΙΣΤΙΚΟ STATISTICAL ΠΕΡΙΣΚΟΠΙΟ PERISCOPE Από τον Editor Από τον Πρόεδρο Το θέμα της Στατιστικής Υπηρεσίας είναι γνωστό Όλα είναι έτοιμα για το 23ο Πανελλήνιο Συνέδριο σε όλους. Μετά από εκτενή διαβούλευση ψηφίστηκε Στατιστικής με θέμα «Στατιστική και Διαδίκτυο» το στις 3 Μαρτίου 2010 από τη Βουλή των Ελλήνων οποίο θα διοργανωθεί στη Βέροια από 7 έως 11 η σύσταση της Ελληνικής Στατιστικής Αρχής η Απριλίου 2010 από το Τμήμα Μαθηματικών του οποία θα αποτελεί την εθνική στατιστική υπηρεσία Αριστοτελείου Πανεπιστημίου Θεσσαλονίκης. σύμφωνα με τους κανονισμούς του Ευρωπαϊκού Προσκεκλημένοι ομιλητές του συνεδρίου είναι ο Κοινοβουλίου και θα διοικείται από 7μελές Καθηγητής Γιώργος Μητακίδης του Διοικητικό Συμβούλιο. Τέσσερα από τα μέλη του Πανεπιστημίου Πατρών και ο Καθηγητής Victor ΔΣ, ένα από τα οποία ορίζεται ως Πρόεδρος και ένα Ivanov του Joint Institute for Nuclear Research, ως Αντιπρόεδρος, επιλέγονται από τη Διάσκεψη Ρωσία. Οι δηλώσεις συμμετοχής με παρουσίαση των Προέδρων της Βουλής, με εισήγηση του εργασίας έχουν ήδη ξεπεράσει τις 90 εκ των οποίων Υπουργού Οικονομικών και ύστερα από δημόσια 12 είναι poster. προκήρυξη. Η καταληκτική ημερομηνία για Υπενθυμίζεται ότι διοργανώνεται και φέτος ο υποβολή υποψηφιοτήτων είναι η Παρασκευή 26 διαγωνισμός για το Βραβείο Νέου Στατιστικού για Μαρτίου 2010. Οι σχετικές πληροφορίες το 2010. Το βραβείο, το οποίο φέτος διεκδικούν βρίσκονται στην ιστοσελίδα τέσσερεις υποψήφιοι, απευθύνεται σε www.opengov.gr/home/?p=579. μεταπτυχιακούς φοιτητές και νέους διδάκτορες και Επιπρόσθετα, με τον ίδιο νόμο, αποφασίστηκε η συνοδεύεται από χρηματικό έπαθλο. σύσταση Συμβουλίου του Ελληνικού Στατιστικού Συστήματος (ΣΥ.ΕΛ.Σ.Σ) το οποίο γνωμοδοτεί Η τελετή έναρξης όπου θα μιλήσει ο Κος στην ΕΛ.ΣΤΑΤ για θέματα που την αφορούν.
    [Show full text]
  • Germans Settling North America : Going Dutch – Gone American
    Gellinek Going Dutch – Gone American Christian Gellinek Going Dutch – Gone American Germans Settling North America Aschendorff Münster Printed with the kind support of Carl-Toepfer-Stiftung, Hamburg, Germany © 2003 Aschendorff Verlag GmbH & Co. KG, Münster Das Werk ist urheberrechtlich geschützt. Die dadurch begründeten Rechte, insbesondere die der Überset- zung, des Nachdrucks, der Entnahme von Abbildungen, der Funksendung, der Wiedergabe auf foto- mechanischem oder ähnlichem Wege und der Speicherung in Datenverarbeitungsanlagen bleiben, auch bei nur auszugsweiser Verwertung, vorbehalten. Die Vergütungsansprüche des § 54, Abs. 2, UrhG, werden durch die Verwertungsgesellschaft Wort wahrgenommen. Druck: Druckhaus Aschendorff, Münster, 2003 Gedruckt auf säurefreiem, alterungsbeständigem Papier ∞ ISBN 3-402-05182-6 This Book is dedicated to my teacher of Comparative Anthropology at Yale Law School from 1961 to 1963 F. S. C. Northrop (1893–1992) Sterling Professor of Philosophy and Law, author of the benchmark for comparative philosophy, Philosophical Anthropology and Practical Politics This Book has two mottoes which bifurcate as the topic =s divining rod The first motto is by GERTRUDE STEIN [1874–1946], a Pennsylvania-born woman of letters, raised in California, and expatriate resident of Europe after 1903: AIn the United States there is more space where nobody is than where anybody is. That is what makes America what it is.@1 The second motto has to do with the German immigration. It is borrowed from a book by THEODOR FONTANE [1819–1898], a Brandenburg-born writer, and a critic of Prussia. An old German woman, whose grandchildren have emigrated to Anmerica is speaking in her dialect of Low German: [ADröwen in Amirika.
    [Show full text]
  • ERICH LEO LEHMANN—A GLIMPSE INTO HIS LIFE and WORK 3 Asked Hsu If He Could Provide Another Thesis Topic for Erich
    The Annals of Statistics 2011, Vol. 39, No. 5, 2244–2265 DOI: 10.1214/11-AOS927 c Institute of Mathematical Statistics, 2011 ERICH LEO LEHMANN—A GLIMPSE INTO HIS LIFE AND WORK By Javier Rojo1 Rice University Through the use of a system-building approach, an approach that includes finding common ground for the various philosophical paradigms within statistics, Erich L. Lehmann is responsible for much of the synthesis of classical statistical knowledge that developed from the Neyman–Pearson–Wald school. A biographical sketch and a brief summary of some of his many contributions are presented here. His complete bibliography is also included and the references present many other sources of information on his life and his work. 1. Biographical sketch. Erich L. Lehmann was born in Strasbourg on November 20th, 1917. He passed away in Berkeley, California on the morning of September 12th, 2009. His family left Germany in 1933, as the Nazis came to power, to settle in Switzerland. He spent five years in Z¨urich and two years in Trinity College in Cambridge studying mathematics. Under the United States French immigration quota—Strasbourg was, by then, part of France as a consequence of the Versailles treaty—he arrived in New York at the end of 1940. Edmund Landau, the famous number theorist, was an acquaintance of the Lehmann family and had suggested Trinity College as the place Erich should go to study mathematics. Landau died in 1938 from a heart attack, but his wife wrote a letter of introduction for Erich to take to Landau’s G¨ottingen colleague Richard Courant who was now in New York developing what became the Courant Institute.
    [Show full text]
  • CURRICULUM VITAE Javier Rojo, Ph.D
    CURRICULUM VITAE Javier Rojo, Ph.D. Korvis Professor of Statistics Address: Dr. Javier Rojo Korvis Professor of Statistics Director, RUSIS @ Oregon State Adjunct Professor, MD Anderson Cancer Center Adjunct Professor, Civil and Environmental Engineering Rice University Weniger Hall 239 Oregon State University Corvallis, Oregon 97331 Telephone: (775) 303-7437 Fax: E-mail: [email protected], [email protected] EDUCATION 1973 B.S. Mathematics -- University of Texas at El Paso Minor: Physics 1976 M.S. Mathematics -- University of Texas at El Paso 1978 M.S. Statistics -- Stanford University 1984 Ph.D. Statistics -- University of California, Berkeley Dissertation: On Lehmann's General Concept of Unbiasedness and the Existence of L-unbiased Estimators Thesis advisor: Erich L. Lehmann ACADEMIC APPOINTMENTS 1/1/17 -- Present Korvis Professor of Statistics Department of Statistics, Oregon State University 04/2015 – Present Appointed Adjunct Professor, Civil and Environmental Engineering, Rice University 11/2014 – Present Appointed to the National Scientific Advisory Committee, Statistical and Applied Mathematical Sciences Institute, SAMSI, Research Triangle Park 1/1/2014 – 12/31/2016 Seneca C. and May B. Weeks Endowed Chair of Statistics Chair, Department of Mathematics and Statistics 1 University of Nevada at Reno 1/2014 – Present Appointed to the Scientific Advisory Committee, Mathematical Biosciences Institute, Ohio State University 10/2006 -- 2013 Appointed Keck Faculty Affiliate, The Gulf Coast Consortia 11/2005 -- Present Adjunct
    [Show full text]
  • Asymptotic and Finite-Sample Properties of Estimators Based on Stochastic Gradients
    Asymptotic and finite-sample properties of estimators based on stochastic gradients The Harvard community has made this article openly available. Please share how this access benefits you. Your story matters Citation Toulis, Panos, and Edoardo M. Airoldi. 2017. “Asymptotic and Finite- Sample Properties of Estimators Based on Stochastic Gradients.” The Annals of Statistics 45 (4) (August): 1694–1727. doi:10.1214/16- aos1506. Published Version doi:10.1214/16-AOS1506 Citable link http://nrs.harvard.edu/urn-3:HUL.InstRepos:37304570 Terms of Use This article was downloaded from Harvard University’s DASH repository, and is made available under the terms and conditions applicable to Open Access Policy Articles, as set forth at http:// nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of- use#OAP Submitted to the Annals of Statistics arXiv: arXiv:1408.2923 ASYMPTOTIC AND FINITE-SAMPLE PROPERTIES OF ESTIMATORS BASED ON STOCHASTIC GRADIENTS By Panos Toulis and Edoardo M. Airoldi∗ University of Chicago and Harvard University Stochastic gradient descent procedures have gained popularity for parameter estimation from large data sets. However, their statis- tical properties are not well understood, in theory. And in practice, avoiding numerical instability requires careful tuning of key param- eters. Here, we introduce implicit stochastic gradient descent proce- dures, which involve parameter updates that are implicitly defined. Intuitively, implicit updates shrink standard stochastic gradient de- scent updates. The amount of shrinkage depends on the observed Fisher information matrix, which does not need to be explicitly com- puted; thus, implicit procedures increase stability without increas- ing the computational burden. Our theoretical analysis provides the first full characterization of the asymptotic behavior of both stan- dard and implicit stochastic gradient descent-based estimators, in- cluding finite-sample error bounds.
    [Show full text]
  • Commemorating the Retirement of R. Clifford Blair JMASM Editors
    Journal of Modern Applied Statistical Methods Volume 3 | Issue 2 Article 28 11-1-2004 Special Section: Commemorating the Retirement of R. Clifford Blair JMASM Editors Follow this and additional works at: http://digitalcommons.wayne.edu/jmasm Recommended Citation Editors, JMASM (2004) "Special Section: Commemorating the Retirement of R. Clifford Blair," Journal of Modern Applied Statistical Methods: Vol. 3 : Iss. 2 , Article 28. Available at: http://digitalcommons.wayne.edu/jmasm/vol3/iss2/28 This Full Issue is brought to you for free and open access by the Open Access Journals at DigitalCommons@WayneState. It has been accepted for inclusion in Journal of Modern Applied Statistical Methods by an authorized editor of DigitalCommons@WayneState. Journal of Modern Applied Statistical Methods Copyright © 2004 JMASM, Inc. November, 2004, Vol. 3, No 2, 518-566 1538 – 9472/04/$95.00 A Conversation With R. Clifford Blair On The Occasion Of His Retirement Shlomo S. Sawilowsky Evaluation and Research Wayne State University _____________________________________________________________________________________ An interview was conducted on 23 November 2003 with R. Clifford Blair on the occasion on his retirement from the University of South Florida. This article is based on that interview. Biographical sketches and images of members of his academic genealogy are provided. Keywords: R. Clifford Blair, nonparametric, Wilcoxon rank sum, rank transform, multivariate permutation tests, step-down multiple comparison test, comparative statistical power _____________________________________________________________________________________ “In the last 30 years there have been The people who lived in our area were important changes in the canons of good farmers who came from southern Georgia. When statistical practice or data analysis. Until the depression came along, they moved into the recently, and thanks to the work of J.
    [Show full text]
  • Selected Works in Probability and Statistics
    Selected Works in Probability and Statistics For further volumes: http://www.springer.com/series/8556 Javier Rojo Editor Selected Works of E. L. Lehmann Editor Javier Rojo Rice University Houston, TX 77005, USA [email protected] ISBN 978-1-4614-1411-7 e-ISBN 978-1-4614-1412-4 DOI 10.1007/978-1-4614-1412-4 Springer New York Dordrecht Heidelberg London Library of Congress Control Number: 2011941600 © Springer Science+Business Media, LLC 2012 All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com) Preface to the Series Springer’s Selected Works in Probability and Statistics series offers scientists and scholars the opportunity of assembling and commenting upon major classical works in statistics, and honors the work of distinguished scholars in probability and statistics. Each volume contains the original papers, original commentary by experts on the subject’s papers, and relevant biographies and bibliographies.
    [Show full text]
  • Personal Reminiscences About Erich L. Lehmann
    Statistical Modelling 2013; 13(5&6): 387–395 Personal reminiscences about Erich L. Lehmann Javier Rojo Statistics Department, Rice University, Houston, TX 77005 Abstract: Several works have been written about Erich L. Lehmann’s fundamental and influential work in statistics and about his life. This work recounts the author’s personal reminiscences about being a student of Erich L. Lehmann and later a colleague and a friend. Key words: History of statistics, Berkeley statistics department, Optimality, Symposia Several works have been written about Erich L. Lehmann’s personal and professional life. van Zwet (2011) narrates some of the author’s interactions with Erich Lehmann and describes Erich’s contributions, mostly in nonparametric statistics. Rojo (2011) provides a biographical sketch, a complete bibliography and a summary of Erich’s influential works. A wealth of information about Erich’s professional relationships, philosophy and life experiences can be found in a wonderful book by Lehmann (2008), and Erich’s various professional and personal facets are also disclosed in DeGroot (1986), Lehmann (1993, 1996) and Reid (1982). Rojo (2012) presents a selection of Erich’s works with expert commentary from several authors. Therefore, this work focuses on the author’s experiences as a student, colleague and friend of Erich L. Lehmann. These reminiscences cover the years from 1978, the time when the author first met Erich, to 2009 when Erich passed away. I had a difficult time deciding on what to share, not because I do not have things to say. Rather, because there are too many wonderful memories to choose from. Erich’s qualities as a person, mentor and friend are too many to share, and space is so limited, that a complete set of memories will have to wait for a future time.
    [Show full text]
  • International Encyclopedia of Statistical Science
    International Encyclopedia of Statistical Science Miodrag Lovric (Ed.) International Encyclopedia of Statistical Science With Figures and Tables 123 Editor: Miodrag Lovric Department of Statistics and Informatics Faculty of Economics University of Kragujevac City of Kragujevac Serbia Library of Congress Control Number: ISBN ---- This publication is available also as: Electronic publication under ISBN ---- Print and electronic bundle under ISBN ---- DOI ./---- Springer Heidelberg Dordrecht London New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September , , in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. © Springer-Verlag Berlin Heidelberg The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Springer is part of Springer Science+Business Media www.springer.com Printed on acid-free paper SPIN: Spi– Foreword by Bradley Efron The Future of Statistics ▸ Strange, as one gets older you’re expected to know and columns, one for each man: enormous by Twenti- more about the future. eth Century standards but nothing remarkable these days. We wonder which of the genes, if any, are more active in The history of statistics as a recognized discipline divides the cancer patients.
    [Show full text]