University of Pennsylvania ScholarlyCommons Publicly Accessible Penn Dissertations 2013 Bayesian Aspects of Classification Procedures Igar Fuki University of Pennsylvania, [email protected] Follow this and additional works at: https://repository.upenn.edu/edissertations Part of the Statistics and Probability Commons Recommended Citation Fuki, Igar, "Bayesian Aspects of Classification Procedures" (2013). Publicly Accessible Penn Dissertations. 863. https://repository.upenn.edu/edissertations/863 This paper is posted at ScholarlyCommons. https://repository.upenn.edu/edissertations/863 For more information, please contact [email protected]. Bayesian Aspects of Classification Procedures Abstract We consider several statistical approaches to binary classification and multiple hypothesis testing problems. Situations in which a binary choice must be made are common in science. Usually, there is uncertainty involved in making the choice and a great number of statistical techniques have been put forth to help researchers deal with this uncertainty in separating signal from noise in reasonable ways. For example, in genetic studies, one may want to identify genes that affect a certain biological process from among a larger set of genes. In such examples, costs are attached to making incorrect choices and many choices must be made at the same time. Reasonable ways of modeling the cost structure and choosing the appropriate criteria for evaluating the performance of statistical techniques are needed. The following three chapters have proposals of some Bayesian methods for these issues. In the first chapter, we focus on an empirical Bayes approach to a popular binary classification problem formulation. In this framework, observations are treated as independent draws from a hierarchical model with a mixture prior distribution. The mixture prior combines prior distributions for the ``noise'' and for the ``signal'' observations. In the literature, parametric assumptions are usually made about the prior distribution from which the ``signal'' observations come. We suggest a Bayes classification rule which minimizes the expectation of a flexible and easily interpretable mixture loss function which brings together constant penalties for false positive misclassifications and $L_2$ penalties for false negative misclassifications. Due in part to the form of the loss function, empirical Bayes techniques can then be used to construct the Bayes classification rule without specifying the ``signal'' part of the mixture prior distribution. The proposed classification technique builds directly on the nonparametric mixture prior approach proposed by Raykar and Zhao (2010, 2011). Many different criteria can be used to judge the success of a classification procedure. A very useful criterion called the False Discovery Rate (FDR) was introduced by Benjamini and Hochberg in a 1995 paper. For many applications, the FDR, which is defined as the expected proportion of false positive results among the observations declared to be ``signal'', is a reasonable criterion to target. Bayesian versions of the false discovery rate, the so-called positive false discovery rate (pFDR) and local false discovery rate, were proposed by Storey (2002, 2003) and Efron and coauthors (2001), respectively. There is an interesting connection between the local false discovery rate and the nonparametric mixture prior approach for binary classification problems. The second part of the dissertation is focused on this link and provides a comparison of various approaches for estimating Bayesian false discovery rates. The third chapter is an account of a connection between the celebrated Neyman-Pearson lemma and the area (AUC) under the receiver operating characteristic (ROC) curve when the observations that need to be classified come from a pair of normal distributions. Using this connection, it is possible to derive a classification rule which maximizes the AUC for binormal data. Degree Type Dissertation Degree Name Doctor of Philosophy (PhD) Graduate Group Statistics First Advisor Linda H. Zhao Keywords Classification procedures, empirical Bayes, False discovery rate, nonparametric mixture prior Subject Categories Statistics and Probability This dissertation is available at ScholarlyCommons: https://repository.upenn.edu/edissertations/863 BAYESIAN ASPECTS OF CLASSIFICATION PROCEDURES Igar Fuki A DISSERTATION in Statistics For the Graduate Group in Managerial Science and Applied Economics Presented to the Faculties of the University of Pennsylvania in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy 2013 Supervisor of Dissertation Linda Zhao, Professor, Statistics Graduate Group Chairperson Eric Bradlow, K.P. Chao Professor, Marketing, Statistics and Education Dissertation Committee Linda Zhao, Professor of Statistics Lawrence D. Brown, Miers Busch Professor, Statistics Fernando Ferreira, Associate Professor of Real Estate BAYESIAN ASPECTS OF CLASSIFICATION PROCEDURES COPYRIGHT 2013 Igar Fuki This work is licensed under the Creative Commons Attribution- NonCommercial-ShareAlike 3.0 License To view a copy of this license, visit http://creativecommons.org/licenses/by-ny-sa/2.0/ To my family. iii Acknowledgement This dissertation and all of my projects in graduate school would not have been possible without Linda Zhao. I would like to thank her for her incredible help and support. Professor Zhao not only set out my research path, but was a source of strength during the most difficult times. Thank you for everything. I would like to thank my committee members, Lawrence Brown and Fernando Ferreira, for their help and invaluable advice with my dissertation. I would also like to express my deepest gratitude to Marja Hoek-Smit, whose door was always open for me. Thank you to Todd Sinai, Warren Ewens, Murray Gerstenhaber, Carol Reich, Keith Weigelt, Daniel Yekutieli, and W. Bruce Allen for their guidance and advice. My thanks also to Vikas Raykar and Xu Han. This thesis is based on several joint working papers, and I would like to thank my senior co-authors for their contributions to the text, for their inspiration, and for teaching me about research and scientific writing. Thank you also to the faculty, staff, and students of the Statistics and Real Estate departments at Wharton, and to many others at the University of Pennsylvania for all of their help. Finally, I want to thank my friends and family for their patience and kind support. iv ABSTRACT BAYESIAN ASPECTS OF CLASSIFICATION PROCEDURES Igar Fuki Linda Zhao We consider several statistical approaches to binary classification and multiple hypothesis testing problems. Situations in which a binary choice must be made are common in science. Usually, there is uncertainty involved in making the choice and a great number of statistical techniques have been put forth to help researchers deal with this uncertainty in separating signal from noise in reasonable ways. For example, in genetic studies, one may want to identify genes that affect a certain biological process from among a larger set of genes. In such examples, costs are attached to making incorrect choices and many choices must be made at the same time. Reasonable ways of modeling the cost structure and choosing the appropriate criteria for evaluating the performance of statistical techniques are needed. The following three chapters have proposals of some Bayesian methods for these issues. In the first chapter, we focus on an empirical Bayes approach to a popular binary classification problem formulation. In this framework, observations are treated as independent draws from a hierarchical model with a mixture prior distribution. v The mixture prior combines prior distributions for the \noise" and for the \signal" observations. In the literature, parametric assumptions are usually made about the prior distribution from which the \signal" observations come. We suggest a Bayes classification rule which minimizes the expectation of a flexible and easily interpretable mixture loss function which brings together constant penalties for false positive misclassifications and L2 penalties for false negative misclassifications. Due in part to the form of the loss function, empirical Bayes techniques can then be used to construct the Bayes classification rule without specifying the \signal" part of the mixture prior distribution. The proposed classification technique builds directly on the nonparametric mixture prior approach proposed by Raykar and Zhao (2010, 2011). Many different criteria can be used to judge the success of a classification proce- dure. A very useful criterion called the False Discovery Rate (FDR) was introduced by Benjamini and Hochberg in a 1995 paper. For many applications, the FDR, which is defined as the expected proportion of false positive results among the observa- tions declared to be \signal", is a reasonable criterion to target. Bayesian versions of the false discovery rate, the so-called positive false discovery rate (pFDR) and local false discovery rate, were proposed by Storey (2002, 2003) and Efron and coauthors (2001), respectively. There is an interesting connection between the local false dis- covery rate and the nonparametric mixture prior approach for binary classification problems. The second part of the dissertation is focused on this link and provides a comparison of various approaches for estimating Bayesian false discovery rates. The third chapter is an account of a connection between the celebrated Neyman- vi Pearson lemma and