8 Henderson and Newton

according to mid-season data and rank(θi) is his unknown true rank. R-values may be computed in all sorts of hierarchical modeling efforts, including semi- parametric models and cases where Markov chain Monte Carlo (MCMC) is used to approx- imate the marginal posterior distribution of each θi given available data. Figure 6 compares the r-value ranking with other rankings in an example from gene-expression analysis, where evidence suggested that the expressionM.A. Newton of a large fractionand N.C. of the humanHenderson genome was associ- ated with the status of a certainUniversity viral infection of (Pyeon,Wisconsin,et al., 2007). Madison, A multi-level USA model Putting lots of things in order: R-VALUESinvolving both null and non-null genes as well as t−distributed non-null effects θi exhibited good fit to the data, but did not admit a closed form for Vα(Di). R-values, computed using MCMC output, again reveal systematic ranking differences from other approaches. Multi-level models drive statistical inference and software in a variety of genomic do- • setting: large scale, non-sparse inference Construction of ranking variables, NBA example mains: forR-value example, limma ranking(Smyth, 2004), maximizesEBarrays (Kendziorski agreementet al. 2003), EBSeq (Leng et al. 2013), among others. Since these models happen to specify distributional forms for • examples: genomics/sports/evaluations/... parametersbetween of interest, the the associated true code couldtop be augmentedα fractionto compute and posterior tail 8 Henderson and Newton • task: rank order the units probabilities Vα(Di) and thus r-values for ranking. The limma system utilizes a conjugate 1. local posteriors p(✓i Di) 2. estimated marginal p(✓i) normal, inverse-gamma model, and so Vα(Di) involves the tail probability of a non-central according to mid-seasonthe reported data and rank(θ i)top is his unknownα fraction, true rank. for allα. • challenge: differential uncertainty | t distribution. The EBSeq system entails a conjugate beta, negative-binomial model, and 35 35 R-values may be computed in all sorts of hierarchical modeling efforts, including semi- so Vα(Di)fordifferential expression involves tail probabilities in a certain ratio distribution binomial likelihood parametric models and cases where Markov chain Monte Carlo (MCMC) is used to approx- 25 beta prior 25 beta (parametric case) (Coelho and Mexia, 2007). One expects the benefits of r-value computation to show espe- imate the marginal posterior distribution of each θi given available data. Figure 6 compares beta posteriors cially in cases involving many non-null units and relatively high variation among2 units in 15 15 Measurement model: D =(X , ) the r-valuetheir ranking variance with parameters other rankings (e.g.,sequencereaddepth). in an example from gene-expressioni analysis,i wherei

• 10 10 lots of units:Improved ranking and selection 17 i evidence suggested that the expression of a large fraction of the human genome was2 associ- density density E(Xi ✓i, )=✓i { } 5 5 i • ated with the status of a certain viral infection (Pyeon, et al., 2007). A multi-level| model data: Di 2 2 { } involving3. both Connections null and non-null2 genes as well as t−distributed2 2 non-nullvar( effXectsi ✓iθ,i exhibitedi )=i • ✓ V↵(x, )=P (✓i ✓↵ Xi = x, i = ) | parameters (of interest): i good↵ fit to the data, but did not admit a closed| form for Vα(Di). R-values, computed using { } 0 0 MCMC3.1. output, Connection again reveal to Bayes systematic rule ranking differences from other approaches. p(D ✓ ) 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 ✓ 1.0 2 2 • model: i i ↵ The proposedAssume r-valuesV↵ are(x, not Bayes) is rules right-continuous in the usual sense, how andever non-decreasing there is a connection in x for every ↵ and 0.30 | Ability Free Throw Ability Multi-level models drive statistical inference and software in a variety of genomic do- to Bayesian inference if one allows both a continuum of loss functions and a distributional 0.25 2 mains: for example, limma (Smyth, 2004), EBarrays2 (Kendziorski et al. 2003), EBSeq (Leng Assume V↵(x, ) is right-continuous and non-decreasingconstraint in onx thefor reported every unit-specific↵ and (relative) ranks. To see this connection, we introduce 0.20 Improved ranking and selection 15 et al. 2013), among others. Since these models happen to specify distributional forms for Hao, L., Q. He, Z. Wang, M. Craven, M. A. Newton, and P. Ahlquist (2013). Limited 3. posterior exceedance probability 4. r-value a collection of loss functions agreement0.15 of independent RNAi screens for virus-required host genes owes moreExamples to false- parameters of interest, the associated code could be augmented to compute posterior tail negative than false-positive factors. PLoS computational biology 9 (9), e1003235. Theorem: P r(Di) ↵,✓i ✓↵ P T (Di) ↵,✓i ✓↵ Jost, J. and X. Li-Jost (1998). Calculus of variations, Volume 64. Cambridge University 1 probabilities Vα(Di) and thus r-values for ranking. The limma system utilizes a conjugate Press.Log Odds {Lα(a, θi)=1 − 1(a ≤ α, θi}≥θα) {  } 0.10 P( θi ≥ θα | Di ) normal, inverse-gamma model, and so V (D ) involves the tail probability of a non-central Kass, R. E. and A. E. Raftery (1995). Bayes factors. Journal of the American Statistical DRay.Allen = 105 116 r(Di)=inf ↵ : P (✓i ✓↵ Di) ↵ for any other rankingα i variable T (D ), P [T (D ) ↵] ↵ Association 90(430), pp. 773–795. 0.35 0.8 two examples i i ● ● ● ● { | } ● ●●● EBSeq ●   ● t distribution. The system entails a conjugate beta, negative-binomial model, and ● ● ● Kendziorski, C., M. Newton, H. Lan, and M. Gould (2003). On parametric empirical 0.30 ● ● ● where action a is a relative rank value in (0, 1), α ∈ (0, 1) indexes the collection, and again ● ● empirical quantile ● ● ● ●●● ●● Bayes methods for comparing multiple groups using replicated gene expression profiles. ●● ● ● ●● −1 Statistics in medicine 22(24), 3899–3914. 0.25 ● ^ ● ● ● ●● so V (D )fordifferential expression involves tail probabilities in a certain ratio distribution ●●● ●● α i ● ● ● ● ●●●●● ●●● θ = F (1 − α) is a quantile in the population of interest. Specifically, no α−loss occurs ● ● ● ● λα ●●● ● α ● ●● ●●● ● 0.05 ● ● ●●●● Laird, N.Genome-wide M. and T. A. Louis (1989). Empirical Bayes association ranking methods. Journal of Edu- ●●●● ● ● ● ●● ● ● ● ● ●● ●● 0.20 ●● r−value ●●●●●●●● ● r-values reduce the differential uncertainty artifact cational and Behavioral Statistics 14 (1), 29–46. ● ● ● ●● (Coelho and Mexia, 2007). One expects the benefits of r-value computation to show espe- 0.01 0.02 0.05 0.10 ● ● ● ●●● ● ● ●●● if the inferred relative rank a and the actual relative rank 1 − F (θ ) both are less than α. ●● ● ● ● P P (✓ ✓ D ) = ↵ i ●● ● ● ●● i ↵ i ↵ ● ● ●● ● ● ● ● ● ●●●● ● ● ● ● ● ●●●●● Lehmann, E. (1986). Testing statistical hypotheses (2nd ed.). Wiley series in probability ● ● ● ● ●● ●● studies ●● ●●● ●● ● ●● ●● ● ● ●● ● ● ●●●● ● ● ● ● ●● 18 Henderson and Newton ●●●● ● ●● ● ●●● cially in cases involving many non-null units and relatively high variation among units in and mathematical statistics: Probability and mathematical statistics. Wiley. ●● ● ●● ● ● ● ● The marginal (pre-posterior) Bayes risk of rule δ(D ) is ●● ● ● { | } Standard Error 0.15 ● ●● ●● ● ●●●●● i ● ● ● ●●●●●●●● ● ● ● ●● ●● ●● ●● ●●●● ●● ● ● ●●● ● ● ●● ●●● ● ● ●● ● ● ● ● ● ●●● 0.4 ●● ● ●●●● ● ● ● ●● ●● ● ● ●● ●● ● ● ● ●● ● ●● ●● ● ●● Leng, N., J. A. Dawson, J. A. Thomson, V. Ruotti, A. I. Rissman, B. M. Smits, J. D. ●● ● ● ● ●● ●● ● ● ●● ●● ●● ● ● ● ● ● ● ●● ● ●●● ●● ●● ● ● ● ●● ● ●● ●● ●●● ● ● ● ●● ● p (i p-value p0.1 ) p ( X x ) ●●●● ●● ●●●●●●● ● ●● ● ● ●● ●● ●● ●● ● ●● ●● their variance parameters (e.g.,sequencereaddepth).i p i E(✓i Xi,i) e0.1 i i 0.1 ●●●● ●●● ● ● ●● ● ● ● ● ● ● ● ● ● p(i r(Di) 0.1) Haag, M. N. Gould, R. M. Stewart, and C. Kendziorski (2013). EBSeq: an empirical ●●●● ●● ● ● ●●●● ● ●● ●●●●● ●●● ● ● ●● ● ● ● ● ●● ●●●●● ●● ●●●●● ● ●● ● ● ● ● ●●● ● ● ●●● ● ● ●● |  ●●● ●● ●● ●● ● ● ● ●● ● ●● ● ●●● ● ●●● ●● ●● { | | } | ●●● ● ● ● ● ● ● ● ● ●●● Log Odds ●● ● ●● ● ● ●● ● ●● ●● ●● ●● ● ●●● |  Fig. 1.bayesType-2 hierarchical diabetes example: model for From inference the full in complement rna-seq experiments. of 127,903SNPsusedinthesecondBioinformatics 29 (8), ●● ●●● ● ● ● ● ●●● ●● ● ●●●● ●● ● ● ●● ● ●● ● ●● ●● ●●●●● ●● ● ●● ●●● ● ● ●● ● ●●●● ● ● ●●● ● ●● ●● ● ● ●●● ●● ● ● ● ● ●● ● ● ● ● ●●●● ● ● ●● ●● ● ●●● ●● ● ●● ● ● ● ●● ● ● ●● ● ● ● ●● ● ●●●● ● ● ● ● ●● ● ● ● ● ●●● stage1035–1043. meta-analysis from Morris et al. 2012, we filtered to a reduced set of 25,558 SNPs that are ● ● ●●● ● ● ●● ●●●●●●● ● ●●●● ● ●● ● ● ● risk =1− P {δ(D ) ≤ α , θ ≥ θ } , (9) ● ● ● 0.10 ●●● ● ● ● ● ● ● ● ●● ● ● ●●●● ●●●●●●● ●●●●●●●●●●● ● ●●● ● ● ●● ●●● ● ●● α i α ●● ● ●●● ●●● ● ● ●● ● ● ●● ● ●● ● ● ●● ● ●● ● ●●●● ● ●●● ● ● ●● ● ●●●● ●● ●●●● ●●●●●●● ●●●●●● ●●● ●● ● ● ● ● ●● ●●●●●● probably associated with T2D, and plot 3371 of those having highest observed association (log odds ●●● ●●●● ● ● ● ● ●● ●●●●●●●●●●●●● ●●● ● ●●● ●● ● ● ●● ● ●● ●●●●● ● ●●●● ● ● ●● ●●●●●●● ●● ●●● ●●●●● ●●● ● ●● ●● ●●● ● ●●●● ● ●●●● ●●●●●●● ●●●●● ●●●● ●●●●● ●●●●●●● ●●● ●●● ●● ● ● ●●● ●●● ●●● ●●●● ● ● ● ●●●● ●●● ●●● ●●●● ● ●● ●●●●● ● ●●●●● ● ● ●● exceedingLin, R., 0.05). T. A. These Louis, estimates S. M. Paddock, are based and onG. genotypeRidgeway data (2006).from Loss 22,669 function T2D cases based and ranking 58,119 ●●●● ● ●● ●●●● ●●● ●●●●●●● ●●●●● ●●●●●●●●●●●●●●● ●● ●●●●● ●● ● ● ●● ● ● ●● ● ●●●●● ●●●● ● ●●●●● ● ●●●●●●●● ●● ● ●●●●● ●●● ● ●● ● ● ●●● ●●● ● ● ●●● ● ●●●●●● ●●● ●● ● ●●● ● ●● ●●●●●●●●● ●● ● ● ● ●● ●●● ●● ● ●●● ●● ●●●●●●● ●● ●●●● ●●●●●●● ●●●●●●●● ●● ●● ●● ● ●●● ● ●●● ● ● ●● ● controlin subjects. two-stage, hierarchical models. Bayesian Analysis 1 (4), 915–946. ●● ●●●● ●●●●●●●●●●●●●● ●●● ● ●●●●●●●●●●●●●●●●●● ● ● ●●● ●● ● ●●● ●●● ●● ●● ● ●● ●● ●● ●● ●●●● ●● ●●●●●●●●●● ● ●●●●●●●●●●●●● ● ●● ●●●● ● ●●●●● ●● ●● ● ● ●● i ● ●●●● ● ● ● ●●● ●●●● ●●●●●● ● ●●●●●●●●●●●●●●●●●●●● ●● ●●●●●●●● ● ● ●● ● ● ● ● ●● Intuitively, the r-value for unit is the ●●● ●●● ● ●●●● ●●●●●● ●● ●●●● ●●●● ●●●●●●● ● ●● ● ● ●● ●● ●● ● 0.2 ● ● ● ● ●● ●● ● ●●●●●●●●●●● ● ●●●●● ●●●●● ●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●● ● ● ● ●● ● ● ● ●●● ●● ●●●●● ●●●●●● ●●● ●●●●● ●●● ●●● ● ●●●●●● ●●●●● ●●● ● ● ● ● ● ●●● ●●● ●● ● ●● ●● ●● ●●●●●●●●●●●●●●●●●●●● ●●● ● ●●● ●●●●●● ●●●●● ●●●●●●●●● ●●● ● ● ● ● ● ● ●●● ● ●●●●●●●● ●●●●● ●●●●●●●●●● ●●●●●●●●●●● ●●●●●●●●●●● ●●●●●●●●● ● ● ●●●● ● ● ●● ● ● ● ● ●●● ● ● ● ●●●●● ● ● 3. Connections ● ●● ● ● ● McCarthy, D. J. and G. K. Smyth (2009). Testing significance relative to a fold-change ● ●●●●●●●●●● ●● ●●●●●●●● ● ●●●●●●●● ● ●●● ●●● ●●●●●● ●●●●●●●●●●●● ●● ● ● ● ● ●● ●●●● ● ●● ●● ●●● which is one minus the agreement (2). In the absence of other considerations, the Bayes rule ●● ● ● ●●● ● ●●● ●●● ●●● ● ●● ● ● ●● ● ● ● ●● ● p( ) ●● ● ● ● ●●● ●● ● ● ● ●● ●● ● ● ● ● ●●●●● ●●●●●●●●● ●●●●●● ● ●●●●●● ●● ●●●●●●● ●●●●●●●●●● ● ●●●●● ●●●●● ● ●● ● ●● ● ● ● ●● ●●● ● i ●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●● ●●●●●● ● ●●●●● ● ●● ●● ●●●●● ● ● ●● ●● ●● ●● ● ● threshold is a TREAT. Bioinformatics 25 (6), 765–771. ●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●●●●●● ●● ● ●●●●● ● ● ● ● ●●● ●●● ●● ●● ●● ● ● ● ●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●● ●●● ● ●●●●●● ●●●●●●●●● ● ●●● ● ● ●● ●● ●● ●● ●●●● ●●●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●●● ●●●● ● ●● ●● ● ●●●● ●● ●● ● ● ● ● ● ●●● ●● ●● ●●● ● ●●● ● ●●●●●●● ● ●●●●●●●●●●●●●●●●●● ● ●●●●●●●●●●●●● ●●●●●●●●● ●●●●●●●●●● ● ● ● ● ● ● ●● ●● ●● ●● ● ●●● ● 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●● ●● ●● ●●●●●●● ●●●● ● ● ●● ●● ● ●● ● ●●● ●●● ●● ●●● ●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●● ●●●●●●●●●●● ● ●●● ●●●● ● ● ●● ●● ●● ●●● ●●● smallest such that when ranking units ●●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● α ●● ●●●●●●●● ●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●● ● ●● ●● ● ●●●● ● ● ●● ● ● ● ● ●● 0.50 ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●● ●●●●●●● ●●● ●●●●●●●●● ●●● ●● ●● ● ●● ● ● ●● ●●● ●●● ●● ● for loss L degenerates to δ(D ) = 0. Degeneration is avoided if we enforce on the reported ●●●●●● ●● ●●●●● ●● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● α i σ σ Morris, A. P., B. F. Voight, T. M. Teslovich, T. Ferreira, A. V. Segre, V. Steinthorsdottir, ●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●● ●●● ●●●● ●●●●●●● ● ●●● ●● ●●● ● ●●● ●●● ●● σ σ ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●● ●● ●● ●● ● ● ● ● ●● ●● 0.05 ●●●●●● ●●● ●●●●●●●●●● ●●●●●●●● ●●●●●●●●●●●●●●●● ●●●●●●●●●●● ● ● ●●●● ● ● ●● ●●● ●●● ●● ● ●●● ● ● ●● ● ● probability exceedance ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●● ●●●●● ● ●●● ● ●●●● ●●● ● ● ● ● ●●● R. J. Strawbridge, H. Khan, H. Grallert, A. Mahajan, et al. (2012). Large-scale association ●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●● ●●●● ●●●●●● ●● ● ●● ●●●●●● ●●●●● ● ● ●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●● ●●● ●●●●●●●●●● ● ● ● ● ● 0.1 ●●● Ai ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●● ●● ●● ●● ●●●●●● ● ● ● ●●● 3.1. Connection to Bayes rule ● ● ●● ● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●●●●●●● ●●●●●●●● ● ●●● ●● ●● ●● ●●● ●●● analysis provides insights into the genetic architecture and pathophysiology of type 2 ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●● ●●●●● ●●●● ●● ● ● ●● ●● ● ●● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●● ● ● ●● ● ● ●● ● ● ●● ●● rank the additional structure that it share with the true relative rank 1−F (θ )theproperty ●●●●●●●●●●● ●●●●●●●●●●●●●●●● ●● ●●● ●● ● ● ●●● ● ● ●● ●● ●● ● ● ● ● Measurement model simulation with and ✓ independenti 0.01 ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●0.02●●●●●●● ●●●● ●●●● ●●●● ●● ●● ● ● ●● ● ● ●0.05● ●●● ● ● ● 0.10 ●●● i i ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●● ●● ● ●●● ●●●● ●●●● ● ● ● ●● by the posterior probability of being in diabetes. Nature genetics 44(9), 981–990. ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●● ●●● ●●● ● ●●● ●●● ● ● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●● ●●●●●●● ●●●●●●● ● ● ●● ●● ●● ● ● ● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●● ●● ●●● ● ●● ●● ●●● ● ● ● ● ● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ●●●●●● ● ●● ●● ●●● ●● ● ●● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●● ●● ●● ●●●●●●●● ● ● ●● ● ● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ●● ●●● ● ●● ●●●●●● ● ●● ● ● ● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●● ● ●●●●●●● ● ● ● ● The proposed r-values are not Bayes rules in the usual sense, however there is a connection ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●● ●● ● ●●●● ●● ●●● ● Niemi, J. (2010). Evaluating individual player contributions in . In JSM Proceed- ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ●●●●●●Standard●●●●● ●●● ●●●●●●● ●Error●●●● ●●●●●●●● ● ● ● ● ● of being uniformly distributed over the population of units. Such a constrained Bayes rule ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●● ●●● ●● ● ●●● ● ● ● ●● ● ● 0.05 DLeBron.James = 439 585 ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●● ● ●●● ●●●● ●●●● ●● ● ● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●● ● ●●● ● ● ●●● ●● ●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●● ● ●● ●●●●●●● ● ●●●● ● ● ● ● ● ings, Statistical Computing Section, Alexandria, VA, pp. 4914–4923. American Statistical ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●●●●●●●●●●●●●●●●●●●● ●● ●●●●● ● ●● ● ● ● i ● ●● ●● ● ● ● ● ● the top α fraction of the system, unit Association. 0.20 to Bayesianthen inference minimizes if the one modified allows both objective a continuum function: of loss functions and a distributional Noma, H., S. Matsui, T. Omori, and T. Sato (2010). Bayesian ranking and selection methods r-values are Bayes rules under a continuum of losses and constraints using hierarchical mixture models in microarray studies. Biostatistics 11 (2), 281–289. remains in the top α fraction of constraintthe on the reported unit-specific (relative) ranks. To see this connection, we introduce Normand, S.-L. T., M. E. Glickman, and C. A. Gatsonis (1997). Statistical methods for a collection of loss functions riskα + γαP {δ(Di) ≤ α} profiling providers of medical care: issues and applications. Journal of the American ranked list. Statistical Association0.10 92(439), 803–814. 0 Improved ranking and selection 25 loss constraint Paddock, S. M. and T. A. Louis (2011). Percentile-based empirical distribution function 0.50 proportion of set detected by RN 0.002 0.005 0.010 0.020 0.050 0.100 0.200 0.500 1.000 estimates for performance evaluation of healthcare providers. Journal of the Royal Sta- where γα is chosen to enforce the (marginal) size constraint P {δ(Di) ≤ α} = α. tistical Society: Series C (Applied Statistics) 60(4), 575–589. Lα(a, θi)=1Improved− 1( rankinga and≤ selectionα, θ21i ≥ θα) α Improved ranking and selection 19 0.05 r = 0.016

Gene10 20 set 50 analysis100 200 500 1000 where action a is a relative rank value in (0, 1), α ∈ (0, 1) indexes the collection, and again 0.20 −1 set size N θα = F (1 − α) is a quantiler-values in the population have a distinct of interest. signal/noise Specifically, notrade-offα−loss occurs if the inferred relative rank a and the actual relative rank 1 − F (θi) both are less than α. Fig. 2. RNAi example: From a recent version of Gene Ontology, 5719 terms (gene sets) annotate 0.10 r-values rank players better from mid-season between 10 and 1000 human genes. Shown is a summary of the integration of these terms with the Table 2. Leading free-throw shooters, 2013-2014 regular season of the National Basketball list of 984 genes detected by RNAi as being involved in influenza virus replication (from Hao et al. The marginal (pre-posterior) Bayes risk of rule δ(Di) is

2013). The x-axis shows set size and the y-axis shows the proportion of the set that was detected proportion of set detected by RNAi Association. From 461 players who attempted at least one freethrow,shownarethetop25 by RNAi. The plot is restricted to 3626 sets for which the observed proportion exceeds 5%. data, as validated on complete season data. players as inferred by r-value. Data Di on player i include the number of made free throws yi and the number of attempts ni.Othercolumnsindicatefree-throwpercentageFTP= yi/ni, Improved ranking and selection 15 0.05 r−value MLE riskα =1−MLE P {δ(Di) ≤ α , θ ≥ θα} , (9) which is the maximum likelihood estimate (MLE) of the underlying ability θi;posteriormean 0.30 posterior mean Hao, L., Q. He, Z. Wang, M. Craven, M. A. Newton, and P. Ahlquist (2013). Limited E(θi|Di),r-valueinf{α : P (θi ≥ θα|Di) ≥ λα};qualifiedrank,Q.R,whichistherankofFTP agreement of independent RNAi screens for virus-required host genes owes more to false- 10 20 50 100 200 500 1000 MLE ● a. MLE b. p−value negative than false-positive factors. PLoS computational biology 9 (9), e1003235. amongst players for whom yi ≥ 125;andranksassociatedwiththeMLE,posteriormean,and ● ● ● ● ● ● ● ● which● ● is● one minus the agreement (2). In the absence of other considerations, the Bayes rule r-value. ● ● ● ● MLE p−value Jost, J. and X. Li-Jost (1998). Calculus of variations, Volume 64. Cambridge University set size N ● ● ● ● ● ● Press. player iyi ni FTP PM RV Q.R MLE.R PM.R RV.R ● ● ● N[ok] N[ok] 0.25 ● ● ● for loss Lα degenerates to δ(Di) = 0. Degeneration is avoided if we enforce on the reported Brian Roberts 125 133 0.940 0.913 0.002 1 17 1 1 ● ● ● ● Kass, R. E. and A. E. Raftery (1995). Bayes factors. Journal of the American Statistical ● ● ● ● Ryan Anderson 59 62 0.952 0.898 0.003 15 2 2 ● ● 1 Association 90(430), pp. 773–795. ● ● rank the additional structure that it share with the true relative rank 1−F (θi)theproperty Danny Granger 63 67 0.940 0.893 0.005 16 3 3 ● 1/4 Kendziorski, C., M. Newton, H. Lan, and M. Gould (2003). On parametric empirical ● ● Bayes methods for comparing multiple groups using replicated gene expression profiles. Kyle Korver 87 94 0.926 0.892 0.008 19 4 4 ● of being uniformly distributed over the population1/16 of units. Such a constrained Bayes rule ● Statistics in medicine 22(24), 3899–3914. Mike Harris 26 27 0.963 0.866 0.010 14 15 5 0.20 ● 0 MLE ● ● ● ● ● ● ● ● ● then minimizes the modified objective function:−1/16 Laird, N. M. and T. A. Louis (1989). Empirical Bayes ranking methods. Journal of Edu- 1.0 J.J.Redick 97 106 0.915 0.886 0.011 22 6 6 ● ● ● ● ● ● cational and Behavioral Statistics 14 (1), 29–46. ● ● ●● −1/4 ● ● ●● ● c. posterior mean d. maximal agreement ●● ● ● ● ● ●● 105 116 0.905 0.880 0.016 25 8 7 ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ●● ● −1 ● ● ● ●● ● ●● ● ● ● ● ● Lehmann,Sports E. (1986). Testing statistics statistical hypotheses (2nd 2013-14 ed.). Wiley series in probability ● ● ● ●●● ●●● ● ●● ●●●● Mike Muscala 14 14 1.000 0.844 0.017 7 34 8 ●● ●● ● ●● ●●●●●● ● ●●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ●●●● ● ● ●●● ● ● ●● ● ● ● ●● ● ● ● ● ●● ●● ●● ●●● 0.8 ● ● ● ● ●● and mathematical statistics: Probability and mathematical statistics. Wiley. ● ● ● ●● ●●●●●● ●●● ●● ● ●●●●● posterior mean risk + γ P {δ(D ) ≤ α} ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● Dirk Nowitzki 338 376 0.899 0.891 0.018 2 30 5 9 ● α α(X−R)/(X+R) i ● ● ● ● ● ●● ●● ●●● ● ● ● 0.15 ● ● ● ●● ● ●● ● ● ● ●● ● ●● ● Fig. 3. Threshold functions, T2D example, data and axes as in Fig 1: Calculations use an inverse- ● ● ● ● ● ● ● ●●● ●●●● ● ● ● ● ● ●● ●● ● ● ● ● ● ● 2 ● ● ● ● ● ● ● ● ● ● N[ok] gamma model for .Fortytwothresholdfunctionsareshown,rangingin values from a small Leng, N., J. A. Dawson, J. A. Thomson,NBA V. Ruotti, A. I. Rissman, B. M. Smits, J. D. ● ●●●●● ●● ●●● ●● TreyBurke 102 113 0.903 0.877 0.018 28 9 10 σ α ● ● ● ● ● ● ● ●● ●● ● ●● ●●●●● ● ●● ●● ● ● ● positive value (red) just including the first data point up to α =0.10 (blue). (Most data points are Haag, M. N. Gould, R. M. Stewart, and C. Kendziorski (2013). EBSeq: an empirical ● ● ● ●●● ● ● ● ●●● ● ● ● ● ● ●● ● ● Fig. 5. Ranking via various methods compared to r-value ranking; RNAi example; data and axes as ● ● ● ●● Reggie Jackson 158 177 0.893 0.877 0.024 3 32 11 11 truncated by the plot, as in Fig 1; also, the grid is uniform on the scale of log [− log (α)].) Units bayes hierarchical model for inference in rna-seq experiments. Bioinformatics 29 (8), ● ● ● ● ● ● 2 2 0.6 ● ●● ● in Fig. 2. Plotted is (X − R)/(X + R) where X is the rank (from the top) of the set by the method ● ● ●● ●● ● ● associated with a smaller α (i.e., more red) are ranked more highly by the given ranking method. 1035–1043. ● ● ●● ● ● ● Kevin Martin 303 340 0.891 0.882 0.025 4 33 7 12 where γ is chosenbeing compared, to enforce and R is the rank by the r-value. (marginal) size constraint P {δ(D ) ≤ α} = α. ● ● ● ● ● α i ● ● ● ● ● Note: ranking by posterior expected Two units landing on the same curve would be ranked in the same position. ● ● ● ●● ● ● ● Gary Neal 94 105 0.895 0.869 0.025 31 14 13 Lin, R., T. A. Louis, S. M. Paddock, and G. Ridgeway (2006). Loss function based ranking ● ●

● 0.10 ● ● ● in two-stage, hierarchical models. Bayesian Analysis 1 (4), 915–946. ● ● ● D.J. Augustin 201 227 0.885 0.873 0.031 5 38 12 14

461 players 0.4 ● rank is essentially the same as ranking ● ● Stephen Curry 308 348 0.885 0.877 0.031 6 39 10 15 McCarthy, D. J. and G. K. Smyth (2009). Testing significance relative to a fold-change Free Throw Percentage Free Throw threshold is a TREAT. Bioinformatics 25 (6), 765–771. PattyMills 73 82 0.890 0.860 0.032 34 19 16 58029 free throw attempts ● by posterior mean Henderson, N.C. and Newton, M.A. (2013). ● CourtneyLee 99 112 0.884 0.861 0.035 40 18 17 Morris, A. P., B. F. Voight, T. M. Teslovich, T. Ferreira, A. V. Segre, V. Steinthorsdottir, 0.2 ● ● ● ●

0.05 ● R. J. Strawbridge, H. Khan, H. Grallert, A. Mahajan, et al. (2012). Large-scale association Steve Nash 22 24 0.917 0.834 0.039 20.5 44 18 ● ● ● ● ● analysis provides insights into the genetic architecture and pathophysiology of type 2 Greivis Vasquez 95 108 0.880 0.857 0.040 41 22 19 ● ● Making the cut: improved ranking and selection diabetes. Nature genetics 44(9), 981–990. ● ● ● ● ●

0.0 Robbie Hummel 15 16 0.938 0.825 0.043 18 55 20 ● Niemi, J. (2010). Evaluatingespn.go.com individual player contributions in basketball. In JSM Proceed- ● Mo Williams 78 89 0.876 0.850 0.046 42 24 21 ● ings, Statistical Computing Section, Alexandria, VA, pp. 4914–4923. American Statistical 1 5 10 50 100 500 1000 for large-scale inference. arXiv:1312.5776 ● ● ● ● ● ● ● ● ● ● ● Association. 703 805 0.873 0.870 0.048 7 45 13 22

# Free Throw Attempts 0.00 Aaron Brooks 83 95 0.874 0.850 0.049 44 26 23

Noma, H., S. Matsui, T. Omori, and T. Sato (2010). Bayesian ranking and selection methods E[ similarity_t{} | complete season ] Ranks(theta) , Ranks.hat[midseason] using hierarchical mixture models in microarray studies. Biostatistics 11 (2), 281–289. 371 426 0.871 0.865 0.050 8 47 16 24 R package: rvalues Nando de Colo 31 35 0.886 0.831 0.057 37 48 25 5 10 15 20 25 Normand, S.-L. T., M. E. Glickman, and C. A. Gatsonis (1997). Statistical methods for profiling providers of medical care: issues and applications. Journal of the American Statistical Association 92(439), 803–814. t = rank from top http://www.stat.wisc.edu/~newton/ Paddock, S. M. and T. A. Louis (2011). Percentile-based empirical distribution function Thursday, Julyestimates 10, 14 for performance evaluation of healthcare providers. Journal of the Royal Sta- tistical Society: Series C (Applied Statistics) 60(4), 575–589.