Confidence Intervals for Nonparametric Empirical Bayes Analysis

Nikolaos Ignatiadis Stefan Wager [email protected] [email protected] September 2021

Abstract In an empirical Bayes analysis, we use data from repeated sampling to imitate in- ferences made by an oracle Bayesian with extensive knowledge of the data-generating distribution. Existing results provide a comprehensive characterization of when and why empirical Bayes point estimates accurately recover oracle Bayes behavior. In this paper, we develop flexible and practical confidence intervals that provide asymptotic frequentist coverage of empirical Bayes estimands, such as the posterior mean or the local false sign rate. The coverage statements hold even when the estimands are only partially identified or when empirical Bayes point estimates converge very slowly.

Keywords: Empirical Bayes, mixture models, local false sign rate, partial identifi- cation, bias-aware inference

1 Introduction

Empirical Bayes methods enable frequentist estimation that emulates a Bayesian oracle. Suppose we observe Z generated as below, and want to estimate θG(z),

µ G, Z p( µ), θG(z) = EG h(µ) Z = z , (1) ∼ ∼ ·   for some known function h( ) R. Given knowledge of G, θG (z) can be directly evaluated via Bayes’ rule. An empirical· Bayesian∈ does not know G, but seeks an approximately optimal estimator θˆ(z) θG(z) using independent draws Z1,Z2, ..., Zn from the distribution (1). The empirical≈ Bayes approach was first introduced by Robbins[1956] and has proven to

arXiv:1902.02774v4 [stat.ME] 8 Sep 2021 be successful in a wide variety of settings with repeated observations of similar phenomena, such as genomics [Efron et al., 2001, Love et al., 2014], education [Lord, 1969, Gilraine et al., 2020] and actuarial science [B¨uhlmannand Gisler, 2006]. Table1 provides concrete applications of model (1) for these subject areas. In all examples, the posterior mean θG(z) = EG µ Z = z is a statistic of interest, as it describes the (mean squared error) optimal shrinkage rule for estimating µ. In the genomics application, it is also of interest to   determine the local false-sign rate θG(z) = P µz 0 Z = z , i.e., the posterior probability that µ has a different sign than Z. ≤   As elaborated later, there is by now a large literature proposing a suite of estimators θˆ(z) for θG(z). Many of these estimators have theoretical guarantees under nonparametric specification of G, say G , where is a convex class of distributions. The goal of this ∈ G G

1 Subject i Zi µi Zi µi · ∼ Actuarial science Contract Number of insurance claims Risk profile Poisson (µ ) i Education Student Score in test with N Latent ability Binom(N, µi) multiple choice questions

Genomics Gene t-statistic comparing Standardized (µi, 1) N expression between conditions effect size

Table 1: Example applications for empirical Bayes inference in model (1). paper is to move past point estimation, and develop nonparametric confidence intervals for θG(z), i.e., intervals with the following property:

ˆ ˆ+ α(z) = θα−(z), θα (z) , lim inf PG [θG(z) α(z)] 1 α for all G . (2) I n ∈ I ≥ − ∈ G h i →∞ Despite widespread use of empirical Bayes methods, the problem has received surprisingly little attention. In fact, we are not aware of confidence intervals with property (2) beyond two special cases: one proposal by Lord and Cressie[1975] for inference about the posterior mean in the binomial model and another by Robbins[1980] for the same task in the Poisson model.

1.1 Motivating application: Predicting automobile insurance claims To motivate our interest in confidence intervals of the form (2), we revisit the historical work of Bichsel[1964]. Bichsel developed a theoretical framework for assigning automobile insurance premium rates, in a way that accounts for the claims experience of each individual. He analyzed a dataset (Table2) of claims made in the year 1961 by holders of a Swiss automobile insurance policy. Bichsel posited that Zi(t), the number of claims made in year t by the i-th insurance holder, is distributed as Poisson (µi), where µi is i’s latent risk. µi is a random draw from a distribution G that captures the heterogeneity of the insurance portfolio. Bichsel further assumed that the number of claims Zi(t),Zi(t0) in different years t = t0 are i.i.d. conditionally on µi. Given these assumptions, Bichsel sought to estimate the6 expected number of claims in the next year, among all insurance holders that made Zi(1961) = z claims in 1961,

θG(z) = EG Z(1962) Z(1961) = z = EG µ Z(1961) = z . (3)     Bichsel reasoned, that if θG(z) were known, it could be used by the insurance company for policy decisions, such as increasing or decreasing the premium of a policy holder with z claims in 1961. Since G, and consequently θG(z), were not known to Bichsel, he considered an empirical Bayes approach.1

1 θG(z) is a property of G, i.e., of the portfolio heterogeneity. The goal is to best assess how many claims will be made across all individuals in the portfolio that made z claims in 1961, and not to reason about the risk µi of any individual policy holder with Zi(1961) = z. The problem of forming intervals containing the true µi (for individuals) is of scientific importance, see e.g., Morris[1983], Laird and Louis [1987], Armstrong, Koles´ar,and Plagborg-Møller[2020], Koenker[2020] for some proposals; however it is not the problem we consider in this work.

2 z # Zi = z θˆ (z) F-localization α(z) AMARI α(z) { } NPMLE I I 0 103704 0.14 0.13 – 0.14 0.13 – 0.14 1 14075 0.25 0.23 – 0.27 0.24 – 0.26 2 1766 0.44 0.36 – 0.53 0.38 – 0.49 3 255 0.69 0.48 – 0.94 0.53 – 0.91 4 45 0.82 0.52 – 1.64 0.58 – 1.39 5 8 ≥ Table 2: Empirical Bayes confidence intervals in an actuarial application. The first two columns report the number of claims z made in 1961 by 119853 holders of a Swiss automobile insurance policy [Bichsel, 1964]. The next column reports the nonparametric maximum likelihood (NPMLE) point estimates of the posterior mean θG(z) = EG µ Z = z under model (1) with Z µ Poisson (µ). The last two columns report the two types of ∼   confidence intervals developed in this work; the F -localization intervals (Section 2) and

AMARI intervals (Section 4).

The problem of point estimation for θG(z) is well understood. One popular nonparamet- ric solution2 is to first estimate G through the nonparametric maximum likelihood estimator (NPMLE) of Kiefer and Wolfowitz[1956] and Simar[1976]: one estimates G as the maxi- z mizer of the marginal log-likelihood log(fG(Zi)), fG(z) = exp( µ)µ /z! dG(µ), among i − all possible prior distributions G. Then, with G in hand, one estimates θG(zb) through the ˆ P R plug-in principle, i.e., θG(z) = θGb(z) (shown in the third column of Table2). However, in so far as θG(z) may be used forb policy decisions of the insurance company, it is also important to assess the uncertainty in estimating it. In this paper, we develop two complementary approaches that address the problem of inference for empirical Bayes estimands, and enable the construction of intervals with the property (2) under the general model (1). The last two columns of Table2 show the two confidence intervals that we propose for Bichsel’s data. The assumption we make in forming these intervals is that G is supported on [0, 5]. The ‘F-localization’ intervals (third column of Table2) have simultaneous coverage for all z, while the ‘AMARI’ intervals (fourth column) have pointwise coverage. We next provide a high-level overview of our two constructions.

1.2 Empirical Bayes confidence intervals

In our approach the data analyst first specifies (1), i.e., θG(z), the empirical Bayes estimand of interest (e.g., the posterior mean (3)) and the conditional distribution of Z given µ (e.g., Poisson (µ)), which we represent by its conditional density p( µ) with respect to a σ-finite · measure λ on a subset of R (e.g., the counting measure on N 0). We also require the data ≥ analyst to specify a convex class of priors such that G . For example, for our analysis of Bichsel’s data in Table2, we assumed thatG G = ∈([0 G, 5]), where:3 ∈ G P ( ) := G distribution : support(G) for R. (4) P K { ⊂ K} K ⊂ 2In his work, Bichsel modeled G parametrically as a Gamma distribution with unknown parameters. 3We provide more guidance for choosing G in Section8.

3 1.2.1 F -localization Our first confidence interval construction is based on the notion of F -localization. The key idea is to construct a confidence set for the marginal distribution of Z and then determine all G consistent with this confidence set. Let us denote the marginal distribution of Z ∈ G by FG and its dλ-density by fG, i.e.,

fG(z) = p(z µ)dG(µ) ,FG(t) = PG [Z t] = 1 (z t) fG(z)dλ(z). (5) ≤ ≤ Z Z

We then define an F -localization as an (asymptotic) 1 α confidence set n(α) of distribu- tions, i.e., a set such that − F

lim inf G [FG n(α)] (1 α) 0. (6) n P →∞ { ∈ F − − } ≥

With an F -localization n(α) in hand, and deferring the construction of such to Section2, F + we can form confidence intervals α(z) = [θˆ−(z), θˆ (z)] for θG(z) by letting, I α α + θˆ−(z) = inf θG(z) G ( n(α)) , θˆ (z) = sup θG(z) G ( n(α)) , (7) α { | ∈ G F } α { | ∈ G F } where ( ) = G FG . (8) G F ∈ G ∈ F  ˆ ˆ+ The intervals (7) satisfy (2), since PG[θG(z) [θα−(z), θα (z)]] PG [FG n(α)] , and the same argument also demonstrates that coverage∈ holds simultaneously≥ over∈ all F possible empir- ical Bayes estimands θG(z) = EG h(µ) Z = z , where both z and h can vary. In Section2 we explain how for common choices of and n(α), (7) can be computed by solving two  G F linear programs. It is interesting to consider the F -localization approach in the context of the dichotomy of Efron[2014, 2019] on F - versus G-modeling. Under model (1), it is typically straight- forward to estimate FG, because the observed Zi are direct measurements from FG. In contrast, estimation of G is a difficult inverse problem. In some cases, the empirical Bayes estimand of interest may be expressed directly in terms of FG: for example, Robbins[1956] proves that the posterior mean in the Poisson model (p( µ) = Poisson (µ) in (1)) is equal to, ·

fG(z + 1) θG(z) = EG µ Z = z = (z + 1) , fG(z) = PG [Z = z] , z N 0. (9) fG(z) ∈ ≥   When a formula as (9) is available, it is convenient to proceed by F -modeling, i.e., to estimate FˆG and evaluate the corresponding F -formula by the plugin principle. In the Poisson model, letting FˆG be the empirical distribution of the Zi, the plugin principle leads to the estimate θˆRobbins(z) = (z + 1)# Zi = z + 1 /# Zi = z for (9). A caveat of F -modeling, however, is that, natural constraints{ on the empirical} { Bayes} estimand are not enforced. For instance, the posterior mean (9) in the Poisson model is non-decreasing in z, but θˆRobbins( ) does not enforce such monotonicity. In contrast, natural constraints such as monotonicity· are automatically enforced under G-modeling, that is, if one first estimates G and then lets ˆ θG(z) = θGb(z). For inference using F -localization, the two perspectives are complementary.b The data analyst constructs (6), an F -modeling task, and the Bayes structure of the problem is enforced through (7). For example, in the Poisson posterior mean problem, the lower bounds ˆ of the confidence intervals, θα−(z), are monotonic in z, and similarly for the upper bounds ˆ+ θα (z).

4 1.2.2 AMARI (Affine Minimax Anderson-Rubin Intervals) The F -localization approach is generic, streamlined to implement and enables simultaneous inference for all empirical Bayes estimands of interest. The F -localization intervals for a specific estimand θG(z), however, can be overly wide. Our second construction, AMARI, seeks to do better than F -localization, i.e., to provide shorter confidence intervals, by focus- ing on a specific estimand (compare e.g., columns 3 and 4 of Table2). The starting point for AMARI is the observation that we can write the empirical Bayes estimand θG(z), as a ratio of two linear functionals of G, i.e.,

h(µ)p(z µ) dG(µ) aG(z) θG(z) = = , (10) R p(z µ ) dG(µ) fG(z) where fG( ) is the marginal density ofRZ and aG(z) is used to denote the numerator. Hence, by a construction· that goes back to at least Fieller[1940], the following two hypothesis tests are equivalent for c R, ∈ lin lin H : θG(z) = c H : θ (z; c) = 0, where θ (z; c) = aG(z) cfG(z). (11) 0 ⇐⇒ 0 G G − By inverting the test for H0 : θG(z) = c we can form confidence intervals for θG(z). The upshot of (11) then is that it suffices to construct confidence intervals for linear functionals lin L(G) of G, say L(G) = θG (z; c). We provide the details of this reduction in Section4, and proceed to explain our approach to inference for linear functionals of G. Our core proposal is to estimate L(G) as an affine estimator, i.e., one of the form n 1 L = L(G) = Q(Zi), (12) n i=1 X where Q( ) is chosen to optimizeb a worst-caseb bias-variance tradeoff depending on the prior class .· To form confidence intervals, we first estimate the variance and worst-case bias of (12G) as

n n 2 1 2 V = Q (Zi) Q(Zi) n , (13) n(n 1)  −  i=1 i=1 ! − X X  b 2  2  B = sup BiasG[Q, L] ] , BiasG[Q, L] = Q(z)fG(z)dλ(z) L(G). (14) G ( n) − ∈G F Z  Here, theb worst case bias is computed with respect to ( n)(8), where n = n(αn) is G F F F an F -localization at level αn 0 as n . With V, B in hand, we build bias-aware → → ∞ confidence intervals α for L(G) [e.g., Armstrong and Koles´ar, 2018, Imbens and Manski, I 2004, Imbens and Wager, 2019] b b

1/2 α = L tα(B, V ), tα(B,V ) = inf t : P b + V W > t < α for all b B , (15) I ± | | ≤ n h i o where Wb (0b, 1)b is a standard Gaussian random variable. Sections3 and4 have formal results establishing∼ N asymptotic coverage properties for these intervals. Conceptually, our AMARI intervals build on recent work by Noack and Rothe[2019], who consider inference of average treatment effects in the fuzzy regression discontinuity design. There, the estimand also takes the form of a ratio of two linear functionals as in (10). Noack and Rothe[2019] name their approach after Anderson and Rubin[1949], who develop confidence intervals in the linear instrumental variable model, and so similarly we acknowledge Anderson and Rubin[1949] as part of the acronym AMARI.

5 1.3 Related Work As discussed briefly above, the empirical Bayes principle has spurred considerable interest over several decades. One of the most successful applications of this idea involves compound estimation of a high-dimensional Gaussian mean: we observe Z (µ,I), and want to ∼ N recover µ under squared error loss. If we assume that the individual µi are drawn from a prior G, then empirical Bayes estimation provides a principled shrinkage rule [Efron and Morris, 1973, Efron, 2011], whose theoretical properties are well-understood [Brown and Greenshtein, 2009, Jiang and Zhang, 2009]. The compound estimation problem when the individual Zi are Poisson, is also reasonably well understood [Brown et al., 2013]. The more general empirical Bayes problem (1) has raised interest in applications [Efron, 2012, 2016, Stephens, 2016, Koenker and Gu, 2017]; however, the accompanying formal results are less comprehensive. Muralidharan[2012] considered compound estimation in (1), when p( µ) is a one-dimensional exponential family. For p( µ) = p( µ), a smooth location· family, some authors, including Butucea and Comte[2009· ], Pensky· −[2017], have considered rate-optimal estimation of linear functionals of G; and their setup covers, for example, the numerator aG(z) in (10). The main message of these papers, however, is rather pessimistic: for example, Pensky[2017] shows that for many linear functionals, the minimax rate for estimation in mean squared error over certain Sobolev classes is logarithmic (to some negative power) in the sample size. G In this paper, we study a closely related problem but take a different point of view. Even if minimax rates of optimal point estimates θˆ(z) may be extremely slow (or even if estimands are only partially identified), we seek confidence intervals for θG(z) that still achieve accurate coverage in reasonable sample sizes and explicitly account for bias. The results of Butucea and Comte[2009] and Pensky[2017] imply that the length of our confidence intervals must go to zero very slowly in general; but this does not mean that our intervals cannot be useful in finite samples (and, in fact, our real data applications in Section5 and numerical experiments in Section6 suggest that they can be). To the best of our knowledge, with the exception of a handful of special cases, the problem of nonparametric inference in empirical Bayes problems has been left unexplored. Furthermore, practitioners using empirical Bayes ideas typically do not conduct inference and instead only consider point estimates of functionals of the unknown prior G. Among recent empirical Bayes works, Efron[2014, 2016, 2019] has advocated estimating (and report- ing) the variance of empirical Bayes estimates θˆ(z), and then using these variance estimates for uncertainty quantification. Such intervals, however, do not account for bias and so could only achieve valid coverage via undersmoothing; and it is unclear how to achieve valid undersmoothing in practice, noting the very slow rates of convergence in empirical Bayes problems. Efron[2014, 2016, 2019] himself does not suggest his intervals be combined with undersmoothing, and rather uses them as pure uncertainty quantification tools. Two notable existing results for inference as in (2) concern the posterior mean in the Binomial [Lord and Cressie, 1975, Lord and Stocking, 1976] and Poisson problems [Robbins, 1980, Karlis et al., 2018]. The F -localization approach we propose, generalizes the approach of Lord and Cressie[1975] and Lord and Stocking[1976] for inference of the posterior mean in the Binomial problem to the general empirical Bayes problem (1). We provide more details regarding this connection at the end of Section 2.1, and in Section 5.1 we revisit the data application of Lord and Cressie[1975]. The Poisson posterior mean problem is special, and particularly amenable to the task of forming confidence intervals, because of the existence of Robbins’ formula (9), as we elaborate in Section 7.1.

6 From a methodological perspective, our work relies upon advances in convex program- ming and is inspired by Koenker and Mizera[2014], who demonstrated that it is fruitful to revisit traditional ideas in empirical Bayes estimation through the lens of modern convex optimization. The F -localization approach requires solving two linear programs (or more generally, quasi-convex programs, cf. Section2). AMARI, our second approach, builds heavily on the literature on affine minimax estimation of linear functionals in Gaussian problems. Donoho[1994] and related papers [Armstrong and Koles´ar, 2018, Cai and Low, 2003, Donoho and Liu, 1991, Low, 1995, Johnstone, 2011] show that there exist affine esti- mators that achieve quasi-minimax performance and can be efficiently derived via convex programming. In turn, such affine estimators have recently proven useful for statistical inference in a number of settings, such as semiparametrics [Hirshberg and Wager, 2021, Kallus, 2020] and regression discontinuity designs [Armstrong and Koles´ar, 2018, Imbens and Wager, 2019, Eckles et al., 2020].

2 Simultaneous confidence intervals through F -localization

In this section we discuss our first approach, namely F -localization confidence intervals. The key idea is to ‘localize’ the marginal distribution FG with high probability, i.e., to construct a set n(α)(6) such that FG n(α) with (asymptotic) probability at least F ∈ F 1 α. n(α) then implies a confidence set G : FG n(α) for G, which we project − F { ∈ G ∈ F } to form confidence intervals for θG(z) as in (7). A convenient and universal F -localization proceeds by restricting F to be in a Kolmogorov- 1 n Smirnov ball around the empirical distribution function Fn(t) = 1 (Zi t), n i=1 ≤ P DKW b n (α) = F distribution : sup F (t) Fn(t) log (2/α) (2n) . (16) F t − ≤  ∈R q   By Massart’s tight constant for the Dvoretzky–Kiefer–Wolfowitz b (DKW) inequality [Mas- DKW sart, 1990], the above is a finite-sample F -localization, i.e., PG FG n (α) 1 α for all n and for any choice of p( µ) in (1). ∈ F ≥ −   This construction is rooted in· empirical Bayes tradition. In his discussion, Robbins

[1956], suggests that one could achieve asymptotically optimal empirical Bayes regret4 by choosing G such that supt FG(t) Fn(t) cn, with cn 0 as n and then using a | b − ˆ | ≤ → → ∞ plug-in estimate of the posterior mean θ(z) = θGb(z); see also Donoho and Reeves[2013] for a modernb refinement and implementationb to achieve optimal empirical Bayes regret in the Gaussian problem.5 ˆ+ ˆ The optimization problem (7) defining θα (z) (and similarly for θα−(z)) can be readily solved using modern convex optimization solvers, as long as can be efficiently discretized G (cf. Supplement D.2). The simplest case occurs when and n(α) may be represented by linear constraints. In that case, we may use the CharnesG andF Cooper[1962] transformation ˆ+ for linear-fractional programming and compute θα (z) by solving a linear program. As a concrete example (see (21) below for the general case), consider the prior class = ( ) G P K 4 2 That is, to learn a denoiser θˆ(z) such that E[(µ − θˆ(Z)) ] converges to the mean squared error Bayes risk for estimating µ in model (1). 5Anderson[1969] suggested to use the DKW band to form confidence intervals for the mean of a [0, 1]-valued random variable, as follows: one takes the minimum, resp. maximum of R zdF (z) subject DKW to F ∈ Fn (α) and F supported on [0, 1]; cf. Romano and Wolf[2000].

7 ˆ+ from (4) with = µ1, . . . , µp a finite set. Then, we can compute θα (z) by solving the linear program:K { } p

maximize h(µj)p(z µj)gj ζ, (g )p j j=1 j=1 X p p subject to gj = ζ, p(z µj)gj = 1, gj 0, j = 1, . . . , p, ζ 0, ≥ ≥ (17) j=1 j=1 X X p

sup gj p(˜z µj)dλ(˜z) ζFn(t) ζ log (2/α) (2n). t R ( ,t] − ≤ · ∈ j=1 Z −∞ X q  b The optimization variables ζ, g in (17) have the interpretation ζ = 1/f (z), g = ζ [ µ ] j G j PG j for j = 1, . . . , p, where G . · { } ∈ G 2.1 Refined F -localization The F -localization (16) is universal, and so works for any choice of likelihood p( µ) in (1). In some settings, however, it is also possible to construct F -localizations that· | are tailored towards properties of a specific choice of likelihood p( µ). We provide two such constructions in this section, the Gauss and χ2 F -localizations.· Empirically | we observe that our tailored F -localizations outperform (16) in− terms of the length of (7) for the empirical Bayes estimands we consider in our data examples (Section5) and simulations (Section6). It is an interesting future theoretical question to determine how one should choose the F -localization to direct power towards specific empirical Bayes estimands and likelihoods. However such considerations are outside the scope of this work.

Gauss-F -localization: Our first tailored F -localization is applicable in the Gaussian em- pirical Bayes problem, i.e., (1) with Z µ (µ, σ2) with known noise variance σ2 > 0. In this case, the marginal density f is the∼ Nconvolution of G with the Gaussian density, G and so is extremely smooth and can be estimated at quasi-parametric rates [Kim, 2014]. Here we build on this observation and seek to construct confidence intervals in terms of f ,M := supz [ M,M] f(z) , the supremum norm for the Lebesgue density of F on the k k∞ ∈ − | | compact set [ M,M], M > 0. To this end, we first form point estimates of fG( ) using the kernel density− estimator (KDE) ·

n 2 2 ˆK 1 Zi z sin (1.1z/2) sin (z/2) σ fn (z) = K − ,K(z) = 2 − , hn = . (18) nhn hn πz /20 i=1 log(n) X   K( ) is a smoothing kernel of infinite order6 that was studied by Politis and Romanop [1993]. We· form confidence bands using Efron’s multinomial bootstrap [Efron, 1979]. In the b-th bootstrap resample, we draw (W b,...,W b) Multinomial (n, (1/n, . . . , 1/n)) and compute 1 n ∼ n b ˆK ˆK,b ˆK,b 1 b Zi z cˆn = fn fn , fn (z) = Wi K − . (19) − ,M nhn hn ∞ i=1   X The proposition below defines the Gauss-F -localization and proves its asymptotic validity. 6K(·) is a superkernel [Devroye, 1992], i.e., it is absolutely integrable, integrates to 1 and its characteristic function is equal to 1 on [−1, 1].

8 Proposition 1 (Coverage of ,M localization in the Gaussian empirical Bayes problem). k·k∞ 2 b Assume (1) holds with p( µ) = (µ, σ ). Let cˆn(α) be the (1 α)-quantile of cˆn with respect to the Bootstrap distribution· | N in (19). Then, −

Gauss ˆK n (α) = F distribution with Lebesgue density f : f fn cˆn(α) (20) F − ,M ≤  ∞  asymptotically covers the true distribution at level 1 α, in the sense of (6). − Similarly to (16), (20) also enforces linear constraints on FG,G . Hence, if may be represented using linear constraints, then an analogous linear program∈ G to (17) canG be ˆ+ used to compute θα (z). More generally, whenever and n(α) may be represented by G F ˆ+ linear constraints, then, by the Charnes and Cooper[1962] transformation, θα (z)(7) may be computed by the following linear program,

+ θˆ (z) = sup a (z) f (z) = 1, G ζ ( n(α)), ζ 0 , where (21) α Ge | Ge ∈ G F ≥ n o ζ ( n(α)) = ζ G G ( n(αe)) , f (z) = p(z µ)dG(µ), a (z) = h(µ)p(z µ)dG(µ). G F · ∈ G F Ge Ge Z Z  e e χ2-F -localization: Our second construction pertains to categorical likelihoods, i.e., when 2 Zi and is a finite set with # = N + 1, N N. It is based on Pearson’s χ distance, ∈ Z Z Z ∈ N ˆ 2 χ2 (nfn(z) nf(z)) 2 n (α) = F ( ) with pmf f : − χN,1 α , (22) F ∈ P Z nf(z) ≤ − ( z=0 ) X ˆ 2 where fn(z) = # Zi = z /n is the empirical probability of z and χN,1 α is the 1 α quantile of the χ2 distribution{ with} N degrees of freedom. The validity of (22− ) in the− sense of (6) follows from standard asymptotics in categorical data analysis [Agresti, 2013] and coverage will (approximately) hold in finite samples as long as n fG(z) is sufficiently large for all z. The χ2-F -localization approach to inference in the empirical· Bayes problem is not new. Lord and Cressie[1975] and Lord and Stocking[1976] considered the Binomial problem with Zi µi Binom(N, µi) for N N and = ([0, 1]). They suggested to form confidence ∼ ∈ G P intervals for the posterior mean θG(z) = EG µ Z = z by the F -localization approach (7) 7 with n(α) as in (22). For the F -localization intervals in the introductory Poisson ex- F   ample (Table2), we used the χ2 F -localization with categories 0,..., 4 and grouping all − observations Zi 5 as a sixth category. For the χ2-F≥-localization, the Charnes-Cooper transformation (21) is not directly ap- plicable, yet the resulting optimization problem is quasi-convex and tractable; see Supple- ment D.1 for details.

3 Inference for linear functionals of G

Our next goal is to develop the AMARI approach for targeted inference about θG(z). How- ever, as a preliminary for this task, we need to develop some general results on inference for

7It may seem surprising that Lord and Cressie[1975] consider only the case of a Binomial likelihood, G = P([0, 1]) and posterior mean estimands. One reason is that they devise a numerical scheme for computing (7) that relies on these choices, cf. Section 5.1.

9 general linear functionals L(G) in the empirical Bayes problem; and this will be the focus of this Section. Formally, L( ) is a map from R, that is linear in G, i.e., · G → L λG + (1 λ)G = λL(G) + (1 λ)L(G) for all G, G , λ [0, 1]. (23) − − ∈ G ∈   The main reason we are interestede in confidencee intervals for eL(G) is that we will use these as building blocks of the AMARI intervals for θG(z) in Section4. Nevertheless, the class (23) includes functionals that are interesting in their own right. Some examples of linear functionals of interest include L(G) = PG [µ = 0] (the proportion of null effects), 2 L(G) = PG [µ 0] (the proportion of non-negative effects), and L(G) = EG µ (the second ≥ moment of the prior). Inference for PG [µ 0] has been considered for example by Es and   Uh[2005], Dattner et al.[2011], Efron[2016].≥ Greenshtein and Itskov[2018] and Brennan et al.[2020] form confidence intervals for L(G) using constructions that are analogous to the F -localization intervals developed in this work.8 Such F -localization intervals have simultaneous coverage over all possible choices of (linear) functionals, but can be overly wide for a specific linear functional L(G). Instead, the intervals we develop in this section are targeted towards a specific linear functional and so can be shorter.

3.1 Affine minimax inference for linear functionals Our key idea is to estimate the linear functional L(G) of G with an affine estimator, i.e., an estimator of the form L = Q(Zi)/n (12). The class of affine estimators is convenient because it enables explicit control of the worst case bias (14) and it is broad enough to include kernel density estimators asb inP (18) and the Fourier estimators of Butucea and Comte[2009], Pensky[2017]. The latter provably attain minimax optimal rates for estimation of linear functionals of G in the empirical Bayes problem, when p( µ) is a smooth location family. We choose Q( ) in a purely computational and data-driven· way. To do so, we first · construct a pilot F -localization n = n(αn) with αn 0 and a pilot estimate f¯n( ) of the F F → · marginal density fG( )(5). n could be, for example, any of the F -localizations described · F in Section2. For f¯n( ) we use the Kolmogorov-Smirnov minimum distance estimator, that was studied in the empirical· Bayes problem by Deely and Kruse[1968] and Heinrich and Kahn[2018]: 9

f¯n(z) = f (z), Gn argmin sup FG(t) Fn(t) . (24) Gbn ∈ G t − ∈G  ∈R  ¯ ¯ n, fn enable us to navigate a bias-varianceb trade-off in choosingb Q( ). fn facilitates esti- matingF the variance of any fixed Q( ), through the quadratic form (in· Q( )), · · 2 2 Var ¯ [Q] = Q (z)f¯n(z)dλ(z) Q(z)f¯n(z)dλ(z) . (25) fn − Z Z  n facilitates computationd of the F -localized worst-case bias (14) of Q( ) among all priors F · Gn n := ( n)(8). With these two ingredients, we choose Q( ) = Qn( ) in a data- driven∈ G way, byG minimizingF the localized worst-case bias, subject to controlling· the· estimated variance of Q( ): · 2 1 minimize sup BiasG[Q, L] s.t. Varf¯n [Q] Γn,Q( ) constant in R [ M,M]. (26) Q( ) G n ≤ · \ − · ∈R n ∈G  8Solving (7) for linear functionals is typicallyd more straightforward compared to ratio functionals (10). For example, the Charnes and Cooper[1962] transformation is not required. 9Case-by-case constructions would be possible here too or one could use the NPMLE.

10 Here, M > 0 is a (large) constant, and we restrict attention to functions Q( ) that are constant outside the interval [ M,M] to avoid regularity issues at infinity and so· that our − inference is not unduly sensitive to outliers. Γn > 0 is a hyperparameter that controls the bias-variance trade-off. For smaller values of Γn, we enforce that the Q( ) solving (26) takes · on smaller values of Varf¯n [Q], at the cost of potentially increasing the F -localized worst-case bias. We explain how we choose Γn below, after first outlining how we solve (26). First, to (formally)d enforce Q( ) to be constant outside [ M,M] as in (26), we (formally) censor Z in (1) and define · −

ZM = Z if Z [ M,M],ZM = / if Z < M,ZM = . if Z > M. (27) ∈ − − In view of (27), we only need to define Q( ) on the set /, . [ M,M]. ZM has condi- M · M { } ∪ − tional density p (z µ) = p(z µ) for z [ M,M], p (/ µ) = ( ,M) p(z µ)dλ(z) and M ∈ − M −∞ p (. µ) = (M, ) p(z µ)dλ(z) with respect to the measure λ = δ/ + δ. + λ, where δx ∞ M R ¯M M is a point mass at x . The true marginal density fG (and estimated density fn ) of Z R M ¯M ¯ is supported on /, . [ M,M] with fG (/) = ( ,M) f(z)dz, fn (/) = ( ,M) fn(z)dz, and similarly for{ . and} ∪z− [ M,M]. −∞ −∞ To solve (26) we build∈ upon− a construction ofR Donoho[1994], who formalizesR a powerful heuristic due to Charles Stein on hardest one-dimensional subproblems. We refer the inter- ested reader to Supplement B.2 for details and proofs in the context of our application and also to Donoho and Liu[1989], Donoho[1994], Low[1995], Armstrong and Koles´ar[2018] and references therein. The consequence of interest here is that to solve (26), it suffices to solve the following surrogate optimization problem:

2 f M (z) f M (z) 2 G1 G−1 M δ sup L(G1) L(G 1) G1,G 1 n, − dλ (z) . (28) − − − ∈ G f¯M (z) ≤ n  Z n


The surrogate optimization problem is parameterized by another hyperparameter δ10 and it is a second order conic program (SOCP) [Boyd and Vandenberghe, 2004] that is tractable by modern conic optimizers, such as MOSEK [ApS, 2020].11 The value of the supremum in (28) is called the modulus of continuity ωn(δ) at δ > 0. We say that the modulus problem (28) δ δ δ δ is solvable at δ > 0 if there exist G1,G 1 n such that L(G1) , L(G 1) < and − ∈ G | | | − | ∞ 2 δ δ M M ¯M M 2 L(G1) L(G 1) = ωn(δ), n fGδ (z) fGδ (z) fn (z) dλ (z) = δ , (29) − − · 1 − −1 Z   . δ δ δ δ and we call G1,G 1 solutions of ωn(δ). G1,G 1 are close observationally, that is their marginal distributions− have distance at most δ/−√n in terms of the ‘pseudo’-χ2-distance in (28), and they exhibit the largest separation of the linear functional L(G). These two pri- ors determine the worst-case optimal Q( ) in (26) at a specific value of Γn that depends on δ, δ δ δ · δ δ i.e., Γn = Γn(δ). Let G0 = (G1 + G 1)/2 and define Q( ) = Q( ; δ) = Q( ; δ, ωn0 (δ),G1,G 1) as, − · · · −

M M M M M f δ ( ) f δ ( ) f δ (z) f δ (z) f δ (z) n ω0 (δ) G1 G−1 G1 G−1 G0 · n · − · − dλM (z) + L(Gδ). (30) δ  f¯M ( ) −  f¯M (z)   0  · Z  10 δ maps to the hyperparameter Γn of (26), see below.  11See SupplementE for implementation details including discretization considerations.

11 Algorithm 1: Affine minimax confidence intervals for linear functionals L(G).

1 Form a pilot estimate f¯n( ) of the marginal density fG( ) as in (24) and a pilot · · F -localization (6) n = n(αn). F F 2 Choose δn as in (31). 3 Solve the modulus problem (28) at δn and use the solution to compute Q( ) as in (30), as well as its worst-case bias B. · 4 Form the estimate L of L(G) as in (12) and its estimated variance V as in (13). 5 Form bias-aware confidence intervals asb in (15). b b

Here ωn0 (δ) is the derivative of ωn( ) at δ, in case it exists, or an element of the superdif- 12· ferential of ωn( ) at δ otherwise. Q( , δ) from (30) is optimal for the min-max prob- · 2 · 2 lem (26) at Γn = ωn0 (δ) . The worst-case squared-bias B (14) of Q( ) over n is equal to 2 · G (ωn(δ) δω0 (δ)) /4. To navigate the bias-variance trade-off, we allow δ = δn to vary with − n n, and we choose δn as the minimizer of the worst-case meanb squared error,

2 2 2 (ωn(δ) δω0 (δ)) /4 + ω0 (δ) = sup BiasG[Q( , δ),L] : G n + Γn(δ), (31) − n n · ∈ G among all δ ∆ (0, ), for a set ∆ bounded away from 0 and ,13 for which the modulus ∈ ⊂ ∞ ∞ problem is solvable. We then construct Q( ) = Q( , δn), which is optimal for (26) with 2 · · Γn = Γn(δn) = ωn0 (δn) and finally we form the confidence interval for L(G) as described in Section 1.2.2. Algorithm1 summarizes our proposal for inference of linear functionals L(G). We prove the asymptotic coverage of the proposed confidence intervals by leveraging the representation of Q( ) in (30) and by verifying a central limit theorem for L (12). · Theorem 2 . Assume that for all (Central limit theorem for affine minimax estimator)b G , the linear functional L(G) is well-defined with sup L(G) < and that f M ( ) 2(λM ). Ge Ge Furthermore,∈ G assume that, | | ∞ · ∈ L e e e δn δn ` u A. For each n, the modulus problem (28) has solutions G1 ,G 1 at δn [δ , δ ], where δ`, δu (0, ) are fixed (i.e., do not change with n). − ∈ ∈ ∞ B. G lies in the convex set of distributions (G ). G ∈ G M C. There exists η > 0 s.t. infz /,. [ M,M] fG (z) > η. ∈{ }∪ − ¯M ¯M M ¯M D. fn is a density ( fn (z)dλ (z) = 1 and fn 0). It holds that PG [An] 1, where ≥ → An is the event on which, R ¯M M M M fn ( ) fG ( ) cn, fGδn ( ) fGδn ( ) cn,FG n, (32) · − · ≤ 1 · − −1 · ≤ ∈ F ∞ ∞

for a sequence of constants cn 0 as n . → → ∞ M E. n and f¯ are independent of Z ,...,Zn. F n 1 12 0 ˜ 0 ˜ ˜ That is, ωn(δ) satisfies, ωn(δ) ≤ ωn(δ) + ωn(δ)(δ − δ) for all δ > 0. Such an element exists, because ωn(δ) is concave in δ > 0 [Rockafellar, 1970]. We provide details in Supplement B.1. 13 In our implementation, we use the concrete choice ∆ = ∆grid := {0.2, 0.4,..., 6.5, 6.7}.

12 Then, letting Q( ) = Q( ; δn)(30) and L (12) the affine estimator of the linear functional L(G), it holds that,· · b 1/2 L L(G) BiasG[Q, L] V D (0, 1), PG[ BiasG[Q, L] B] 1 as n , − − −→N | | ≤ → → ∞   . whereb BiasG[Q, L] is defined in (b14). It follows that the intervals (15) provideb asymptotically 14 correct coverage of the target L(G), i.e., lim infn PG [L(G) α] 1 α. →∞ ∈ I ≥ − We emphasize that Q( ) changes with n, and so, the central limit theorem above is that of a triangular array. The statistical· assumption driving Theorem2 is Assumption B, namely that model (1) holds with G . Using a good choice of prior class is critical, and we discuss this choice further in Section∈ G 8. The rest of the assumptions areG under control of the analyst and may be verified before any data analysis is conducted. Assumption A guarantees that we can solve (26) by convex programming, cf. Supplement B.2, and Assumption C is an overlap condition. Assumption D concerns the quality of the pilot localization n and pilot F density estimator f¯n, while Assumption E requires that both n and f¯n are independent F of Z1,...,Zn. Assumption E holds if we use sample-splitting. Following Hajek[1962] and Bickel[1982] we demonstrate that it suffices to retain an asymptotically vanishing fraction of the Zi to estimate n, f¯n. F Proposition 3. Suppose we use k = kn samples from model (1) to construct n(αn), f¯n F and the remaining n kn samples for step 4 of Algorithm1. Suppose further that k/n 0, − → αn 0 and k αn as n . Then, Assumption D of Theorem2, is satisfied in the following→ cases:· → ∞ → ∞

1. Gaussian likelihood: Z µ (µ, σ2), and we use the DKW (16) or the Gauss-F - localization (20) and f¯ is the∼ minimum N distance estimator (24). n

2. Binomial or Poisson likelihood: Z µ Binom(N, µ) or Poisson (µ), G is not degenerate15 and we use the DKW (16) or∼ the χ2-F -localization∼ (22) (wherein, in the

Poisson case, we treat all observations > M as a single category) and f¯n is the minimum distance estimator (24). In practice, we use the full data twice and do not sample-split; we have not observed any overfitting or loss of coverage thereby.

4 Pointwise confidence intervals with AMARI

In this section we return to our main task of forming confidence intervals for empirical Bayes estimands θG(z) and discuss our second approach, AMARI (Affine Minimax Anderson– Rubin Intervals). In contrast to the F -localization intervals, the AMARI intervals are targeted towards a specific empirical Bayes estimand and have a pointwise (rather than

14Although the result stated only holds elementwise, we can obtain a uniform statement in the sense of, e.g., Robins and Van Der Vaart[2006] by adding slightly more constraints on the class G. Specifically, η   M consider G = G ∈ G : infz∈{/,.}∪[−M,M] fG (z) > η , for some η > 0, i.e., qualitatively, prior distri- butions for which the induced marginal density cannot vanish anywhere. The proof of Theorem2 implies η η that then the above statements apply uniformly over G , lim infn→∞ inf {PG [L(G) ∈ Iα]: G ∈ G } ≥ 1 − α, η provided that PG [An] → 1 uniformly in G ∈ G , where An has been defined in the statement of Theorem2. 15 That is, PG [µ ∈ {0, 1}] < 1 in the Binomial case, and PG [µ = 0] < 1 in the Poisson case.

13 simultaneous) coverage guarantee. The upshot is that AMARI intervals can be substantially shorter. The starting point for AMARI is (10), i.e., the fact that we can write the empirical Bayes estimand θG(z) as a ratio of linear functionals of G, aG(z)/fG(z). Next, fix c R lin ∈ and write, L(G) := θG (z; c) := aG(z) cfG(z) as in (11). One may directly verify that L( ) is a linear functional of G, as defined− in (23). The Affine Minimax component of the· AMARI acronym refers to the fact that we will use the affine minimax approach of lin lin Section 3.1 to form confidence intervals α (z; c) for θG (z; c), treating the latter as a generic linear functional L(G). The Anderson-RubinI component of AMARI enables us to construct lin confidence intervals for θG(z) by lifting our intervals for θG (z; c) following the approach of Noack and Rothe[2019], who in turn build upon Anderson and Rubin[1949] and Fieller [1940, 1954]. The following Corollary captures the basic idea of our approach:

lin Corollary 4. Suppose that for each c R, we construct a confidence interval α (z; c) for lin ∈ I θG (z; c)(11) following Algorithm1. Suppose furthermore that the assumptions of Theorem2 lin hold for the linear functional L(G) = θG (z; c∗), where c∗ = θG(z). Then,

lin lim inf G [θG(z) α(z)] 1 α, α(z) = c 0 α (z; c) . (33) n P R →∞ ∈ S ≥ − S ∈ ∈ I  lin Proof. By definition of c∗ and α(z), it holds that θG(z) α(z ) 0 α (z; c∗). On lin S ∈ S ⇐⇒ ∈ I the other hand, θG (z; c∗) = aG(z) c∗fG(z) = aG(z) (aG(z)/fG(z))fG(z) = 0, and there- lin − lin − fore PG [θG(z) α(z)] = PG θG (z; c∗) α (z; c∗) . We conclude by Theorem2. ∈ S ∈ I We provide Corollary4 for intuition. However, the confidence set α(z) from (33) has S some disadvantages. First, α(z) will in general not be an interval. Second, computing lin S the interval α (z; c), even for a single c, is computationally demanding and requires the I lin solution of (28) along a grid of δ values (31), and so, computing α (z; c) for ‘all’ values of c is not computationally tractable. Instead, in our actual implementationI of AMARI, which we describe in Section 4.1 below, we use an ‘accelerated’ Anderson-Rubin procedure that is lin computationally streamlined (we only need to form α (z; c) for two values of c), and leads I 16 to a confidence interval α(z) for θG(z), rather than a confidence set. I 4.1 Implementation of Anderson-Rubin inversion for AMARI We now describe our actual implementation of AMARI. The key intuition is that we start ` u with a preliminary interval such that θG(z) [c , c ] with high probability, and then find ` u lin ` ∈ lin u the affine minimax Q ,Q (26) for θG (z; c ), resp. θG (z; c ). Then, for any other c, say c = κc` + (1 κ)cu, κ (0, 1), instead of resolving (26), we use Qc = κQ` + (1 κ)Qu. The variance− of Qc then∈ can be directly computed from the covariance of Q`,Qu, and− their individual variances, while the worst-case bias of Qc can be upper bounded by the convex combination Bc = κB` + (1 κ)Bu of the worst-case biases of Q` and Qu. Algorithm2 describes all steps of AMARI.− Step 4 can be computed efficiently using grid search, since ` u the evaluationb of α(bz; c) for differentb values of c [c , c ] is fast. As a consequenceI of Theorem2, we can now prove,∈ that the AMARI confidence intervals asymptotically covere the empirical Bayes estimand θG(z).

16The ‘accelerated’ Anderson-Rubin approach could be fruitful in other settings; for example it could allow replacing the local linear estimators in the fuzzy regression discontinuity approach of Noack and Rothe[2019] by the affine minimax estimators of Imbens and Wager[2019].

14 Algorithm 2: AMARI confidence intervals for empirical Bayes estimands θG(z)

1 Construct a pilot F -localization n = n(αn) at level αn, as in Step 1 of ` u F F Algorithm1. Let [ c , c ] be the F -localization interval (7) for θG(z) based on n. lin ` lin u F 2 Apply Algorithm1 to the linear functionals θG (z; c ) and θG (z; c ) defined in (11). Let Q`,Qu be the corresponding affine minimax kernels, L`, Lu the point estimates (12), B`, Bu the worst case biases (14), V `, V u the variances and ` u 1 n ` u n ` b b n u Cov L , L = n(n 1) i=1 Q (Zi)Q (Zi) i=1 Q (Zi) ( i=1 Q (Zi)) n . − − ` b bu c ` b b u 3 For c h= κc +i (1 κ)c ,P κ [0, 1], let L = κL +P (1 κ)L ,
P   Bdc = bκB`b+ (1 −κ)Bu, V ∈c = κ2V ` + (1 κ)2V u + 2−κ(1 κ)Cov L`, Lu and c − c c − − α(z; c) = L tα(B , V ) as in (15).b b b h i I ± ` u ` u 4 Reportb theb interval αb(z)b = [inf ,bsup ] [c , cb ], = c [cd, c ]b : 0 b α(z; c) . e b Ib b C C ∩ C { ∈ ∈ I } e Theorem 5 (Coverage of AMARI intervals). Consider the confidence intervals constructed in Algorithm2. Suppose the pilot F -localization interval endpoints c`, cu are finite for lin ` all n and let the assumptions of Theorem2 hold for L(G) = θG (z; c ) and L(G) = lin u 17 ` u θG (z; c ). Furthermore, assume that Q and Q are not perfectly anticorrelated, i.e., ` u u ` 1/2 there exists ε > 0, such that, PG[Cov[L , L ] (V V ) 1 + ε] 1 as n . Then, ≥ − → → ∞ lim infn PG [θG(z) α(z)] 1 α. →∞ ∈ I ≥ −  d b b b b The additional assumption of Theorem5 on the correlation between Q` and Qu is mild and can be verified from the data at hand; in applications we typically find a positive correlation.

5 Empirical applications

In this section we apply the F -localization and AMARI intervals developed above in the context of two different applications; one in education and one in genomics.

5.1 Predicting student ability in psychometric tests Lord and Cressie[1975] studied a dataset of scores by n = 12, 990 students on a psychological test with N = 20 multiple choice questions (5 choices per question) and posited that Zi µi Binom(20, µ ), where µ is the ‘true-score’ of student i [Lord, 1969]. Lord and Stocking∼ i i [1976] were working for the Educational Testing Service (ETS) at the time and the following motivation for confidence intervals of the posterior mean θG(z) = EG µ Z = z with the property (2) seems plausible: if the ETS were to use an estimate θˆ(z) for student assessment,   then it would be important to account for the uncertainty in estimating the regression function EG µ Z due to both variability (which can be large even for large sample sizes, e.g., n = 12, 990 in this example) and partial identification.18   The empirical frequencies fˆn(z) = # Zi = z /n of test scores are shown in Figure1a). { } Panel b) shows three 95% confidence intervals for θG(z) that make no assumptions on G,

17The proof of Theorem2 uses triangular array asymptotics, and so, the linear functional L(G) may depend on n. 18 In the Binomial empirical Bayes problem (p(· µ) = Binom(N, µ), N ∈ N>0) and without further restric-   tions on G, the posterior mean θG(z) = EG µ Z = z is only partially identified and cannot be consistently estimated, even as n → ∞. We discuss this issue further in Section 7.2.

15 a) b)

1.00 AMARI χ2-F-Loc DKW-F-Loc 0.3 0.75 ] z ) = z ( n 0.2 Z

ˆ 0.50 f | µ q [ E 0.1 0.25

0.0 0.00 0 5 10 15 20 0 5 10 15 20 z z

Figure 1: Empirical Bayes confidence intervals in a psychometric test with 20 1/2 questions taken by 12, 990 students [Lord and Cressie, 1975]. a) fˆn(z) vs. z, where fˆn(z) is the proportion of students that answered z out of 20 questions correctly. b) 95% confidence intervals for the posterior mean θG(z) = EG µ Z = z .   i.e., G = ([0, 1]) (4). The χ2-F -localization intervals (22) were developed by Lord and Cressie∈ G [1975P ], Lord and Stocking[1976]. We computed these intervals by grouping the lowest scores z = 0 and z = 1 together (to ensure the χ2-interval (22) has the right coverage) and then used the parametric convex programming approach from Supplement D.1 with the discretization ([0, 1]) ( ) and an equidistant grid on [0, 1] with 300 points. The intervals agree withP the ones≈ P reportedK inK Lord and Cressie[1975]. The latter used a numerical optimization routine due to Martha Stocking that leveraged the fact that in the Binomial empirical Bayes problem (N = 20) and when θG(z) is the posterior mean, then the worst case G in (7) must be discrete and supported on at most 11 points. We also report intervals based on the DKW-F -localization, as well as the AMARI intervals (with 2 pilot χ -F -localization using αn = 0.01). We see that, for low scores z, there is substantial uncertainty. For students with score z = 0, one could predict their true score as being almost 0, or one could predict their true score as better than random guessing (1/5); and both predictions would be consistent with the data. On the other hand, for intermediate values of z the intervals become substantially shorter. In this example we also observe that the DKW-F -Localization intervals are overly wide. The AMARI intervals are shorter than the χ2-intervals for most z; at the cost of no longer being simultaneous.

5.2 Identifying genes associated with prostate cancer Our next dataset is the ‘Prostate’ dataset [Efron, 2012, Singh et al., 2002], by now a classic dataset used to illustrate empirical Bayes principles. The dataset consists of Microarray expression levels measurements for m = 6033 genes of 52 healthy men and 50 men with prostate cancer. For each gene, a t-statistic Ti is calculated (based on a two-sample equal 1 variance t-test) and z-scores are calculated as Zi = Φ− (F100(Ti)), where Φ is the standard normal CDF and F ( ) is the CDF of the t-distribution with 100 degrees of freedom. We 100 · posit Zi µi · (µ, 1), where µi is the standardized effect size and is expected to be close to null for most∼ N genes [Efron et al., 2001]. We seek to form confidence intervals for two

16 empirical Bayes estimands. Our first estimand of interest is the posterior mean, θG(z) = EG µ Z = z , which could be used to denoise the noisy measurements Zi by θG(Zi) and to provide estimates of µi that   are (nearly) immune to selection bias [Efron, 2011]. The standard empirical Bayes approach provides point estimates of these oracle quantities by sharing information across genes, but the empirical Bayes estimation error may be rather opaque and so it is not clear to what extent the estimates E [µi Zi] eliminate selection bias. Our confidence intervals attach a | measure of uncertainty to the estimation of θG(z). Second, we considerb the local false sign rate θG(z) = PG µz 0 Z = z , which measures ≤ the posterior probability that the sign of an observed signal Zi disagrees with the sign of the   true effect µi. Local false sign rates provide a principled approach to multiple testing without assuming that the distribution G of the effect sizes µi is spiked at 0, and form an attractive alternative to the local false discovery rate, lfdr(z) = PG µi = 0 Zi = z , without requiring a sharp null hypothesis [Stephens, 2016, Zhu et al., 2018]. Inferential emphasis is thus placed   on whether we can reliably detect the direction of an effect. Below, for ease of visualization, we report results on the (substantively equivalent) quantity θG(z) = PG µi 0 Zi = z ≥ (instead of µz 0 Z = z ) so that the resulting confidence bands are monotonic in z. PG   While both the≤ posterior mean and the local false sign rate are routinely reported in the   analysis of genomics datasets [Stephens, 2016, Zhu et al., 2018], the statistical difficulty of estimating them in the Gaussian empirical Bayes model is vastly different. The posterior mean can be estimated at the quasi-parametric rate log(n)3/4/√n over the class of priors with Lebesgue density and finite first moment [Matias and Taupin, 2004]. Meanwhile,G minimax point estimates for the local false sign rate over Sobolev classes of priors converge at extremely slow rates, e.g., polynomial in 1/ log(n)[Butucea and Comte, 2009, Pensky, 2017]. Consequently, we expect that our confidence intervals for the posterior mean will be substantially shorter than the ones for the local false sign rate. We form 95% confidence intervals using 3 2 methods, namely the DKW (16) and Gauss (20) (with M = 3) F -localization methods,× as well as AMARI (with Gauss-F - localization pilot, αn = 0.01), each applied based on two specifications for . First, we consider the Gaussian location mixture G dG(µ) 1 µ u (τ 2, ) := G distribution : = ϕ − dΠ(u), Π ( ) , (34) LN K dλLeb τ τ ∈ P K  Z    where τ > 0, R, λLeb is the Lebesgue measure and ϕ is the standard Gaussian density. This is a naturalK ⊂ choice of smooth priors in the Gaussian empirical Bayes problem [Magder and Zeger, 1996, Cordy and Thomas, 1997] and the noise level τ provides an interpretable way of specifying the smoothness of the priors. Here we make the concrete choice = (0.252, [ 3.3]).19 G LNSecond,− we consider Gaussian scale mixtures with mode at zero, discretized as suggested by Stephens[2016], i.e., for 0 < τ` < τu and η > 1: dG(µ) µ dΠ(τ) (τ`, τu, η) := G dbn : = ϕ , Π ( τ`, η τ`, . . . , τu ) . (35) SN dλLeb τ τ ∈ P { · }  Z    2 1/2 20 We take τ` = 0.1, τu = 10.7 > (maxi Zi 1 ) and η = 1.1. Stephens[2016] argues that the unimodal Gaussian scale mixture− leads to more accurate inference provided that it  holds; our intervals allow a quantitative assessment of this claim for any analyzed dataset.

19 2 We discretize it as LN (0.25 , K) with K an equidistant√ grid on [−3, 3] of step size equal to 0.05. 20Stephens[2016] uses a coarser grid with η = 2.

17 a) c) e)

1.0 1.00 Gauss-F-Loc Gauss-F-Loc DKW band 2

DKW-F-Loc ] DKW-F-Loc

z 0.8

] AMARI AMARI

0.75 z 1 = =

Z 0.6 ) | z Z

( 0

0.50 0 | n b

F 0.4 µ ≥ [ 1 µ E 0.25 − [ 0.2 P 2 − 0.00 0.0 4 2 0 2 4 3 2 1 0 1 2 3 3 2 1 0 1 2 3 − − z − − − z − − − z b) d) f)

0.4 1.0 Gauss-F-Loc Gauss-F-Loc Histogram 2

KDE band DKW-F-Loc ] DKW-F-Loc

z 0.8

0.3 ] AMARI AMARI z

1 = =

Z 0.6 ) | z Z ( 0.2 0 0 n | ˆ f 0.4 µ ≥ [ 1 µ E 0.1 − [ 0.2 P 2 − 0.0 0.0 4 2 0 2 4 3 2 1 0 1 2 3 3 2 1 0 1 2 3 − − z − − − z − − − z

Figure 2: Empirical Bayes inference for the Prostate dataset [Efron, 2012, Singh et al., 2002]. a) Empirical distribution of the Zi and DKW-F -localization band. b) His- togram of the Zi and Gauss-F -localization. c) 95% confidence intervals for the posterior mean EG µ Z = z assuming G is a Gaussian location, resp. d) scale mixture. e) 95% confidence intervals for the local false sign rate EG µ 0 Z = z assuming G is a Gaussian   ≥ location, resp. f) scale mixture.  

Figure2 shows the results of the analysis. For the posterior mean, we observe that all intervals suggest that many effects are close to null and so there is substantial shrinkage towards zero. EG µ Z = z is almost flat in the interval [ 1.5, 1.5]; and we can say so with confidence. All four F -localization bands are quite similar,− while AMARI leads to substan-   tially shorter intervals, and the improvement is more noticeable for = (0.252, [ 3.3]). For the local false sign rate, the intervals are long when assuming GG LN (0.252, [−3.3]). The DKW-F -localization intervals perform worst, while the AMARI and∈ Gauss- LN F -localization− intervals perform comparably (with AMARI leading to shorter intervals only for more ex- treme values of z). As explained above, long confidence intervals are expected in this case. On the other hand, if we are willing to assume that G is a Gaussian scale mixture with mode at 0, then inference for the local false sign rate is much more precise, exactly as ar- gued by Stephens[2016]. However, the assumption is strong, and for example it implies that the local false sign rate at 0 is equal to 1/2; all intervals proposed here have vanishing length in that case.

18 a) b)

Spiky Spiky 1.2 NegSpiky NegSpiky 0.3 1.0 Leb

0.8 ) z

/dλ 0.2 ( )

0.6 G µ f ( 0.4

dG 0.1 0.2

0.0 0.0 4 2 0 2 4 − − 4 2 0 2 4 µ − − z

Figure 3: Priors used in simulations. a) Lebesgue density of priors and b) marginal density fG(z).

6 Simulations

The setting of our simulations is similar to the Prostate data example in Section 5.2. We consider model (1) with Z µ (µ, 1), n = 5000, and two different data-generating priors, ∼ N GSpiky = 0.4 (0, 0.252) + 0.2 (0, 0.52) + 0.2 (0, 1) + 0.2 (0, 22), N N N N (36) GNegSpiky = 0.8 ( 0.25, 0.252) + 0.2 (0, 1). N − N GSpiky was used in the simulations of Stephens[2016] and is a unimodal symmetric prior centered at 0, while GNegSpiky is a prior with strong peak just to the left of zero, reflecting many slightly negative effects. Figure3 shows the Lebesgue densities of the priors and the induced marginal densities fG(z). We seek to form 95% confidence intervals for the posterior mean and the local false sign rate using the DKW-F -localization, Gauss-F -localization (M = 4) and AMARI approaches (with pilot Gauss-F -localization, αn = 0.01) and equal to the Gaussian location mixture class (0.252, [ 4, 4]). G WeLN also consider− an additional plug-in baseline [Efron, 2016, Narasimhan and Efron, 2020]. We estimate G by (penalized) maximum likelihood over a flexible exponential family with a natural spline (5 degrees of freedoms) as the sufficient statistic and base measure U[ 4, 4]. We then obtainb θˆ(z) by applying Bayes rule with prior G. As is standard in the lit- − erature, this baseline constructs confidence intervals for θG(z) using the delta method, which captures the variance of θˆ(z) but not its bias. Such confidence intervalsb are only guaranteed to cover θG(z) in the presence of undersmoothing or if the parametric specification is correct. See SupplementF for implementation details of this plug-in baseline. Figure4 shows the results of the simulations for the posterior mean, averaged over 400 Monte Carlo replicates. The length of the different confidence intervals is qualitatively similar to what we observed in Figure2. The log-spline intervals are shortest; however they do not achieve nominal coverage, while all other methods do. The pointwise coverage of the F -localization intervals is close to 100%, while the coverage of AMARI is closer to the nominal 95%. The simultaneous coverage of AMARI for EG [µ Zi = z] as z varies in Figure4 is 75% for GSpiky and 70% for GNegSpiky. The F -localization| methods have simultaneous coverage above 95%. Figure5 shows the simulation results for the local false sign rate. Most conclusions are similar to the ones we made for the posterior mean. However, here the Gauss-F -localization

19 a) Spiky G b) Spiky G

3 Gauss-F-Loc ( ) 1.00 LN DKW-F-Loc ( ) 2 LN AMARI ( ) LN Logspline

] 0.75

z 1 Ground truth =

Z 0

| 0.50 µ [ 1 Coverage E − Gauss-F-Loc ( ) 0.25 LN DKW-F-Loc ( ) 2 LN − AMARI ( ) LN Logspline 3 0.00 − 3 2 1 0 1 2 3 3 2 1 0 1 2 3 − − − z − − − z c) NegSpiky G d) NegSpiky G

3 Gauss-F-Loc ( ) 1.00 LN DKW-F-Loc ( ) LN AMARI ( ) 2 LN Logspline

] 0.75

z Ground truth 1 = Z

| 0.50 0 µ [ Coverage E 1 Gauss-F-Loc ( ) 0.25 LN − DKW-F-Loc ( ) LN AMARI ( ) 2 LN − Logspline 0.00 3 2 1 0 1 2 3 3 2 1 0 1 2 3 − − − z − − − z

Figure 4: Simulation results: Inference for the posterior mean in the Gaussian empirical Bayes problem. a) Expected confidence intervals in the simulation with the prior GSpiky (36). 4 different inference methods are shown, as well as the ground truth as a function of z. b) Coverage of the above confidence intervals as a function of z. c, d) Inference results in the simulation with the prior GNegSpiky (36). leads to shorter intervals compared to the DKW-F -localization. Furthermore, in this case, both F -localization intervals and AMARI have pointwise coverage close to 100%; the reason is that the worst case bias is substantial, and so bias-aware intervals lead to conservative inference for most G . In fact, AMARI has simultaneous coverage above 95% for ∈ G P [µ 0 Z = z] as z varies in Figure5. ≥ | Gaussian scale mixture : We next repeat our simulations with the same settings, but using a different choice of ,G namely the Gaussian scale mixture class (35) (0.1, 15.6, 1.1). The scale mixture class G (0.1, 15.6, 1.1) is strongly misspecified for GNegSpikySN . This was SN detected by our proposed methods, as the intersection of FG : G (0.1, 15.6, 1.1) and { ∈ SN } F -localizations n was empty. Thus, in Figure6 we report the results of our simulations only for GSpiky.F We observe that the assumption that G is a scale mixture centered at 0, instead of a location mixture, leads to substantially more precise inference, and especially so for the local false sign rate.

Degrees of freedom for the logspline approach: One might at this point wonder whether one can reduce the bias of the plug-in logspline approach and achieve nominal

20 a) Spiky G b) Spiky G

1.00 Gauss-F-Loc ( ) 1.00 LN DKW-F-Loc ( ) LN AMARI ( ) ] LN z 0.75 Logspline 0.75

= Ground truth Z | 0.50

0 0.50 ≥ Coverage µ [ 0.25 Gauss-F-Loc ( ) P 0.25 LN DKW-F-Loc ( ) LN AMARI ( ) LN Logspline 0.00 0.00 3 2 1 0 1 2 3 3 2 1 0 1 2 3 − − − z − − − z c) NegSpiky G d) NegSpiky G

Gauss-F-Loc ( ) 1.00 LN DKW-F-Loc ( ) LN 0.8 AMARI ( ) ] LN z Logspline 0.75

= Ground truth 0.6 Z |

0 0.50 0.4 ≥ Coverage µ [ Gauss-F-Loc ( ) P 0.25 LN 0.2 DKW-F-Loc ( ) LN AMARI ( ) LN Logspline 0.0 0.00 3 2 1 0 1 2 3 3 2 1 0 1 2 3 − − − z − − − z

Figure 5: Simulation results: Inference for the local false sign rate in the Gaussian empirical Bayes problem. The panels are analogous to the ones of Figure4. coverage by increasing the degrees of freedom of the spline; we explore this in Figure7 for the above simulation with the prior GNegSpiky. In general, coverage indeed improves as the degrees of freedom increase; however, with many degrees of freedom, the variance can be so large that the resulting confidence intervals are longer than the intervals proposed in this work. More importantly it is not clear a-priori, i.e., without knowing the ground truth, how to properly undersmooth the plug-in estimation and choose a number of degrees of freedom that provides good coverage. Efron[2016] does not suggest undersmoothing, and instead, acknowledges that using a low-dimensional parametric family induces ‘definitional bias’ in point estimates, ‘the pay-off being reduced variability’. On the other hand, as this example highlights, if we want confidence intervals that cover the true local false sign rate, it is important to explicitly account for bias.

7 On the asymptotic power of F -Localization and AMARI

Our goal in this work is to provide a unified approach for constructing intervals with the coverage property (2) that lead to useful confidence statements in applied situations (cf. Sections5 and6). Given the generality of (1), we suspect it may be difficult to develop a unified theory of optimality. Nevertheless in this section we consider the issue of optimality and asymptotic relative efficiency in two concrete settings to provide the following conceptual insights. First, we describe a situation in which AMARI is asymptotically efficient and

21 a) Posterior mean b) Posterior mean

Gauss-F-Loc ( ) 1.00 2 SN DKW-F-Loc ( ) SN AMARI ( ) SN Ground truth

] 1 0.75 z =

Z 0

| 0.50 µ [ Coverage

E 1 − 0.25 Gauss-F-Loc ( ) SN DKW-F-Loc ( ) SN 2 AMARI ( ) − SN 0.00 3 2 1 0 1 2 3 3 2 1 0 1 2 3 − − − z − − − z c) Local false sign rate d) Local false sign rate

1.00 Gauss-F-Loc ( ) 1.00 SN DKW-F-Loc ( ) SN AMARI ( ) ] 0.75 SN z Ground truth 0.75 = Z | 0.50

0 0.50 ≥ Coverage µ [

P 0.25 0.25 Gauss-F-Loc ( ) SN DKW-F-Loc ( ) SN AMARI ( ) SN 0.00 0.00 3 2 1 0 1 2 3 3 2 1 0 1 2 3 − − − z − − − z

Figure 6: Simulation results: Inference in the Gaussian empirical Bayes prob- lem with = a Gaussian scale mixture. a,b) Inference for the posterior mean (same simulationG SN setting as Figure4a,b). c,d) Inference for the local false sign rate (same simulation setting as Figure5a,b).

1.00 Gauss-F-Loc ( ) LN 9 1110 12 DKW-F-Loc ( ) 8 LN AMARI ( ) 0.75 76 LN Logspline

0.50 5 Coverage 0.25 4 0.00 23 0.1 0.2 0.3 0.4 0.5 Expected CI length

Figure 7: Coverage versus expected length of confidence intervals: Here the sim- ulation setting is the same as that of Figure5, panels c,d) but we only consider inference at z = 2, i.e., for θG(2) = PG µ 0 Z = 2 . We apply the exponential family plug-in estimator for a range of degrees of≥ freedom (from 2 to 12 shown by the number as well   as progressively darker blue color), while in Figure5 only the estimator with 5 degrees is considered.

22 outperforms the F -localization approach. Second, we illustrate the form of the Q( ) that solves the worst-case bias-variance problem (26) in a familiar context. Third, we elaborate· on the issue of partial identification.

7.1 Asymptotic relative efficiency in the Poisson model Consider the Poisson model in which we seek to conduct inference for the posterior mean θG(z) = EG [µ Z = z]. In view of Robbins’ formula (9), θG(z) can be estimated at the parametric 1/√| n rate and so we can compare confidence intervals and estimators in terms of their asymptotic relative efficiency. Before studying θG(z), we first discuss inference for the linear functional L(G) = fG(z) = PG [Z = z] for a fixed z. A consistent estimator in this case is given by the sample proportion fˆ(z) = # Zi = z /n, which has the limiting distribution, { }

√n fˆ(z) fG(z) D (0, fG(z)(1 fG(z)), (37) − −→N − and so we can build an asymptotic 1 α confidence interval with asymptotic length equal to − 2q1 α/2 fG(z)(1 fG(z))/√n, where q1 α/2 is the 1 α/2 quantile of the standard normal distribution.− Tierney− and Lambert[1984]− prove that− among a class of regular estimators, p the estimator in (37) is asymptotically efficient and so, asymptotically, the information that FG is a Poisson mixture, is not helpful for inference of L(G). In this setting, it turns out that AMARI21 matches the efficient confidence interval based on (37) and is shorter than the DKW-F -Localization interval.

Proposition 6. We consider inference for L(G) = fG(z), in the Poisson model at level α (0, 1) for a fixed z N 1. We treat the observations as right-censored for Zi > M as∈ in (27), for fixed M ∈ z≥+ 1, and consider the prior class = ([a, b]), 0 a < b < . If G and G is supported≥ on at least M + 2 points, thenG asPn it≤ holds that: AMARI∞ has∈ G asymptotically the same length as the confidence interval constructed→ ∞ using (37), namely 2q1 α/2 fG(z)(1 fG(z))/n(1 + oPG (1)). The DKW-F -Localization interval has asymptotic− length 2 2 log(2−/α)/n(1 + o (1)). p PG The key argumentp in the proof of the above proposition is that, in this setting, the optimal Q( ) solving (26) is with high probability equal to 1( = z) for n large enough, in · · which case, L = # Zi = z /n. In finite samples, however, the optimal Q( ) takes the form { } · of a kernel smoother that upweights Zi in a neighborhood of z. To illustrate, we simulate from the Poissonb empirical Bayes model (1) with G = U[0, 2] and let n vary. We specify = ([0, 4]). The optimal Q( ) of AMARI for different values of n is shown in Figure8a). FigureG P 8b) shows the expected· length of the AMARI and DKW- F -Localization confidence intervals, as well as the asymptotic lengths from Proposition6. As expected, AMARI has shorter length than the DKW-F -localization intervals. The information that FG is a Poisson mixture is not helpful to both approaches for n large, however, both AMARI and DKW-F - Localization can use this information for smaller n to provide sharper inference. The next proposition and Figure8c) pertain to the posterior mean θG(z) = EG [µ Z = z] and are analogous to Proposition6 and Figure8b). Our findings are similar; AMARI| out- performs the DKW-F -localization intervals and for small n, both methods perform better than predicted by the asymptotic limit.

21We slightly abuse terminology in this section, and use the term AMARI also for Algorithm1, i.e., for our inference approach for linear functionals. The pilot estimates for AMARI are chosen as in Proposition3.

23 a) b) c)

1.00 n = 103 AMARI AMARI 5 0.75 n = 10 AMARI asymp. 1 AMARI asymp. 7 1 10 n = 10 10− DKW-F-Loc DKW-F-Loc DKW-F-Loc asymp. DKW-F-Loc asymp. 0.50 100 2 0.25 10−

0.00 1 10− 3 10− 0.25 − Expected CI length Expected CI length 2 10− 0.50 − 10 4 Q(0) Q(1) Q(2) Q(3) Q(4) Q(5) Q(6) Q(7) − 102 103 104 105 106 107 108 102 103 104 105 106 107 108 n n

Figure 8: Relative efficiency in the Poisson empirical Bayes problem. We draw n samples µi U[0, 2],Zi µi Poisson (µi). a) Optimal Q( )(26) of AMARI for esti- ∼ | ∼ · mating L(G) = fG(3), when = ([0, 4]). Each Q( ) corresponds to a single simulation and a different value of n. ForG example,P for n = 107·, Q( ) is the indicator 1( = 3). All Q( ) are constant for z 6, 7,... by construction (26). b)· Confidence interval· length of AMARI· and DKW-F -Localization∈ { } in the same setting as panel a), averaged over 50 Monte Carlo replicates. The panel also shows the asymptotic interval length for the two meth- ods as per Proposition6. c) Analogous to panel b) for inference of the posterior mean θG(3) = EG µ Z = 3 . Asymptotic lengths are derived in Proposition7.  

Proposition 7. We consider inference for θG(z) = EG [µ Z = z] in the setting of Propo- sition6. Then: |

AMARI PG √n α (z) 2(z + 1)q1 α/2 fG(z + 1) (1 + fG(z + 1)/fG(z)) fG(z) , I −−→ − DKW-F-Loc PG √n (z) 2(z + 1) 2 log(2p/α) (1 + fG(z + 1)/fG(z)) fG(z). Iα −−→ · p  In this case, AMARI is asymptotically equivalent to the intervals constructed by Robbins [1980] and Karlis et al.[2018]. The confidence intervals of Robbins[1980] are based on the joint central limit theorem for # Zi = z /n and # Zi = z + 1 /n, the delta method and his formula (9). { } { }

7.2 Sharp partial identification in the Bernoulli model Above we compared methods in a problem in which parametric rates are attainable. Here we compare methods under partial identification, i.e., when confidence intervals will not shrink to a point mass, even as n . We consider model (1) with Zi µi Bernoulli(µi), i.e., the Binomial model with a→ single ∞ (N = 1) trial. Furthermore, we do not∼ impose additional structure on , i.e., we assume that G = ([0, 1]) (4). Zi is supported on 0, 1 and we G ∈ G P z { 1} z can take λ = δ0 + δ1 to be the counting measure on 0, 1 and p(z µ) = µ (1 µ) − . The marginal distribution F is fully determined by f (1).{ } − G G We first consider inference for the second moment L(G) = µ2 dG(µ), which is a linear functional of G. The distribution of Zi, however, does not point identify L(G), unless R 2 22 fG(1) 0, 1 , and the partial identification interval for L(G) is equal to [fG(1) , fG(1)]. ∈ { } 2 The posterior mean, θG(1) = EG [µ Z = 1] = µ dG(µ) fG(1) is also not identified and | its partial identification interval is equal to [fG(1), 1]. The next proposition shows that, as R  22 The right bound is attained by the prior on [0, 1] with PG [{0}] = fG(0), PG [{1}] = fG(1), and the left bound by the point mass prior with PG [{fG(1)}] = 1. Both of these priors induce the same marginal distribution FG.

24 n , both AMARI and F -localization intervals converge to the corresponding partial identification→ ∞ intervals, and so all proposed intervals have the same asymptotic length (which is the best possible). Proposition 8. Consider inference for L(G) = µ2 dG(µ) in the Bernoulli model with the DKW-F -localization (22), the χ2-F -localization (22), or AMARI (using any of the above R as a pilot F -localization). We use = ([0, 1]) and suppose that 0 < fG(1) < 1. The length of all these confidence intervals asymptoticallyG P converges to the length of the partial iden- PG tification interval, i.e., α (fG(1)(1 fG(1))) 1. Similarly, all of the above confidence |I | − → intervals for θG(1) = EG [µ Z = 1] asymptotically match the partial identification intervals, PG| i.e., α(z) (1 fG(1)) 1. |I | − →  8 On the choice of G Throughout this paper, we have taken the choice of for granted and suggested some choices such as (4), (34) and (35) in our numerical examples.G There are two difficulties regarding this choice; first, needs to capture the true G and second, if is infinite-dimensional, then it has to be suitablyG discretized to numerically solve optimizationG problems such as (7) or (28)23. Recent successful applications of empirical Bayes for point estimation use a dis- cretized convex class . For example, Koenker and Mizera[2014] and Koenker and Gu[2017] use the nonparametricG maximum likelihood estimator (NPMLE) for a plethora of different likelihoods in (1) with G ( ) and a finite set (an equidistant discretization of a com- pact interval). The above∈ classes P K areK typically discretized so densely that a nonparametric G approach to forming confidenceb intervals, as pursued in this work, is warranted. In cases where a choice of has been used for estimation, we suggest that the point estimates be accompanied by confidenceG intervals using the same choice of prior class.

8.1 Sensitivity analysis for G The choice of , is not innocuous and the sensitivity of our intervals to the non-parametric specification ofG is an important consideration for their practical adoption. One could hope to choose onG the basis of goodness-of-fit testing. However, goodness-of-fit tests are only able to ruleG out that are inconsistent with the data, but there may be many choices of that are consistentG with the data, and for each of these, depending on the target of inference,G the length of the confidence intervals may vary substantially or remain relatively stable. To illustrate these ideas further, we suppose that G is location mixture of Gaussians as in (34). These classes are nested as (˜τ 2, R) (τ 2, R) forτ ˜ < τ.24 If we define, LN ⊃ LN 2 τ ∗(G) = sup τ > 0 G (τ , R) , | ∈ LN 2 then inference using our methods will be valid with = (τ , R) for any τ τ ∗(G), G LN ≤ and will be more conservative, the smaller τ is. So ideally we would like to use τ = τ ∗(G), however, τ ∗(G) is a one-sided discontinuous statistical functional in the sense of Donoho [1988], so that it is impossible to derive a non-trivial lower bound on it in a data-driven way; see also Donoho and Reeves[2013]. On the other hand, it is possible to derive upper

23We provide guidance for the numerical discretization of infinite-dimensional G in Supplement D.2. 24 2 2 Suppose G ∈ LN (τ , R), i.e., G = N (0, τ ) ?H for τ > 0 and a distribution H, where ? denotes convolution. For any 0 < τ˜ < τ, we can write G = N (0, τ˜2) ? H˜ with H˜ = N (0, τ 2 − τ˜2) ?H, and so 2 G ∈ LN (˜τ , R).

25 25 bounds on τ ∗(G). Hence, a goodness-of-fit test may be able to reject values of τ that are too large, however it cannot disambiguate between choices of small τ. Thus, the only way to obtain practically meaningful results is by the analyst choosing a range of τ’s that appear to be plausible. The analyst can further interpret the results and evaluate how pessimistic a choice of τ (or more generally, of ) may be by inspecting the worst-case priors that determine the confidence interval for a givenG estimand (the worst case priors in (7) for F -Localization, and the worst cases priors in (28) for the affine minimax approach). In SupplementG, we explore these issues in the context of the Prostate data analysis of Section 5.2 using the split-likelihood-ratio of Wasserman et al.[2020] for goodness- of-fit testing. While conducting the suggested sensitivity analysis, it is important to recall that the minimax estimation error for θG(z) decays extremely slowly (often poly-logarithmically) with sample size [Butucea and Comte, 2009, Pensky, 2017] for some of the problems we consider (e.g., local false sign rate in the Gaussian empirical Bayes problem). In this case, unlike in classical estimation problems, we cannot expect to make our confidence intervals meaningfully shorter by, say, collecting 100 times more data than we have now. From this perspective, the amount of assumptions (smoothness, unimodality and so forth) we are willing to impose on determines the accuracy with which we can ever hope to learn θG(z), and the sensitivity analysisG discussed above is closely aligned with recommendations for applications with partially identified parameters [Armstrong and Koles´ar, 2018, Imbens and Wager, 2019, Rosenbaum, 2002].

9 Discussion

We have presented two general approaches towards building confidence intervals for empir- ical Bayes estimands in model (1) that work for any choice of h( ), convex class of priors · and likelihood p( µi). Our methods are computationally intensive and require repeat- Gedly solving non-trivial· convex optimization problems. Nevertheless, in light of an available software implementation, our confidence intervals are practical, and can accompany applied work using nonparametric empirical Bayes point estimates. As in Koenker and Mizera [2014], our implementation is facilitated by recent advances in convex optimization. Here, we focused on inference for empirical Bayes estimands of the form EG [h(µ) Z = z] in model (1); our approach, however, can also handle other empirical Bayes estimands.| As explained in Section3, simpler versions of our methods can be used to form confidence inter- vals for linear functionals of G such as PG [µ 0]. Our methods are also directly applicable to tail (rather than local) empirical Bayes quantities,≥ such as the tail (marginal) false sign rate PG [µ Z 0 Z z] as considered in, e.g., Yu and Hoff[2019]. Another important · ≤ | ≥ perc class of estimands consists of posterior quantiles θG (z; p) = inf t : PG µ t Z = z p for p (0, 1). F -Localization can be used to conduct inference for posterior≤ quantiles≥ by ∈    inverting simultaneous confidence intervals for PG µ t Z = z , t R. However, it seems more challenging to generalize AMARI to posterior quantiles.≤ ∈   A further important direction for future work is to handle generalizations of model (1). In some applications, it would be important to allow for unknown structural parameters or global parameters, such as the variance parameter σ2 in the Gaussian model. (1) could d also be extended to higher-dimensions, e.g., µi R , and to heteroskedastic problems in ∈ which the likelihood pi( µi) can vary across i, for example, the Gaussian location model · 25For example, note that τ ∗(G)2 ≤ Var [Z] − σ2. G

26 2 with per-observation noise standard deviation σi, so that pi( µi) = µi, σi [Gu and Koenker, 2017, Weinstein et al., 2018]. · N

Software We provide reproducible code for all numerical results in the following Github repository: https://github.com/nignatiadis/empirical-bayes-confidence-intervals-paper. A package implementing the method is available at https://github.com/nignatiadis/ Empirikos.jl. The package has been implemented in the Julia programming language [Bezan- son et al., 2017] and depends, among others, on the packages JuMP.jl [Dunning et al., 2017] and Distributions.jl [Besan¸conet al., 2021].

Acknowledgments This paper was first presented on May 24th, 2018 at a workshop in honor of Bradley Efron’s 80th birthday. We are grateful to Timothy Armstrong, Bradley Efron, Jiaying Gu, Guido Imbens, Panagiotis Lolas, Michail Savvas, Paris Syminelakis, Han Wu and seminar par- ticipants at several venues for helpful feedback and discussions. We thank Jiaying Gu for suggesting the Anderson-Rubin construction. Some of the computing for this project was performed on the Sherlock cluster. We would like to thank Stanford University and the Stan- ford Research Computing Center for providing computational resources and support that contributed to these research results. We acknowledge support from a Ric Weiland Graduate Fellowship, a gift from Google, and National Science Foundation grant DMS-1916163.

References

A. Agresti. Categorical Data Analysis. Wiley Series in Probability and Statistics. Wiley, 2013. ISBN 9781118710944. Theodore W Anderson and Herman Rubin. Estimation of the parameters of a single equation in a complete system of stochastic equations. The Annals of Mathematical Statistics, 20 (1):46–63, 1949.

Theodore Wilbur Anderson. Confidence limits for the expected value of an arbitrary bounded random variable with a continuous distribution function. Bulletin of the In- ternational Statistical Institute, 43:249–251, 1969. MOSEK ApS. The MOSEK Optimization Suite Manual, Version 9.2, 2020. URL https: //www.mosek.com/.

Timothy B Armstrong and Michal Koles´ar.Optimal inference in a class of regression models. Econometrica, 86(2):655–683, 2018. Timothy B Armstrong, Michal Koles´ar,and Mikkel Plagborg-Møller. Robust empirical Bayes confidence intervals. arXiv preprint arXiv:2004.03448, 2020.

Vidmantas Bentkus and Friedrich G¨otze. The Berry-Esseen bound for Student’s statistic. The Annals of Probability, 24(1):491–503, 1996.

27 Mathieu Besan¸con, Theodore Papamarkou, David Anthoff, Alex Arslan, Simon Byrne, Dahua Lin, and John Pearson. Distributions.jl: Definition and modeling of probabil- ity distributions in the JuliaStats ecosystem. Journal of Statistical Software, 98(16):1–30, 2021. ISSN 1548-7660. Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A fresh approach to numerical computing. SIAM review, 59(1):65–98, 2017. Fritz Bichsel. Erfahrungs-Tarifierung in der Motorfahrzeughaftpflicht-Versicherung. Mit- teilungen der Vereinigung Schweizerischer Versicherungsmathematiker / Bulletin of the Swiss Association of Actuaries, 64:119–130, 1964.

Peter J Bickel. On adaptive estimation. The Annals of Statistics, pages 647–671, 1982. Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004. Jennifer Brennan, Ramya Korlakai Vinayak, and Kevin Jamieson. Estimating the number and effect sizes of non-null hypotheses. In International Conference on Machine Learning, pages 1123–1133, 2020. Lawrence D Brown and Eitan Greenshtein. Nonparametric empirical Bayes and compound decision approaches to estimation of a high-dimensional vector of normal means. The Annals of Statistics, pages 1685–1704, 2009.

Lawrence D Brown, Eitan Greenshtein, and Ya’acov Ritov. The Poisson compound decision problem revisited. Journal of the American Statistical Association, 108(502):741–749, 2013. Hans B¨uhlmannand Alois Gisler. A course in credibility theory and its applications. Springer Science & Business Media, 2006.

Christina Butucea and Fabienne Comte. Adaptive estimation of linear functionals in the convolution model and applications. Bernoulli, 15(1):69–98, 2009. T Tony Cai and Mark G Low. A note on nonparametric estimation of linear functionals. The Annals of Statistics, 31(4):1140–1153, 2003.

A. Charnes and W. W. Cooper. Programming with linear fractional functionals. Naval Research Logistics Quarterly, 9(3-4):181–186, 1962. Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Gaussian approximation of suprema of empirical processes. The Annals of Statistics, 42(4):1564–1597, 2014. Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Empirical and multiplier boot- straps for suprema of empirical processes of increasing complexity, and related Gaussian couplings. Stochastic Processes and their Applications, 126(12):3632–3651, 2016. Chris Coey, Lea Kapelevich, and Juan Pablo Vielma. Towards practical generic conic opti- mization. arXiv preprint arXiv:2005.01136, 2020.

Clifford B Cordy and David R Thomas. Deconvolution of a distribution function. Journal of the American Statistical Association, 92(440):1459–1465, 1997.

28 Itai Dattner, Alexander Goldenshluger, and Anatoli Juditsky. On deconvolution of distri- bution functions. The Annals of Statistics, pages 2477–2501, 2011. JJ Deely and RL Kruse. Construction of sequences estimating the mixing distribution. The Annals of Mathematical Statistics, 39(1):286–288, 1968.

Luc Devroye. A note on the usefulness of superkernels in density estimation. The Annals of Statistics, pages 2037–2056, 1992. David Donoho and Galen Reeves. Achieving Bayes MMSE performance in the sparse signal+ Gaussian white noise model when the noise level is unknown. In 2013 IEEE International Symposium on Information Theory, pages 101–105. IEEE, 2013.

David L Donoho. One-sided inference about functionals of a density. The Annals of Statis- tics, pages 1390–1420, 1988. David L Donoho. Statistical estimation and optimal recovery. The Annals of Statistics, pages 238–270, 1994.

David L Donoho and Richard C Liu. Geometrizing rates of convergence, III. The Annals of Statistics, 19(2):668–701, 1991. David L Donoho and Richard Chieng Liu. Hardest one-dimensional subproblems. Depart- ment of Statistics, University of California, 1989. Iain Dunning, Joey Huchette, and Miles Lubin. JuMP: A modeling language for mathemat- ical optimization. SIAM Review, 59(2):295–320, 2017. doi: 10.1137/15M1020575. Dean Eckles, Nikolaos Ignatiadis, Stefan Wager, and Han Wu. Noise-induced randomization in regression discontinuity designs. arXiv preprint arXiv:2004.09458, 2020. Bradley Efron. Bootstrap methods: Another look at the Jackknife. The Annals of Statistics, 7(1):1–26, 1979. Bradley Efron. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106(496):1602–1614, 2011. Bradley Efron. Large-scale inference: Empirical Bayes methods for estimation, testing, and prediction. Cambridge University Press, 2012.

Bradley Efron. Two modeling strategies for empirical Bayes estimation. Statistical Science, 29(2):285, 2014. Bradley Efron. Empirical Bayes deconvolution estimates. Biometrika, 103(1):1–20, 2016. Bradley Efron. Bayes, oracle Bayes and empirical Bayes. Statistical Science, 34(2):177–201, 2019. Bradley Efron and Carl Morris. Stein’s estimation rule and its competitors– An empirical Bayes approach. Journal of the American Statistical Association, 68(341):117–130, 1973. Bradley Efron, Robert Tibshirani, John D Storey, and Virginia Tusher. Empirical Bayes analysis of a microarray experiment. Journal of the American Statistical Association, 96 (456):1151–1160, 2001.

29 Bert Van Es and Hae-won Uh. Asymptotic normality of kernel-type deconvolution estima- tors. Scandinavian journal of statistics, 32(3):467–483, 2005. Edgar C Fieller. The biological standardization of insulin. Supplement to the Journal of the Royal Statistical Society, 7(1):1–64, 1940.

Edgar C Fieller. Some problems in interval estimation. Journal of the Royal Statistical Society: Series B (Methodological), 16(2):175–185, 1954. Michael Gilraine, Jiaying Gu, and Robert McMillan. A new method for estimating teacher value-added. Technical report, National Bureau of Economic Research, 2020. Evarist Gin´eand Richard Nickl. Mathematical foundations of infinite-dimensional statistical models, volume 40. Cambridge University Press, 2016. Eitan Greenshtein and Theodor Itskov. Application of non-parametric empirical Bayes to treatment of non-response. Statistica Sinica, 28(4):2189–2208, 2018. Jiaying Gu and Roger Koenker. Unobserved heterogeneity in income dynamics: An empirical Bayes perspective. Journal of Business & Economic Statistics, 35(1):1–16, 2017. Jaroslav Hajek. Asymptotically most powerful rank-order tests. The Annals of Mathematical Statistics, pages 1124–1147, 1962. Philippe Heinrich and Jonas Kahn. Strong identifiability and optimal minimax rates for finite mixture estimation. Annals of Statistics, 46(6A):2844–2870, 2018.

David A Hirshberg and Stefan Wager. Augmented minimax linear estimation. The Annals of Statistics, forthcoming, 2021. Guido Imbens and Stefan Wager. Optimized regression discontinuity designs. Review of Economics and Statistics, 101(2):264–278, 2019.

Guido W Imbens and Charles F Manski. Confidence intervals for partially identified pa- rameters. Econometrica, 72(6):1845–1857, 2004. Wenhua Jiang and Cun-Hui Zhang. General maximum likelihood empirical Bayes estimation of normal means. The Annals of Statistics, 37(4):1647–1684, 2009.

Iain M Johnstone. Gaussian estimation: Sequence and wavelet models. Manuscript, 2011. Iain M Johnstone and Bernard W Silverman. Discretization effects in statistical inverse problems. Journal of complexity, 7(1):1–34, 1991. Nathan Kallus. Generalized optimal matching methods for causal inference. Journal of Machine Learning Research, 21(62):1–54, 2020.

S. Karlin and W.J. Studden. Tchebycheff Systems: With Applications in Analysis and Statistics. Interscience Publishers, New York, 1966. Dimitris Karlis, George Tzougas, and Nicholas Frangos. Confidence intervals of the premi- ums of optimal bonus malus systems. Scandinavian Actuarial Journal, 2018(2):129–144, 2018.

30 Jack Kiefer and Jacob Wolfowitz. Consistency of the maximum likelihood estimator in the presence of infinitely many incidental parameters. The Annals of Mathematical Statistics, pages 887–906, 1956. Arlene KH Kim. Minimax bounds for estimation of normal mixtures. Bernoulli, 20(4): 1802–1818, 2014.

Roger Koenker. Empirical Bayes Confidence Intervals: An R vinaigrette. http://www. econ.uiuc.edu/~roger/research/ebayes/cieb.pdf, 2020. Roger Koenker and Jiaying Gu. REBayes: Empirical Bayes mixture methods in R. Journal of Statistical Software, 82(8):1–26, 2017.

Roger Koenker and Ivan Mizera. Convex optimization, shape constraints, compound de- cisions, and empirical Bayes rules. Journal of the American Statistical Association, 109 (506):674–685, 2014. Nan M Laird and Thomas A Louis. Empirical Bayes confidence intervals based on bootstrap samples. Journal of the American Statistical Association, 82(399):739–750, 1987.

Frederic M Lord. Estimating true-score distributions in psychological testing (an empirical Bayes estimation problem). Psychometrika, 34(3):259–299, 1969. Frederic M Lord and Noel Cressie. An empirical Bayes procedure for finding an interval estimate. Sankhy¯a:The Indian Journal of Statistics, Series B, pages 1–9, 1975.

Frederic M Lord and Martha L Stocking. An interval estimate for making statistical infer- ences about true scores. Psychometrika, 41(1):79–87, 1976. Michael I Love, Wolfgang Huber, and Simon Anders. Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2. Genome biology, 15(12):550, 2014.

Mark G Low. Bias-variance tradeoffs in functional estimation problems. The Annals of Statistics, pages 824–835, 1995. Laurence S Magder and Scott L Zeger. A smooth nonparametric estimate of a mixing distribution using mixtures of Gaussians. Journal of the American Statistical Association, 91(435):1141–1151, 1996.

Pascal Massart. The tight constant in the Dvoretzky–Kiefer–Wolfowitz inequality. The Annals of Probability, pages 1269–1283, 1990. Catherine Matias and Marie-Luce Taupin. Minimax estimation of linear functionals in the convolution model. Mathematical Methods of Statistics, 13(3):282–328, 2004.

A. Meister. Deconvolution Problems in Nonparametric Statistics. Lecture Notes in Statistics. Springer Berlin Heidelberg, 2009. ISBN 9783540875574. Carl N Morris. Parametric empirical Bayes confidence intervals. In Scientific inference, data analysis, and robustness, pages 25–50. Elsevier, 1983. Omkar Muralidharan. High dimensional exponential family estimation via empirical Bayes. Statistica Sinica, pages 1217–1232, 2012.

31 Balasubramanian Narasimhan and Bradley Efron. deconvolveR: A G-modeling program for deconvolution and empirical Bayes estimation. Journal of Statistical Software, 94(1):1–20, 2020. Claudia Noack and Christoph Rothe. Bias-aware inference in fuzzy regression discontinuity designs. arXiv preprint arXiv:1906.04631, 2019.

Victor M Panaretos and Yoav Zemel. Statistical aspects of Wasserstein distances. Annual review of statistics and its application, 6:405–431, 2019. Marianna Pensky. Minimax theory of estimation of linear functionals of the deconvolution density with or without sparsity. The Annals of Statistics, 45(4):1516–1541, 2017.

Iosif Pinelis. Moment matching: construction of a mixture of Gaussian distribution with lower moments identical to Gaussian. MathOverflow, 2017. URL https:// mathoverflow.net/q/229723. Dimitris N Politis and Joseph P Romano. On a family of smoothing kernels of infinite order. Computing science and statistics, Proceedings of the 25th Symposium on the Interface, San Diego, CA, pages 141–145, 1993. Herbert Robbins. An empirical Bayes approach to statistics. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics. The Regents of the University of California, 1956.

Herbert Robbins. An empirical Bayes estimation problem. Proceedings of the National Academy of Sciences, 77(12):6988–6989, 1980. James Robins and Aad Van Der Vaart. Adaptive nonparametric confidence sets. The Annals of Statistics, 34(1):229–253, 2006. R Tyrrell Rockafellar. Convex analysis. Number 28 in Princeton Landmarks in Mathematics and Physics. Princeton university press, 1970. Joseph P Romano and Michael Wolf. Finite sample nonparametric inference and large sample efficiency. Annals of Statistics, pages 756–778, 2000. Paul R Rosenbaum. Observational studies. Springer, 2002.

Leopold Simar. Maximum likelihood estimation of a compound Poisson process. The Annals of Statistics, pages 1200–1209, 1976. Dinesh Singh, Phillip G. Febbo, Kenneth Ross, Donald G. Jackson, Judith Manola, Christine Ladd, Pablo Tamayo, Andrew A. Renshaw, Anthony V. D’Amico, Jerome P. Richie, Eric S. Lander, Massimo Loda, Philip W. Kantoff, Todd R. Golub, and William R. Sellers. Gene expression correlates of clinical prostate cancer behavior. Cancer cell, 1(2): 203–209, 2002. Matthew Stephens. False discovery rates: a new deal. Biostatistics, 18(2):275–294, 2016. Marie-Luce Taupin. Semi-parametric estimation in the nonlinear structural errors-in- variables model. Annals of Statistics, pages 66–93, 2001.

32 Luke Tierney and Diane Lambert. Asymptotic efficiency of estimators of functionals of mixed distributions. The Annals of Statistics, pages 1380–1387, 1984. Aad W Van Der Vaart and Jon A Wellner. Weak Convergence and Empirical Processes: With Applications to Statistics. Springer, 1996.

Larry Wasserman, Aaditya Ramdas, and Sivaraman Balakrishnan. Universal inference. Proceedings of the National Academy of Sciences, 117(29):16880–16890, 2020. Asaf Weinstein, Zhuang Ma, Lawrence D Brown, and Cun-Hui Zhang. Group-linear em- pirical Bayes estimates for a heteroscedastic Normal mean. Journal of the American Statistical Association, pages 1–13, 2018.

Chaoyu Yu and Peter D Hoff. Adaptive sign error control. Journal of Statistical Planning and Inference, 2019. Anqi Zhu, Joseph G Ibrahim, and Michael I Love. Heavy-tailed prior distributions for sequence count data: removing the noise and preserving large differences. Bioinformatics, page bty895, 2018.

33 A Gaussian F -localization: Proof of Proposition1

Throughout this section we assume that model (1) holds with p( µ) = (µ, σ2). Further- more, without loss of generality, we assume that σ2 = 1. The key· ideaN of the proof is the following: we first use the smoothness of fG(z) in the Gaussian empirical Bayes problem to ˆ ˆK verify that f(z) = fn (z)(18) has bias of order O(1/ n log(n)). Thus the dominant error in sup fˆ(z) f (z) is stochastic and equal to sup fˆ(z) [fˆ(z)] . We z [ M,M] G p z [ M,M] EG then study∈ − the variance| − of the| stochastic term (pointwise) and∈ − use results| of− Chernozhukov| et al.[2014, 2016] to verify the accuracy of the bootstrap approximation. Notation: We often omit the dependence on n, for example, we write h for hn = 1/ log(n). All integrals in this section are computed with respect to the Lebesgue measure. p A.1 Bias of KDE

For a function ψ : R R, we write ψ∗(t) for its Fourier transform, i.e., ψ∗(t) = exp(itx)ψ(x)dx, assuming it exists. The→ crucial property of the kernel K( ) in (18) that we will use to control · R bias, is that K∗ is equal to 1 on [ 1, 1] [Politis and Romano, 1993]: − 1, if t [ 1, 1] ∈ − K∗(t) = 0, if t 1.1 . (S38)  | | ≥ 11 10 t , if t [1, 1.1] − · | | | | ∈ We are ready to state our result on the bias.

Proposition S9. Consider estimating the marginal density fG (for some effect size distri- bution G) with the KDE (18). Then, for some constant C it holds that, 2 ˆ 2 ˆ 1 BiasG[f(z), fG(z)] = EG f(z) fG(z) C for all z R. − ≤ n log(n) ∈  h i  Proof.

ˆ 1 Zi z BiasG[f(z), fG(z)] = fG(z) EG K − − h h   

1 u z = fG(z) K − fG(u)du − h h Z  

1 1 z ∗ = exp( itz)fG∗ (t)dt fG∗ (t ) K · − (t)dt 2π − − h h Z Z    1 = exp( itz)f ∗ (t)dt f ∗ (t) exp( itz)K∗(th)dt 2π − G − G − Z Z

1 fG∗ (t)dt ≤ 2π t 1 | | Z{| |≥ h } 1 exp t2/2 ≤ 2π t 1 − Z | |≥ h { }
1 1 h exp ≤ π −2h2   1 1 = π n log(n) p S1 In the 3rd line we used the Fourier inversion formula, as well as the Plancherel isome- try [Meister, 2009, Theorem A.4]. We also used the facts that K is square integrable, K is even, K∗(t) = 1 on [ 1, 1], K∗(t) 1 outside [ 1, 1] and that fG∗ (t) = exp(iµt t2/2)dG(µ) exp( t2/−2). Finally,| we| ≤ used the Gaussian− tail inequality.| | | − | ≤ − R We note that Taupin[2001, Section 3.2] also sketches the above argument.

A.2 Variance of KDE We next study the variance term.

Proposition S10. There exist constants c, C > 0 and n0 N, such that, ∈ c C VarG fˆ(z) for all z [ M,M], n n nh ≤ ≤ nh ∈ − ≥ 0 h i Proof. A consequence of Proposition S9 is that,

Zi z EG K − = h (fG(z) + o(1)) . h    For the second moment, we get:

2 Zi z 2 u z EG K − = K − fG(u)du h h    Z   2 = h K (u)fG(uh + z)du Z 2 = h fG(z) K (u)du + o(1) .  Z  Combining these two results, we find that, 2 1 Zi z fG(z) K (u)du + o(1) 2 VarG K − = (fG(z) + o(1)) . h h h −    R We conclude after noting that the o(1) terms are uniform in z [ M,M] and that ∈ − 0 < inf fG(z) sup fG(z) < . z [ M,M] ≤ z [ M,M] ∞ ∈ − ∈ −

A.3 Validity of bootstrap approximation Let us define the following suprema,

W = √nh sup f(z) fˆ(z) (S39) z [ M,M] − ∈ −

Wc = √nh sup E fˆ(z) f ˆ(z) (S40) z [ M,M] − ∈ − h i

β = √nh sup E fˆ(z) f(z) (S41) z [ M,M] − ∈ − h i

W ∗ = √nh sup fˆ∗(z) fˆ(z) , (S42) z [ M,M] − ∈ −

S2 where fˆ∗(z) is a single bootstrap evaluation of the KDE as in (19). Our high-level strategy is to argue that the distribution of W ∗ conditionally on the data Z = (Z1,...,Zn) is close to the unconditional distribution of W and that W and W are essentially indistinguishable (compared to the fluctuations of the above suprema), because β, i.e. the worst case bias over z, is small (Proposition S9). c We first record the following fact: Define the class of all functions that are dilations and translations of the kernel K( ), i.e., H · = K(a +b) a > 0, b R . (S43) H · ∈  This class has envelope function K( ) , it is pointwise measurable and is of VC type, i.e., there exist constants A e, v k1 such· k∞ that for any finitely discrete probability measures Q and any ε (0, K( )≥ ], it≥ holds that ∈ k · k∞ A v N , 2(Q), ε , (S44) H L ≤ ε  
where N , 2(Q), ε is the ε-covering number of with respect to the 2(Q) norm. This follows directlyH L from Gin´eand Nickl[2016, PropositionH 3.6.12], since theL kernel K( ) is of
bounded variation. · The fact that is VC type will allow us to construct a coupling W and W , where W is H the supremum of a Gaussian process. Concretely, let G be a Gaussian process indexed by [ M,M] with mean 0 and covariance, f f − 1 Z t Z s Cov [G(s), G(t)] = Cov K − ,K − . (S45) h · h h      Then the following holds,

Proposition S11. G is a tight Gaussian process on `∞([ M,M]). Furthermore, there − exists a coupling W,W (with W defined in (S40)) and r1, r2 such that

f W =D sup G(t) z [ M,M] ∈ − and, f 1/6 P W W > r1 r2, r1 = O (nh)− log(n) , r2 = O(1/ log(n)). − ≤ h i  

Similarly, we can constructf a conditional coupling of the bootstrap statistic W ∗ and W ∗, where W ∗ conditionally on Z has the same distribution as (the unconditional law) of W . f Propositionf S12. There exists a coupling W ∗,W ∗ (with W ∗ defined in (S42)), suchf that

(W ∗ Z) =D f sup G(z) z [ M,M] | | ∈ − f and such that there exists an event with P [ c] = O(1/ log(n)) on which E E 1/6 p P W ∗ W ∗ > r∗ Z r∗, r∗ = O (nh)− log(n) , r∗ = O(1/ log(n)). − 1 ≤ 2 1 2 h i   p f

S3 Before proceeding with the proof of Proposition1, we need one final ingredient. We define Levy’s function for W as

f (r) = sup P W t r . (S46) t R − ≤ ∈ h i

The following Proposition holds as a consequence f of Chernozhukov et al.[2014, Lemma A.1]. Proposition S13. The Levy concentration function (S46) satisfies:

r log(log(n)) 0 as n = (r) 0 as n . · → → ∞ ⇒ → → ∞ We postpone the proofp of the above three Propositions to the end of this section and proceed with the main argument.

Proof of Proposition1: Write c∗ = cn(α) √nh for the 1 α conditional quantile of W ∗, i.e., · − c∗ = inf t : P W ∗ t Z 1 α . b ≤ ≥ − It holds that,   

P [F n] = P W c∗ ∈ F ≤ (i) h i P [Wc c∗ β] ≥ ≤ − (ii) P W c∗ β (r1) r2 ≥ ≤ − − − (iii) h i P fW c∗ (β) (r1) r2 ≥ ≤ − − − (iv) h i c 1 fα P [ ] r∗ (r∗) (r1) (β) r2 ≥ − − E − 2 − 1 − − − (v) = 1 α o(1) − − We justify the individual steps (i) (v). − (i) follows by the triangle inequality, since W W + β. (ii) follows from Proposition S11, along with the≤ definition of the Levy function (S46). In more detail: c

P W c∗ β P W c∗ β, W W r1 + P W W > r1 ≤ − ≤ ≤ − − ≤ − h i h i h i P W c∗ β, W W r1,W c ∗ β + P [0 W c∗ + β r1] + r2 f ≤ f ≤ − f − ≤ ≤ f− ≤ − ≤ h i P [W c∗ β] + (r1) + r2. ≤ f≤ − f (iii) follows from the definition of the Levy function (S46). (iv) follows by properties of the Bootstrap approximation. First, note that by definition of c∗, it holds P W ∗ c∗ Z 1 α ≤ ≥ −  

S4 On the other hand, consider the event . On that event, E

P W ∗ c∗ Z P W ∗ c∗ Z (r∗) r∗ ≤ ≥ ≤ − 1 − 2 h i 1  α (r∗ )  r∗, f ≥ − − 1 − 2 where the first inequality follows from Proposition S12 and the definition of the Levy func- tion (S46) and the second inequality follows by definition of c∗. Hence,

c P W ∗ c∗ E P W ∗ c∗ Z ; 1 α (r∗) r∗ P [ ] . ≤ ≥ ≤ E ≥ − − 1 − 2 − E h i h h i i c (v) From Propositionsf S11 and S12f it follows that P [ ] , r2, r∗ = o(1), so it remains to argue E 2 about the Levy terms. From the same propositions, it also holds that r1, r1∗ = O(1/ log(n)) and β = O(1/ log(n)3/4) (Proposition S9). Thus, applying Proposition S13, it also follows p that the terms (r1∗), (r1) and (β) are o(1).

A.4 Proofs of intermediate results Proof of Proposition S11. This follows directly from Chernozhukov et al.[2014, Proposition 1/6 3.1.] by noting that the first term, i.e., O (nh)− log(n) is dominant for the choice h = h = 1/ log(n). Assumptions (B1), (B4), (B5) of Chernozhukov et al.[2014] hold n
trivially, (B2) holds by (S44) and (B3), i.e., that Z has a bounded Lebesgue density on p R follows from the fact that fG(z) 1/√2π for all z, since fG is the convolution of G with a standard Gaussian pdf. Finally, we≤ note that Chernozhukov et al.[2014, Proposition 3.1.] is stated in a slightly different form than Proposition S11, however the result directly follows by inspecting the proof, which is based on Chernozhukov et al.[2014, Corollary 2.2].

Proof of Proposition S12. We seek to apply Chernozhukov et al.[2016, Theorem 2.3]. To this end, we will study the rescaled supremum U ∗ = √h W ∗ and rescaled Gaussian process · B = √h G. Then, note that as in the proof of Proposition S11, Assumptions (A)-(C) of Chernozhukov· et al.[2016, Theorem 2.3] are satisfied. Furthermore, recalling (S44), in the notation of that paper, we can apply the result for η 0, Kn = O(log(n)), q = 4, b bounded, σ2 = O(h) and γ = 1/ log(n). It then follows that→ there exists a coupling

(U ∗ Z) =D sup U(z) , z [ M,M] | | ∈ − such that e P U ∗ U ∗ r˜∗ r˜∗, − ≥ 1 ≤ 2 h i andr ˜∗ = O(1/ log(n)), while 2 e log(n)9/4 log(n)h1/3 log(n)2h1/4 log(n)h1/3 r˜∗ = O + + = O . 1 n1/4 n1/6 n1/4 n1/6     By applying Markov’s inequality, we get that with probability at least 1 1/ log(n), it holds that, − p P U ∗ U ∗ r˜∗ Z r˜∗ log(n). − ≥ 1 ≤ 2 · h i p e

S5 This is the event in the statement of Proposition S12 and we can take r∗ =r ˜∗ log(n) = E 2 2 · √ √ 1/ log(n). Recalling that W ∗ = U ∗/ h and defining W ∗ = U ∗/ h, we get, p p r˜1∗ log(n)f e r∗ = = O . 1 √ (nh)1/6 h  

Proof of Proposition S13. First note that by Proposition S10, we have that there exist σ, σ,¯ n0 such that the following holds for the Gaussian process G of Proposition S11 (with ¯ covariance (S45)):

2 2 σ Var [G(z)] σ¯ , for all z [ M,M], n n0. ¯ ≤ ≤ ∈ − ≥ Hence, by Chernozhukov et al.[2014, Lemma A.1.], we have for a constant C (that depends on σ, σ¯) that: ¯ (r) C r E W + max 1, log(σ/r) . (S47) ≤ · · { ¯ } n h i p o We next bound, E[W ] = E[supz [ M,M] G(z) ]. To this end, we make the following obser- ∈ − f| | vations: first, G is a Gaussian process, and so in particular it is a sub-Gaussian process 2 2 with respect to d (fz, z0) = E[(G(z) G(z0)) ]. Applying Proposition S10 again, we find − that supz,z0 [ M,M] d(z, z0) D < is finite (and D can be chosen the same for all n). In ∈ − ≤ ∞ addition, by (S44), we find that for some A0, v0 e, ≥ 0 A v N([ M,M], d, ε) 0 . − ≤ ε √h  ·  and so, D log (N([ M,M], d, ε))d = O log(1/h) . 0 − Z p p  By Dudley’s Entropy integral [Van Der Vaart and Wellner, 1996, Corollary 2.2.8], we thus also get that, E W = O log(1/h) . h i p  Combing the above with (S47), we findf that (r) 0 for r = o(1/ log(log(n))). → p B Proofs for AMARI inference

Notation: Throughout this supplement we omit the M superscript, e.g., we write fG M M instead of fG , λ instead of λ and so forth.

B.1 Properties of the modulus of continuity ¯ Proposition S14. Assume n is convex, infz f(z) > 0 and supG n L(G) < . Then, G ∈G | | ∞ the modulus ωn( ) defined in (28), as a function of δ > 0, has the following properties: · (a) It is non-decreasing. (b) It is bounded and nonnegative.

S6 (c) It is concave.

(d) For δ > 0, there exists an element ω0 (δ) 0 in the superdifferential of ωn( ) at δ, i.e., n ≥ · ω0 (δ) satisfies the property defined in Footnote 12. It holds that ωn(δ) δ ω0 (δ). n ≥ · n

Proof. (a) and (b) follow directly by the definition of ωn( ). For (c), let us take δa, δb > 0, δa δa δb δb · λ (0, 1) and let (G1 ,G 1), (G1 ,G 1) solve the corresponding modulus problems. If solutions∈ for either of these− do not exist,− we may take an approximate minimizer and use standard approximation arguments. Now for, λ (0, 1) and δ(λ) = λδa +(1 λ)δb, consider δ(λ) δa δb ∈ δ(λ) − Gi = λGi + (1 λ)Gi with i 1, 1 . Then Gi n by convexity of n and furthermore by the− triangle inequality,∈ {− } ∈ G G

2 1/2 ¯ n fGδ(λ) (z) fGδ(λ) (z) /f(z)dλ(z) λδa + (1 λ)δb = δ(λ). · 1 − −1 ≤ −  Z    Hence: δ(λ) δ(λ) ωn(δ(λ)) L(G1 ) L(G 1 ) = λωn(δa) + (1 λ)ωn(δb). ≥ − − − To check (d), we note that the existence of ωn0 (δ) follows from (b,c) and results from con- vex analysis [Rockafellar, 1970]. ωn0 (δ) satisfies the property defined in Footnote 12, or equivalently, ωn(δ) ωn(δ˜) ω0 (δ)(δ δ˜) for all δ˜ > 0. (S48) − ≥ n − Suppose ω0 (δ) < 0, then e.g., letting δ˜ = 2δ in (S48), it would follow that ωn(δ) ωn(2δ) > 0, n − which would be a contradiction to part (a). Thus ω0 (δ) 0. Finally, by nonnegativity of n ≥ ωn( ), it follows that ωn(δ) ωn(δ) ωn(δ˜), and taking δ˜ 0 in (S48), we deduce that · ≥ − → ωn(δ) δ ω0 (δ). ≥ · n B.2 Stein’s heuristic In this Section we provide more details regarding optimization problem (26) and the modulus of continuity problem (28) and provide rigorous arguments for the ideas sketched at the beginning of Section 3.1 As already mentioned, at first sight, it is not obvious how to solve optimization prob- lem (26), since the problem is not concave in G, hence standard min-max results for convex- concave problems are not applicable. Nevertheless, Donoho[1994] provides a solution to this optimization problem by formalizing a powerful heuristic that goes back to Charles Stein. The key steps are as follows:

1. We search for the hardest 1-dimensional subfamily, i.e., we find G1,G 1 n, such − ∈ G that solving problem (26) over ConvexHull(G1,G 1) (instead of over all of n) is as hard as possible. The precise definition of “hardest”− is given by the modulusG problem (28). 2. We find the minimax optimal estimator of problem (26) over the hardest 1-dimensional subfamily.

3. We then find that this solution is in fact optimal over all of n. G To implement Step 1 of the heuristic, we solve the modulus problem (28) at δ (and assume δ δ it is solvable) and let G1, G 1 be solutions and ωn0 (δ) an element of the superdifferential. − δ δ Then Q defined in (30) solves the minimax problem (26) over ConvexHull(G1,G 1) for −

S7 2 Γn = Γn(δ) = ωn0 (δ) (Step 2). In fact, it solves this minimax problem over all of n (Step 3), as can be verified by the proposition below, and so (26) can be computed by solvingG the modulus problem (28) (we postpone compoutational details to SupplementE).

Proposition S15 (Properties of Q in (30)). Assume n is convex, sup L(G) < Ge n 2 G ∈G δ| δ | ∞ and that fG( ) (λ) for all G n. Furthermore, assume that there exist G1,G 1 n e · ∈ L ∈ G − ∈ G that solve the modulus problem at δ > 0, i.e., are such that (29) holds and that infz f¯e(z) > 0. Then: e

(a) Q defined by (30), achieves its worst case positive bias over n for estimating L(G) at δ δ G G 1 and negative bias at G1, i.e., letting BiasG[Q, L] = Q(z)fG(z)dλ(z) L(G), it holds− that, − R sup BiasG[Q, L] = Bias δ [Q, L] G−1 G n ∈G (S49) = Bias δ [Q, L] = inf BiasG[Q, L]. G1 − − G n ∈G

2 ¯ ¯ 2 (b) For Q : R, write Varf¯[Q] = Q (z)f(z)dλ(z) Q(z)f(z)dλ(z) . Let Γn = Z → − Varf¯[Q]/n, then for any other functionR Q with Varf¯[Q]R Γn n, it holds
that: d ≤ · 2 2 d sup BiasG[Q, L]e supdBiaseG[Q, L] . (S50) G n ≥ G n ∈G ∈G e (c) Γn and the worst case bias have explicit expressions in terms of the modulus ωn(δ) and its superdifferential ωn0 (δ): 1 sup BiasG[Q, L] = [ωn(δ) δωn0 (δ)] , (S51) G n 2 − ∈G 2 Γn = ωn0 (δ) . (S52)

Proof. The arguments in this proof are well-known and appear in different forms for example in [Donoho, 1994, Low, 1995, Armstrong and Koles´ar, 2018]. However, the statements there are provided in the context of Gaussian mean estimation and therefore we give a simplified, self-contained exposition. δ δ (a) Below for notational convenience we will write G1,G 1 for G1 and G 1. First let − − us check what the bias is at G1:

BiasG [Q, L] = Q(z)fG (z)dλ(z) L(G ) 1 1 − 1 Z 1 = (L(G1) L(G 1)) −2 − −

nωn0 (δ) fG1 (z) fG−1 (z) fG1 (z) fG−1 (z) fG0 (z) + − fG ( )dλ(z) − dλ(z) δ f¯(z) 1 · − f¯(z) (Z Z
) 2 1 nωn0 (δ) (fG1 (z) fG−1 (z)) = ωn(δ) + − dλ(z) −2 2δ f¯(z) Z 1 ωn0 (δ) 2 = ωn(δ) + δ −2 2δ 1 = [ωn(δ) δω0 (δ)] −2 − n

S8 1 Similarly, we get that: BiasG [Q, L] = [ωn(δ) δω0 (δ)]. Let us now show that the worst −1 2 − n case positive bias over n is indeed obtained at G 1. To this end take any other G n, G − ∈ G G = G1,G 1 and define for λ [0, 1]: 6 − ∈ 2 1/2 fG (z) ((1 λ)fG (z) + λfG(z)) ∆(λ) = n 1 − − −1 dλ(z) · f¯(z) Z   ! J(λ) = L(G1) L((1 λ)G 1 + λG) ωn0 (δ)∆(λ) − − − − Observe that for any λ 0: ≥

J(λ) ωn(∆(λ)) ω0 (δ)∆(λ) ≤ − n (i) ˜ ˜ sup ωn(δ) ωn0 (δ)δ ≤ δ˜ 0 − ≥ n o (ii) ωn(δ) ω0 (δ)δ ≤ − n = J(0)

(i) follows by definition of the modulus ωn and (ii) by noting that δ˜ ωn(δ˜) ω0 (δ)δ˜ is 7→ − n concave and its superdifferential at δ includes the element ωn0 (δ) ωn0 (δ) = 0 (also compare to (S48)). The last equality holds by definition of J(λ). − Continuing, by the chain rule and dominated convergence, it holds that J(λ) is differentiable at 0 and so J 0(0) 0. Furthermore, ≤

n ωn0 (δ) fG1 (z) fG−1 (z) J 0(0) = L(G 1) L(G) + · − (fG(z) fG−1 (z)) dλ(z) − − ∆(0) f¯(z) − Z   And now also note that ∆(0) = δ and:

BiasG[Q, L] BiasG−1 [Q, L] = Q(z)fG(z)dλ(z) L(G) Q(z)fG−1 (z)dλ(z) L(G 1) − − − − − Z  Z  n ωn0 (δ) fG1 (z) fG−1 (z) = L(G 1) L(G) + · − (fG(z) fG−1 (z)) dλ(z) − − δ f¯(z) − Z   = J 0(0) 0. ≤ By the above we conclude that:

BiasG[Q, L] BiasG [Q, L]. ≤ −1 Finally, by repeating the same argument

BiasG[Q, L] BiasG [Q, L]. ≥ 1 ¯ (b) First let us write Γn in terms of ωn0 (δ). Note that Q(z)f(z)λ(z) = 0, since R fG1 (z) fG−1 (z) − f¯(z)dλ(z) = fG (z) fG (z) dλ(z) = 1 1 = 0, f¯(z) 1 − −1 − Z Z

S9 and so, 2 2 2 n ωn0 (δ) fG1 (z) fG−1 (z) Var ¯[Q] = · − f¯(z)dλ(z) f δ2 · f¯(z) Z   2 2 2 d n ω0 (δ) fG1 (z) fG−1 (z) = · n − dλ(z) δ2 · f¯(z) Z
2 2 2 n ω0 (δ) δ = · n δ2 · n 2 = n ω0 (δ) . · n 2 Thus, Γn = ωn0 (δ) . Now take any other function Q( ) and decompose it as, Q( ) = Q + Q ( ), where · · 0 1 · Q0 = Q(z)f¯(z)dλ(z), so that, e e e e R 2 2 e e Var ¯[Q] = Q (z) f¯(z)dλ(z) Γn n = n ω0 (δ) . f 1 ≤ · · n Z Then: d e e

BiasG [Q, L] BiasG [Q, L] −1 − 1

= Q1(ze)fG−1 (z)dλ(z) e L(G 1) Q1(z)fG1 (z)dλ(z) L(G1) − − − − Z  Z  e e = L(G1) L(G 1) + Q1(z) fG−1 (z) fG1 (z) dλ(z) − − − Z
1/e2 fG−1 (z) fG1 (z) = ωn(δ) + Q (z)f¯(z) − dλ(z) 1 f¯(z)1/2 Z
/ e 1/2 2 1 2 2 fG1 (z) fG−1 (z) ωn(δ) Q (z) f¯(z)dλ(z) − f¯(z)dλ(z) ≥ − 1 f¯(z) Z  Z   ! e 2 1/2 2 1/2 ωn(δ) n ω0 (δ) δ /n ≥ − · n = ωn(δ) ω0 (δ)δ − n

= 2 sup BiasG[Q, L] . G n {| |} ∈G

Above we used properties of ωn( ) shown in Proposition S14. Next, ·

sup BiasG[Q, L] max BiasG−1 [Q, L] , BiasG1 [Q, L] G n ≥ ∈G n o n o e e e Bias G−1 [Q, L] + Bias G1 [Q, L] 2 ≥ sup Bias [Q, L] .   eG e ≥ G n {| |} ∈G (c) We already proved these statements as intermediate steps while proving (a) and (b).

B.3 Proof of Theorem2

Proof. A word on notation: We drop the dependence on n, M and δn, whenever this does δn not cause confusion, for example we write G1 instead of G1 and so forth. Furthermore, we

S10 ¯ write E [ ] for conditional expectations with respect to f, n, and P [ ],Var [ ] for conditional probabilities,· resp. variances. F · · Beforee embarking on the formal argument, we briefly sketche ourg proof strategy. Our proof makes heavy use of the representation of Q( ) in (30). As a consequence of (30), it n · suffices to verify a central limit theorem for i=1 Q(Zi), where,

fG1 P( ) fG−1 ( ) Q( ) = · − e · . (S53) · f¯( ) · In other words, we drop the additivee and multiplicative constants in front of Q( ) that appear · in the expression of Q( ) in (30). A central limit theorem (CLT) for Q directly implies a · CLT for Q and thus also for L. To prove the CLT for the sum of the Q(Zie), we note that Q(Z1),..., Q(Zn) are i.i.d. conditionally on f,¯ n, and so it suffices toe verify Lindeberg’s condition conditionally. b F e e All oure calculations of conditional expectations happen on the event An, which has asymptotic probability equal to 1. By definition of the event An in the statement of the theorem, there furthermore exists deterministic n such that for all n n , we also have 0 ≥ 0 that 4cn η and that infz f¯(z) > η/2, fG(z)/f¯(z) [1/2, 2] An. We assume n n0 henceforth.≤ ∈ ⊂ ≥  We start by studying the (conditional) moments of Q(Zi). For the first moment, we want to argue that its square is negligible compared to the second moment. Our argument crucially depends on the following cancellation: e

fG1 (z) fG−1 (z) − f¯(z)dλ(z) = fG (z)dλ(z) fG (z)dλ(z) = 1 1 = 0. f¯(z) 1 − −1 − Z Z Z Using this cancellation, we get:

fG1 (z) fG−1 (z) EG Q(Zi) = − fG(z)dλ(z) f¯(z) Z h i e e fG (z) fG (z) 1 −1 ¯ = ¯− fG(z) f(z) dλ(z) f(z) − Z
¯
fG1 (z) fG−1 (z) fG(z) f(z) = ¯− − fG(z) dλ(z) f(z) fG(z) Z

¯ fG1 (z) fG−1 (z) fG(z) f(z) ¯− − fG(z) dλ(z) ≤ f(z) fG(z) Z cn fG1 (z) fG−1 (z) ¯− fG(z)dλ(z). ≤ η f(z) Z We next turn to lower bound the second moment. Observe that almost surely, by Jensen’s inequality: 2 2 2 fG1 (z) fG−1 (z) EG Q(Zi) EG Q(Zi) = ¯− fG(z)dλ(z) . ≥ f(z) ! h i h i Z

These twoe displayse togethere imply e that: 2 2 cn 2 1 2 EG Q(Zi) EG Q(Zi) EG Q(Zi) . ≤ η2 ≤ 16 h i h i h i e e e e e e S11 Next,

2 2 2 2 fG1 (z) fG−1 (z) fG(z) 1 fG1 (z) fG−1 (z) δn EG Q(Zi) = − dλ(z) − dλ(z) = . f¯(z) f¯(z) ≥ 2 f¯(z) 2n h i Z
Z
Hence,e e

2 2 2 2 15 2 15 δn δn VarG Q(Zi) = EG Q(Zi) EG Q(Zi) EG Q(Zi) . − ≥ 16 ≥ 32 n ≥ 4n h i h i h i h i Furthermore,g e e e e e e e Q( ) EG Q(Zi) 2 Q( ) · − ≤ · h i ∞ ∞ 2 fG (z) fG (z) e e e e 1 − −1 ¯ ∞ ≤ inf f(z) 4c n . ≤ η So:

3 EG Q(Zi) EG Q(Zi) Q( ) G Q(Zi) − E 8 cn 8cn   · − . (S54) 3h/2 i h 1/2 i ∞ 1/2 1/2 ` e e e e 1/2 ≤ 1 /2 ≤ η n (δn/n ) ≤ ηδ Var G Q(Zi) n Vare G Qe(Zi)e n h i h i As arguedg ine the beginning of the proof,g thise also implies the same bound for Q( ), · 3 EG Q(Zi) EG [Q(Zi)] − 8cn   . (S55) Var [Q(Z )]3/2 n1/2 ≤ ηδ` e G ie We could in principle conclude nowg by applying the Lyapunov/Lindeberg CLT conditionally on n, f¯ along with Slutsky. To explain why the coverage of our intervals is uniform (under theF conditions stated in the footnote of the theorem), we instead apply the Berry-Esseen bound (conditionally on n, f¯) for Student’s statistic [Bentkus and G¨otze, 1996, Theorem F 1.1.]. Recall the definition of L in (12), and define

n 1 b 1/2 1/2 T = (Q(Zi) EG [Q(Zi)])/V = L L(G) BiasG[Q, L] /V , n − − − i=1 X   e b b b and let Φ be the standard Normal CDF. Then, there exists a constant C > 0, such that on S26 the event An and for n sufficiently large,

C cn sup PG [T t] Φ(t) min 1, ` . t ≤ − ≤ η δ ∈R  

S26Note that Bentkus and G¨otze e[1996], do not apply the ( n − 1) correction to the sample variance, in contrast to the definition of Vb in (13). The additional error introduced due to this discrepancy is negligible and may be absorbed into C.

S12 It follows (unconditionally) that,

sup PG [T t] Φ(t) = sup EG PG [T t] Φ(t) t R | ≤ − | t R ≤ − ∈ ∈ h i c sup EG 1e(An) PG [T t ] Φ(t) + PG [An] ≤ t R · ≤ − ∈ h  i C cn c e + PG [An] ≤ η δ`

c The first part of the Theorem follows, since cn 0 and PG [An] 0 as n , and so, → → → ∞

Lˆ L(G) BiasG[Q, L] / V D (0, 1). − − −→N   p From Proposition S15, we know that sup Bias b[Q, L] = B, and so, on the event G˜ n G˜ ∈G | | G n An it also holds that BiasG[Q, L] B, i.e., { ∈ G } ⊂ | | ≤ b

PG BiasG[Q, L] B PGb [An] = 1 o(1). | | ≤ ≥ − h i 1/2 1/2 It remains to prove coverage. Let t˜α = tα(B/b V , 1) and ˜b = BiasG[Q, L]/V , then,

ˆ PG [L(G) α] = PG L L(G) b tbα(B, V ) b ∈ I | − | ≤ h ˆ i 1/2 ˜ = PG L L(G) BiasG[Q, L] /V + b t˜ − − b b ≤ h  ˜ ˆ  1/2i ˜ = PG t˜ b L L(G) BiasG[Q, L] / V t˜ b − − ≤ − − b ≤ − h ˜  ˜  i = EG Φ(t˜ b) Φ( t˜ b) o(1) − − − − − b h ˜ ˜i ˜ 1/2 EG Φ(t˜ b) Φ( t˜ b) 1 b B/V o(1) ≥ − − − − · | | ≤ − (i) h   i (1 α)PG BiasG[Q, L] B o(1) b b ≥ − | | ≤ − (ii) h i = 1 α o(1) b − −

In (i) we used the definition of tα( , ) from (15) and in (ii) we used the bound on the bias we derived above. · ·

B.4 Proof of Proposition3

1/4 Proof. Throughout the proof we take cn = k− log(k) with k = kn. There are three things we need to check (for each case).

(i) First we check the quality of the pilot f¯. The key to our argument here is that

√ sup Fn(t) FG(t) = OP(1/ k), t − ∈R see e.g., below (16) for the justification b (and note that here we use k instead of n samples).

S13 Hence,

tn := sup F (t) FG(t) Gbn t − ∈R

sup F (t) Fn(t) + sup Fn(t) FG(t) Gbn ≤ t − t − ∈R ∈R ( ) ∗ b b 2 sup Fn(t) FG(t ) ≤ t − ∈R √ = OP(1/ bk).

We note that ( ) follows by the definition of Gn as the minimum distance estimator (24). ∗ In the remainder of the proof we seek to bound sup fG(z) f¯(z) in terms of tn. z − For the discrete examples from part (b), web only use the fact that λ is the counting measure on (a subset of) N 0, so that e.g, fG(0) = F G(0) and fG( z) = FG(z) FG(z 1) for z > 0. Hence, recalling that≥ f¯(z) = f (z), and by the triangle inequality,− we conclude− Gbn that,

sup fG(z) f¯(z) 2 sup F (t) FG(t) = 2tn = O (1/√k). Gbn P z − ≤ t − ∈R

Let us now turn to the Gaussian example from part (a), say with σ = 1 (without loss of generality). We handle the density at z = /, resp.z = . as in the discrete examples and focus now on the Lebesgue density at z [ M,M]. We record the following fact. We have that, ∈ − fG(z) = EG [ϕ(z µ)] , − where ϕ is the standard Gaussian pdf. Hence,

fG0 (z) = EG [ϕ0(z µ)] . (S56) −

Consequently fG0 (z) is bounded, uniformly over all priors G and all z. Thus, by Taylor’s theorem, there exists a constant C > 0 such that for all G, z and h > 0:

1 fG(z) (FG(z + h) FG(z)) C h. − h − ≤ ·

Arguing by the triangle inequality, we have that

2tn sup fG(z) f¯(z) + 2Ch, z − ≤ h

and by choosing h = √tn, we conclude that:

¯ 1/4 sup fG(z) f(z) OP(√tn) = OP(k− ). z − ≤

(ii) Second we check that the localizations indeed include FG, i.e., that P [FG n] 1. Here we use the fact, that all F -localizations considered in this proposition are nested∈ F → in α, i.e., n(α) n(α0) for α0 α. F ⊂ F ≤ Fix ε > 0. Let n0 be such that αn < ε/2 for all n n0 (recall that αn 0) and n1 be such that, ≥ → PG [FG n(ε/2)] 1 ε/2 ε/2 = 1 ε for all n n1. ∈ F ≥ − − − ≥

S14 Such n1 exists since all F -localizations are asymptotically valid (at a fixed confidence level) in the sense of (6). Thus, using the nestedness property, for n max n0, n1 , we have that, ≥ { } PG [FG n(αn)] PG [FG n(ε/2)] 1 ε. ∈ F ≥ ∈ F ≥ − Since ε > 0 was arbitrary, we conclude.

(iii) It remains to check that fG1 ( ) fG−1 ( ) cn with probability tending to 1, where · − · ∞ ≤ G1,G 1 are the solutions to the modulus problem. Note that by definition, FG1 ,FG−1 − ∈ n(αn). We consider each F -localization separately. F DKW-F -localization: By (16) (with the sample size n replaced by k),

sup FG1 (t) FG−1 (t) sup FG1 (t) Fn(t) + sup FG−1 (t) Fn(t) t − ≤ t − t −

2 log (2/αn) b(2k) . b ≤ q Then, we can argue as in part (i) of this proof that this implies bounds on fG1 ( ) fG−1 ( ) in all cases (note that in the Gaussian case we use the bounded second derivative· − argument· ∞ for z [ M,M] and handle /, . as discrete). ∈ − 2 χ -F -localization: Consider the event Cn = infz f¯(z) > η/2 , where η > 0 is such that infz fG(z) > η. Cn has probability tending to 1 as n (as follows from the proof of (i)  → ∞ above). Note that, on the event Cn, for any z0,

2 u 2 2 2 fG−1 (z) fG−1 (z) 2 (δ ) fG (z0) fG (z0) − . −1 − −1 ≤ η f¯(z) ≤ η · n i

X Thus, fG ( ) fG ( ) = O (1/√n) = o (1/√k). 1 · − −1 · P P ∞ ¯ Gauss-F -localization: Again all our calculations assume that the event infz f(z) > η/2 has occurred. Let us first treat / and . separately. Arguing as in the case for the χ2-F -  localization, we find that, u 2 2 2 (δ ) fG (/) fG (/) , −1 − −1 ≤ η · n and similarly for .. Next, forz [ M,M], we use
the smoothness of the convolved densities. Namely, suppose, ∈ − fG (z) fG (z) =: ε > 0. 1 − −1 Then, fG1 (z) fG−1 (z) > ε/2 in an interval of length c ε, for a small constant c > 0, as follows by the− boundedness of the derivative (S56 ). This means· that,

u 2 2 (δ ) fG1 (z) fG−1 (z) Leb 3 −¯ dλ (z) c0ε , n ≥ [ M,M] f(z) ≥ Z −
for another constant c0. Note that we also used the fact that f¯(z) is uniformly bounded. By rearranging, we find that,

1/3 1/3 fG (z) fG (z) = O (n− ) = o (k− ). 1 − −1 P P

S15 B.5 Proof of Theorem5 Proof. This proof is a continuation of the proof of Theorem2, and so we also use the notation used therein. In particular, all calculations take place on the event An (which has probability tending to 1 as n . Furthermore, we write Cov [ ] for the covariance → ∞ · ` u conditionally on n, f¯. Let c∗ = θG(z) and κ∗ be such that c∗ = κ∗c + (1 κ∗)c and c∗ ` F u − Q = κ∗Q + (1 κ∗)Q . Note that κ∗ [0, 1], since on the eventgAn, FG n, and so ` u − ∈ ∈ F c c∗ c . ≤We first≤ note that our algorithm estimates the bias conservatively. To see, this, note that: c∗ c∗ BiasG[Q ,L] = Q (z)fG(z)dλ(z) L(G) − Z ` u = κ∗Q (z) + (1 κ∗)Q (z) fG(z)dλ(z) L(G) − − Z ` u = κ∗ BiasG[Q ,L] + (1 κ∗) Bias
G[Q ,L]. − Thus,

c∗ ` u sup BiasG[Q ,L] κ∗ sup BiasG[Q ,L] + (1 κ∗) sup BiasG[Q ,L] , G ≤ G − G | |

and the RHS is precisely our bound for the worst-case bias. We next seek to prove that the Lyapunov/Lindeberg bound (S55) that holds for Q`,Qu c∗ on the event An also applies to Q (with a larger constant) on a smaller event, that however also has asymptotic probability equal to 1 (just as An does). The result will follow, using the Berry-Esseen bound of Bentkus and G¨otze[1996] (as in the Proof of Theorem2) and the argument in the proof of Corollary4. All our conditional calculations occur on the event An of Theorem2, and on the event,

` u ` 1/2 u 1/2 A˜n = CovG Q (Zi),Q (Zi) ( 1 + ε/2) VarG Q (Zi) VarG [Q (Zi)] . ≥ − · n     o We will show below that G[A˜n] 1 and so also G[A˜n An] 1 as n . For now we g P Pg ∗ g seek to provide a Lyapunov/Lindeberg→ bound (S55) for Q∩c . To→ this end,→ first ∞ note that on A˜n An, ∩ c∗ VarG Q (Zi)

h ` i u = VarG κ∗Q (Zi) + (1 κ∗)Q (Zi) g − 2 ` 2 u ` u = (κ∗) VarG Q (Zi) + (1 κ∗) Var G [Q (Zi)] + 2κ∗(1 κ∗)CovG Q (Zi),Q (Zi) g − − 2 ` 2 u ` 1/2 u 1/2 (κ∗) VarG Q (Zi) + (1 κ∗) VarG [Q (Zi)] 2κ∗(1 κ∗)(1 ε/2) VarG Q (Zi) VarG [Q (Zi)] ≥ g − g − − g− · 2 ` 2 u (κ∗) VarG Q (Zi) + (1 κ∗) VarG [Q (Zi)] ε/2.   ≥ g − g · g g   In the last step we used the inequality 2ab a2 + b2. On the other hand, g g ≤ 3 c∗ c∗ EG Q (Zi) EG Q (Zi) −   h i 3 e ` e ` u u = EG κ∗ Q (Zi) EG Q (Zi ) + (1 κ∗) Q (Zi) EG [Q (Zi)] − − −      3   3 e 3 ` e ` 3 e ` ` 8 (κ∗ ) EG Q (Zi) EG Q (Zi) + 8 (1 κ∗) EG Q (Zi) EG Q (Zi) . ≤ − − −         e e e e S16 Thus we now combine the two aforementioned inequalities, 3 c∗ c∗ EG Q (Zi) EG Q (Zi) −   Var [Qc∗ (Z )]3/2 n1/2  e G ei 3 3 3 ` ` 3 ` ` 8 (κ∗) EG Q (Zi) EG Q (Zi) + 8 (1 κ∗) EG Q (Zi) EG Q (Zi) g − − −       3/2   ≤ 2 ` 2 u 3/2 1/2 e (κ ) VarG [Qe (Zi)] + (1 κ ) VarG [Q (Zei)] (ε/2) e n ∗ − ∗ ·  3  3 ` ` u u EG Q (gZi) EG Q (Zi) gEG Q (Zi) EG [Q (Zi)] C − −    +   3/2 ` 3/2 1/2  u 3/2 1/2 ≤ ε  Var [Q (Z )] n Var G [Q (Zi)] n   e G ei e e 

cn C0  . g g  ≤ ε3/2ηδ` 

Here C,C0 > 0 are some constants. The last step follows by applying (S55) that holds for ` u Q ,Q by the proof of Theorem2. It remains to prove that A˜n has asymptotic probability tending to 1. To this end, letε ˜ > 0, and also let, n 1 ` ` u u Wn = Q (Zi) EG Q (Zi) Q (Zi) EG [Q (Zi)] . n − − i=1 X      Then: e e ` u ` 1/2 u 1/2 PG Wn CovG Q (Zi),Q (Zi) ε˜VarG Q (Zi) VarG [Q (Zi)] − ≥ h 2 2 i `  ` u  u  2 ` u EeG Q (Zgi) EG Q (Zi) Q (Zi)gEG [Q (Zi)] g nε˜ VarG Q (Zi) VarG [Q (Zi)] ≤ − −      .     4 1/2 4 1/2   e ` e ` eu u g g EG Q (Zi) EG Q (Zi) EG Q (Zi) EG [Q (Zi)] − −       2  `  u  ≤ e e nε˜ VarG [Q (Zi)]eVarG [Q (Zi)] e 2 Ccn 2 , g g ≤ (˜εηδ`) for another constant C > 0. The argument for the last line is analogous to the argument that led up to (S55). This also means that unconditionally,

` u ` 1/2 u 1/2 PG Wn CovG Q (Zi),Q (Zi) ε˜VarG Q (Zi) VarG [Q (Zi)] − ≥ h2 i Ccn c    + gPG [An] 0 as n . g g ≤ (˜εηδ`)2 → → ∞

` ` u u By a similar argument we can prove that nV /VarG Q (Zi) = 1+oP(1), nV /VarG [Q (Zi)] = 1 + o (1) and that P   n b g b g 1 ` ` ` 1/2 Q (Zi) EG Q (Zi) VarG Q (Zi) = o (1), n − P i=1 X    .   and similarly for Qu. Combininge the above results,g with the assumption of the Theorem, it ˜ follows that PG[An An] 1. ∩ →

S17 C Proofs for Section7 on asymptotic power

A word on notation: We drop the dependence on n and M, whenever this does not cause M confusion. For example we may write FG instead of FG . For the results for AMARI, we follow Proposition3 and assume that the pilot quantities f¯ and n are constructed based F on k samples with k , k/n 0 and k αn as n . To keep the notation lighter → ∞ → · → ∞ → ∞ we suppose that we compute f¯ and n based on k fresh samples from model (1) that are F independent from Z1,...,Zn. The asymptotic confidence interval lengths remain the same as under the sample-splitting of Proposition3, because ( n k)/n 1 as n . − → → ∞ C.1 Poisson model (Section 7.1) C.1.1 Moment space calculations We start with some preliminary definitions and a lemma that will be needed for the proofs of the theoretical results of Section 7.1. For any measure H supported on [a, b], we write:

b k mk(H) := µ dH(µ), k = 0,...,M, (S57) Za for its moments.S27 We define the moment space:

M+1 := (m0(H), . . . , mM (H)) R H measure on [a, b], exp(µ)dH(µ) = 1 . M ∈  Z  (S58)

The key lemma in this section is the following: Lemma S16 (Open in Moment space). Let H be a measure supported on [a, b], < a < b < with at least M + 2 points of support, such that exp(µ)dH(µ) = 1.−∞ Then, ∞ (m (H), . . . , mM (H)) is an element of the interior of . 0 M R Proof. We will show at the end of the proof, that we may assume without loss of generality M+2 that there exist points a ξ1 < ξ2 < . . . < ξM+2 b such that minj=1 H( ξj ) > ζ for some ζ > 0. ≤ ≤ { } M+1 Take ε > 0 (which we will specify later). Let (m0 , . . . , mM0 ) R be such that ∆k = 0 ∈ m0 mk(H) satisfies ∆k < ε for all k = 0,...,M. We want to show that (m0 , . . . , m0 ) k − | | 0 M ∈ . To this end, we will consider perturbations of H of the following form. For a RM+2, weM consider: ∈ M+2

Ha = H + ajδξj , j=1 X where δξ is the Dirac point mass at ξ. Our goal is to pick a by solving the following linear system: M+2 aj exp(ξj) = 0 j=1 X M+2 k aj ξj = ∆k for k = 0,...,M. j=1 X S27 m0(H) need not be 1, since we do not only consider probability measures.

S18 Call Ξ the matrix of this linear system, then Ξa = (0, ∆0,..., ∆M )>, where

exp(ξ1) exp(ξ2) exp(ξ3) ... exp(ξM+1) exp(ξM+2)  1 1 1 ... 1 1  Ξ = ξ ξ ξ . . . ξ ξ .  1 2 3 M+1 M+2    ......   M M M M M   ξ ξ ξ . . . ξ ξ   1 2 3 M+1 M+2    We will prove below that Ξ is invertible. We pick ε > 0 small enough, so that:

1 ∆k < ε for k = 0,...,M = Ξ− (0, ∆0,..., ∆M )> < ζ/2. | | ⇒ ∞

With a picked as above, it then holds that:

Ha( ξj ) = H( ξj ) + aj ζ/2 > 0. { } { } ≥ M+2 exp(µ)dHa(µ) = exp(µ)dH(µ) + aj exp(ξj) = 1 + 0 = 1. j=1 Z Z X Hence Ha is a candidate measure with moments:

M+2 k mk(Ha) = mk(H) + ajξj = mk(H) + ∆k = mk0 . j=1 X

Thus (m0, . . . , mM0 ) , and so (m0(H), . . . , mM (H)) lies in the interior of . We still need to prove∈ M the invertibility of Ξ. To this end, we define the functionM

exp(ξ1) exp(ξ2) exp(ξ3) ... exp(ξM+1) exp(ξ) 1 1 1 ... 1 1

D(ξ) = ξ ξ ξ . . . ξ ξ 1 2 3 M+1

...... M M M M M ξ ξ ξ . . . ξ ξ 1 2 3 M+1 and want to prove that D(ξM ) = Ξ = 0. Suppose otherwise. Then ξM is a root of +2 | | 6 +2 D( ), and so are ξ , . . . , ξM . On the other hand, we can write D(ξ) as: · 1 +1 M j D(ξ) = b 1 exp(ξ) + bjξ , − j=0 X for some b 1, . . . , bM R. If b 1 = 0, then D( ) is a polynomial of degree M and can have − ∈ − · at most M roots, which is a contradiction. If b 1 = 0, then by applying Rolle’s theorem − 6 multiple times, we find that bj exp( ) must have a root, which is also a contradiction. Thus · D(ξM+2) = 0 and Ξ is invertible. We also6 need to justify why we could assume in the beginning of the proof that there M+2 exist points a ξ1 < ξ2 < . . . < ξM+2 b such that minj=1 H( ξj ) > ζ for some ζ > 0. We follow the≤ proof idea of Pinelis[2017≤]. By assumption, the support{ } set of H consists of at least M + 2 points in [a, b]. This means that there exist pairwise disjoint closed intervals

S19 J1,...,JM+2 in [a, b] such that H(Jj) > 0 for all j = 1,...,M + 2. For each interval Jj there exists a discrete measure Hj supported on at most M + 3 points in Jj such that:

k k exp(µ)dHj(µ) = exp(µ)dH(µ), µ dHj(µ) = µ dH(µ), k = 0,...,M. ZJj ZJj ZJj ZJj This result follows e.g., from Karlin and Studden[1966, Theorem 1.2, Chapter II] by noting that the proof of the invertibility of Ξ also demonstrates that µ (1, µ, . . . , µM , exp(µ)) is a Tchebycheff system [Karlin and Studden, 1966, Definition 1.1,7→ Chapter I] on every closed, nonempty interval that is a subset of [a, b]. Since Hj(Ij) = H(Ij) > 0, there exists ξj Ij ∈ such that Hj( ξj ) > 0. { } Consider the measure H that is defined on Borel sets A [a, b] as follows: ⊂ M+2 M+2 e H(A) = H A Ij + Hj(A Ij). \ ∩ j=1 j=1  [  X e Then H is also a measure supported on [a, b] and: b b e exp(µ)dH(µ) = exp(µ)dH(µ), mk(H) = mk(H), k = 1,...,M. Za Za We may now repeat thee argument of this proof with H replacinge H to arrive at the conclusion of the Lemma. e C.1.2 Proof for Proposition6: DKW- F -Localization. In this section we provide the proof of the statement for the DKW-F -Localization.

Proof. We divide the proof into three steps. Throughout we write cn = log(2/α)/(2n) and for the length of the DKW-F -localization interval. |I| p • Step 1: We first prove that 4cn almost surely. |I| ≤ M M • Step 2: Let An be the event on which there exist distributions F` , Fu on 0, 1,...,M 1,M,. with F M , F M DKW(α) that make the inequalities used in Step 1 tight.{ We show − } ` u ∈ Fn that PG [An] 1 as n . → → ∞ • Step 3: In Step 2, we ignored the fact that the distributions we constructed need not u be marginal distributions in the empirical Bayes problem. Let Bn be the event that M M M the distribution Fu from Step 2 may be represented as Fu = FG˜ as in (5), where ˜ ` M u ` G . Bn is defined similarly for F` . We then prove that also PG Bn Bn 1. ∈ G ∩ → Using the results from Steps 2 and 3, we see that with probability tending to 1, it holds that 4cn, and so it also follows that /(4cn) = 1 + oPG (1). |I| ≥ DKW|I| Step 1: Take any distribution F n (α). Then the following holds for its density at z 1,...,M . ∈ F ∈ { } f(z) = F (z) F (z 1) − − = Fn(z) Fn(z 1) + F (z) Fn(z) F (z 1) Fn(z 1) − − − − − − −       (S59) = fˆn(z) + F (z) Fn(z) F (z 1) Fn(z 1) b b − − −b − − b     fˆn(z) + 2cn. ≤ b b

S20 In the last inequality we used the definition of the DKW band and fˆn(z) = # Zi = z) /n. { } Similarly, we may conclude that f(z) fˆn(z) 2cn. Combining these two results, we see ≥ − that the DKW-F -Localization band for L(G) = fG(z) must satisfy,

[fˆn(z) 2cn, fˆn(z) + 2cn], I ⊂ − and so its length can be at most 4cn.

Step 2: We define Fu as follows:

Fu(z0) = Fn(z0) for z0 / z 1, z ,Fu(z 1) = Fn(z 1) cn,Fu(z) = Fn(z) + cn. ∈ { − } − − − F is tight for the inequality in (S59), since: u b b b

Fu(z) Fn(z) Fu(z 1) Fn(z 1) = 2cn. − − − − −     Fu satisfies the constraints ofb the DKW-band, howeverb Fu is not necessarily a distribution u function. Let us define A as the event on which Fn(z 1) cn > Fn(z 2) (or > 0 n − − − if z = 1) and Fn(z) + cn < Fn(z + 1). Since the true distribution is a Poisson mixture FG and G is not only supported on the point 0, it holdsb that FG(z 2)b < FG(z 1) (if − − z 2 and FG(zb 1) > 0 if z =b 1) and that FG(z) < FG(z + 1). Since cn 0 and by the ≥ − u → Glivenko-Cantelli Theorem, we conclude that PG [An] 1. We may similarly define and ` → ` u argue for F` (with a corresponding event A ). Since An A A , we conclude. n ⊃ n ∩ n

Step 3: We define HG as the measure that is absolutely continuous w.r.t. G with Radon- Nikodym derivative dHG/dG(µ) = exp( µ). Recall the definition of moments mk( ) in (S57) − · and the moment space (S58). For the measure HG it holds that M b b k k mk(HG) = µ dHG(µ) = µ exp( µ) dG(µ) = k!fG(k). − Za Za Furthermore, since G is supported on at least M + 2 points, so is HG. By Lemma S16 (m0(HG), . . . , mM (HG)) lies in the interior of the moment space , i.e., there exists an M+1 M open set R such that (m0(HG), . . . , mM (HG)) and . We define the bijective mappingU ⊂ ∈ U U ⊂ M

k M+1 M+1 uj T = (T0,...,TM ): R R ,Tk((u0, . . . , uM )) = , k = 0,...,M. → j! j=0 X M+1 Then, for two elements u = (u0, . . . , uM ), v = (v0, . . . , vM ) R , we let: ∈ M d(u, v) = max Tk(u) Tk(v) . k=0 | − | d( , ) is a distance for RM+1 that metrizes the standard topology. Thus there exists ε > 0 so· that:· u for all u with d (u, (m (HG), . . . , mM (HG))) < ε. ∈ U 0 Observing that Tk (m0(HG), . . . , mM (HG)) = FG(k) for k = 0,...,M, we conclude that: 1 d T − (Fu(0),...,Fu(M)), (m0(HG), . . . , mM (HG)) M M M = max Fu(k) FG(k) = sup Fu (t) FG (t) .
k=0 | − | t −

S21 By construction of Fu in Step 2 and the Glivenko-Cantelli theorem, we see that the RHS above converges almost surely to 0. In turn, this means that

1 P T − (Fu(0),...,Fu(M)) 1. ∈ U →

On the event inside the probability above, we can find a measure Hu supported on [a, b], such that: T (m0(Hu), . . . , mM (Hu)) = (Fu(0),...,Fu(M)).

Finally, let Gu be the measure that is absolutely continuous w.r.t. Hu with Radon-Nikodym derivative dGu(µ)/dHu(µ) = exp(µ). Gu is a probability measure, since:

dGu(µ) = exp(µ)dHu(µ) = 1, Z Z by definition of the moment space . Furthermore, it holds that Fu = FG on 0, 1 ...,M,. . M u { } We conclude after arguing analogously for F`.

C.1.3 Proof for Proposition6: AMARI. The modulus problem (28) at δ > 0 takes the form:

2 2 fG1 (z0) fG−1 (z0) δ maximize fG1 (z) fG−1 (z) s.t. − . (S60) G1,G−1 n − f¯(z ) ≤ n ∈G z0 0,...,M . 0
∈{ X }∪{ } We first solve a relaxation of the above optimization problem (we will show that the re- laxation is tight later), in which we introduce variables hj that formally correspond to fG (j) fG (j): 1 − −1 maximize hz h0,...,hM ,h. h2 δ2 subject to j ¯ ≤ n (S61) j f(j) X hj = 0. j X This is a convex optimization problem. Consider the Lagrangian with dual variables ζ R and ξ < 0 (the dual objective is unbounded for ξ = 0): ∈

2 2 hj δ (h; ξ, ζ) = hz + ξ + ζ hj. L  ¯ − n  j f(j) j X X   The derivative with respect to hz is equal to,

∂ (h; ξ, ζ) hz L = 1 + 2ξ + ζ, ∂hz f¯(z) and with respect to hj, j = z: 6

∂ (h; ξ, ζ) hj L = 2ξ + ζ, ∂hj f¯(j)

S22 By the first order optimality conditions, we conclude that hj/f¯(j) = hj0 /f¯(j0) for all j, j0 = 6 z, and so there exists a constant t such that hj = tf¯(j) for all j = z. Furthermore, 6

hz = hj = t f¯(j) = t(1 f¯(z)). − − − − j=z j=z X6 X6

Thus t = hz/(1 f¯(z)) and we only need to optimize in (S61) with respect to a single − − 2 parameter hz, which we seek to maximize. The pseudo-χ constraint takes the form:

h2 h2 j = z + t2f¯(j) f¯(j) f¯(z) j j=z X X6 h2 = z + t2(1 f¯(z)) f¯(z) − h2 h2 = z + z f¯(z) 1 f¯(z) − h2 = z . f¯(z)(1 f¯(z)) − 2 We want the above to be equal to δ /n, and so to maximize hz subject to the above constraint, we get δ hz = f¯(z)(1 f¯(z)), (S62) √n − q as the optimal value of the relaxed modulus problem (S61). To argue that the relaxation is tight (with probability tending to 1), we need to exhibit priors G1,G 1 n such that − ∈ G fG1 (j) fG−1 (j) = hj, where (hj)j is the maximizer of (S61) derived above. To do so, we − ˆDKW proceed as follows. Let fn (z) be the frequency of z in the sample used to construct the pilot DKW-F -localization. Then define the pmf fu on 0,...,M,. as: { } ¯ DKW hz DKW hz0 f(z0) fu(z) = fˆ (z) + , fu(z0) = fˆ (z0) for z0 0,...,M,. z . n 2 n − 2(1 f¯(z)) ∈ { }\{ } −

We make the following observation. First, hz in (S62) is of order OP(1/√n), which is of √ smaller order than the width of the DKW band OP( log(2/αn)/ k). Thus, arguing as in Steps 2 and 3 of Supplement C.1.2, we can prove that the following event has probability p tending to 1: there exists a prior G = G ,n such that FG n and such that 1 1 ∈ G 1 ∈ F fG (z0) = fu(z0) for all z0 0,...,M,. . The same argument also applies to f` defined as 1 ∈ { } f`(z0) = fu(z0) hz0 for z0 0,...,M,. . We thus conclude that the optimal value of the modulus problem− (S60) is equal∈ { to (making} the dependence of f¯(z) on n explicit):

δ ωn(δ) = f¯n(z)(1 f¯n(z)). √n − q ωn(δ) is differentiable in δ with derivative 1 ω0 (δ) = f¯n(z)(1 f¯n(z)) = ωn(δ)/δ. n √n · − q

S23 Let us plug the above into (30) to find the optimal Q( ). First, we consider the part of Q that is a function of z. For j = z we get · 6

n ω0 (δ) fG1 (j) fG−1 (j) n ω0 (δ) hz · n − = · n δ · f¯n(j) − δ · 1 f¯n(z) − n ω0 (δ) ωn(δ) = · n − δ · 1 f¯n(z) 2 − n ωn(δ) = 2 −δ 1 f¯n(z) − = f¯n(z). − For z we get:

2 n ω (δ) fG (z) fG (z) n ω (δ) ω (δ) n ω (δ) f¯ (1)(1 f¯ (z)) n0 1 − −1 = n0 n = n = n n = 1 f¯ (z). · · ·2 − n δ · f¯n(z) δ · f¯n(z) δ f¯n(z) f¯n(z) −

It remains to evaluate the additive component in (30) that does not depend on z. Let

G0 = (G1 + G 1)/2. We note that by construction L(G0) = fG0 (z). Hence the constant term is equal to:−

n ω0 (δ) fG1 (j) fG−1 (j) fG0 (j) · n − + L(G ) − δ · f¯ (j) 0 j n
X 2 n ωn(δ) fG0 (z) 1 fG0 (z) = · 2 − + L(G0) − δ · f¯n(z) − 1 f¯n(z)  −  = f¯n(z)(1 fG (z)) (1 f¯n(z))fG (z) + L(G ) − 0 − − 0 0 = f¯n(z).

We conclude that for j = z, Q(j) = f¯n(z)+f¯n(z) = 0 and for z, Q(z) = 1 f¯n(z)+f¯n(z) = 6 − − 1. Thus Q( ) = 1( = z). Hence L = fˆn(z) = # Zi = z /n. The worst case absolute bias · · { } of L is given by: b 1 B = (ωn(δ) δωn0 (δ)) = 0. b 2 − The confidence intervals of AMARIb in (15)(V as in (13)) hence take the form:

L q1 α/b 2 V, ± − · p with q1 α/2 the 1 α/2 quantile of theb standard Normalb distribution. Since nV fG(z)(1 − − → − fG(z)), we conclude. b C.1.4 Proof for Proposition7: DKW- F -Localization Proof. The proof will be structured very similarly to the proof in Supplement C.1.2 that concerned inference for fG(z). In particular, we follow the same three steps as in that proof. Step 1: Take any distribution F DKW(α). Write θ(z) = (z + 1)f(z + 1)/f(z), we seek ∈ Fn to provide lower and upper bounds on it. Note that when F = FG, then by (9), we have that θ(z) = EG µ Z = z .  

S24 f(z + 1) F (z + 1) F (z) (z + 1) = (z + 1) − f(z) F (z) F (z 1) − − fˆn(z + 1) + F (z + 1) Fn(z + 1) F (z) Fn(z) = (z + 1) − − −     (S63) fˆn(z) + F (z) Fn(z) F (z 1) Fn(z 1) − b− − − −b fˆ (z + 1) + 2c    (z + 1) n n . b b ≤ fˆn(z) 2cn − In the last inequality we used the definition of the DKW band. Similarly, we may conclude that f(z + 1) fˆn(z + 1) 2cn (z + 1) (z + 1) − . (S64) f(z) ≥ fˆn(z) + 2cn

Combining these two results, we see that the DKW-F -Localization band for L(G) = fG(z) must satisfy,

fˆn(z + 1) 2cn fˆn(z + 1) + 2cn 0 := (z + 1) − , (z + 1) . ˆ ˆ I ⊂ I " fn(z) + 2cn fn(z) 2cn # − We will prove that the above inclusion is in fact an equality below (with high probability and for large n). For now, we verify that 0 has the claimed asymptotic length. I

4cn fˆn(z + 1) + fˆn(z) √n 0 = √n(z + 1) |I |   fˆn(z) 2cn fˆn(z) + 2cn − (S65)  fG(z ) + fG(z + 1) = 2(z + 1) 2 log(2/α) 2 + oPG (1). fG(z) p Steps 2 and 3: We define Fu as follows:

Fu(z0) = Fn(z0) for z0 / z 1, z, z + 1 , ∈ { − } Fu(z 1) = Fn(z 1) + cn,Fu(z) = Fn(z) cn,Fu(z + 1) = Fn(z + 1) + cn. − b − − Plugging Fu into (S63b ), we see that the last inequalityb is an equality.b Furthermore, arguing as in Steps 2 and 3 of Supplement C.1.2, we can prove that the following event has probability tending to 1: There exists a prior G = Gn such that FG n and such that FG(z0) = ∈ G ∈ F Fu(z0) for all z0 0,...,M,. . We may define F` that makes (S64) tight analogously. ∈ { } Hence, on an event that has probability tending to 1, it holds that = 0. Thus the I I asymptotic length of is equal to the asymptotic length of 0 computed in (S65). I I C.1.5 Proof for Proposition7: AMARI. Proof. In Supplement C.1.3 we solved the modulus problem in the Poisson problem, when L(G) = fG(z), and proved that the optimal Q( ) in (30) takes the form Q( ) = 1( = z) on an event with asymptotic probability 1. Here we· will start by proving a generalization· · of the above result.

S25 Concretely, we will fix a = (a0, . . . , aM , a.) and we will consider the linear functional L(G) = z0 0,...,M . az0 fG(z0). For notational convenience (and with some abuse of notation), we∈{ identity}∪{f } with the vector (f (0), . . . , f (M), f (.)). We also define the P G G G G matrix D¯ = (f¯G(0),..., f¯G(M), f¯G(.)) and write h = (h0, . . . , hM , h.). The relaxed modu- lus problem (compare to (S61)) takes the form:

maximize a>h h h2 δ2 subject to j ¯ ≤ n (S66) j f(j) X hj = 0. j X Instead of the relaxed modulus problem, we consider the relaxed inverse modulus prob- lem,S28 which is parameterized by t > 0:

1 minimize h>D¯ − h h

subject to a>h = t (S67)

1>h = 0.

(S67) will enable us to also solve (S66) and then to compute ωn(δ). (S67) is also a convex problem, so we introduce the Lagrangian (with dual variables ξ, ζ R) ∈ 1 (h; ζ, ξ) = h>D¯ − h + 2ξ(a>h t) + 2ζ1>h. L − We multiply the dual variables by 2 only for convenience. By the first order optimality conditions, we see that:

1 D¯ − h = ξa ζ1 = h = D¯ (ξa + ζ1) . − − ⇒ −

ξ and ζ are determined by a system of two linear equations. Namely from a>h = t, 1>h = 0.

ξa>Da¯ + ζa>D¯1 = t − ξa>D¯1 + ζ = 0.

In deriving the second of the above inequalities, we used the fact that 1>D¯1 = 1. It follows that: 2 ξ = t a>D¯1 a>Da¯ , ζ = ξa>D¯1. − −  n o The objective value of (S67) is then equal
to:

1 2 2 h>D¯ − h = ξ a>Da¯ + ζ + 2ξζaD¯1

2 2 = ξ a>Da¯ a>D¯1 · − n o 2
2 = t a>Da¯ a>D¯1 . −  n
o S28The proof of Theorem 3 in Cai and Low[2003] uses a similar proof technique using the inverse modulus of continuity.

S26 We have solved the relaxed inverse modulus problem. This yields the solution to the relaxed 1 2 modulus problem (S66) by choosing t so that h>D¯ − h = δ /n, and so, the optimal value of (S66) is equal to: 2 2 δ 2 t = a>Da¯ a>D¯1 . n − Arguing as in Supplement C.1.3, we findn that the relaxed
o modulus problem is tight for the modulus problem (with probability tending to 1) and on the latter event: 2 δ 2 ωn(δ) = a>Da¯ a>D¯1 . n − n o Consequently, ωn is differentiable at δ > 0 with ωn0 (δ) =
ωn(δ)/δ. We can continue as in the proof in Section C.1.3 by plugging the above into (30). The constant additive part of Q( ) is equal to a>D¯1. Hence, identifying Q with the vector (Q(0),...,Q(M),Q(.)), we find· that: n ωn0 (δ) 1 Q = · D¯ − h + a>D¯1 1 δ · nωn(δ)
= (ξa + ζ1) + a>D¯1 1 − δ2 nωn(δ)
= ξ a a>D¯1 1 + a>D¯1 1 − δ2 − = a 

nωn(δ) In the last step, we used the fact that 2 ξ = 1, since: − δ nωn(δ) nωn(δ) 2 ξ = ωn(δ) a>D¯1 a>Da¯ − δ2 − δ2 −  2 n o n δ
2 2 = a>Da¯ a>D¯1 a>D¯1 a>Da¯ = 1. −δ2 · n − − n o  n o We note that the resulting estimator for L(G) is
unbiased, i.e., the
worst case bias in this case is equal to 0. We are ready to return to the study of Algorithm2. Let [ c`, cu] be the pilot F -localization ` u intervals for θG(z). By construction PG θG(z) [c , c ] 1, and furthermore, by the proof ` ∈ → u in Supplement C.1.4, we also have that c = θG(z) + o (1) and c = θG(z) + o (1). By  PG PG the preceding argument, we get for z0 0,...,M,. : ∈ { } ` ` u u Q (z0) = (z +1)1(z0 = z +1) c 1(z0 = z +1),Q ( ) = (z +1)1(z0 = z +1) c 1(z0 = z +1). − · − ` ` u u In particular L = (z + 1)fˆn(z + 1) c fˆn(z), L = (z + 1)fˆn(z + 1) c fˆn(z), and ` u c − c− for any c [c , c ], L = (z + 1)fˆn(z + 1) cfˆn(z). Next note that B = 0 and since ∈ − tα(0,V ) = q1 bα/2√V in (15), to determine the AMARIb confidence interval for θG(z), we − need to determine allbc [c`, cu] such that: b ∈ ˆ ˆ c 0 α(z; c) = (z + 1)fn(z + 1) cfn(z) q1 α/2 V . ∈ I − ± − c p To do this it will bee furthermore convenient to express V in Algorithmb 2 in a slightly different form, namely b c 2 2 1 V = (z + 1) V˜ + c V˜ 2c(z + 1)V˜ , V˜ = fˆn(z)(1 fˆn(z)), 2 1 − 12 1 n 1 − 1 − 1 Vb˜ = fˆn(z + 1)(1 fˆn(z + 1)), V˜ = fˆn(z + 1)fˆn(z). 2 n 1 − 12 −n 1 − − S27 c 2 Having rewritten V as above and shortening q = q1 α/2, we see that: − 2 2 2 2 0 α(z; c) b (z + 1)fˆn(z + 1) cfˆn(z) q (z + 1) V˜ + c V˜ 2c(z + 1)V˜ . ∈ I ⇐⇒ − ≤ 2 1 − 12   h i The lattere condition is a quadratic inequality in c, that we may rearrange as: 2 2 2 2 2 2 2 fˆn(z) q V˜ c + 2(z+1) q V˜ fˆn(z + 1)fˆn(z) c + (z+1) fˆn(z + 1) q V˜ 0. − 1 12 − − 2 ≤       We make the observation that c = (z +1)fˆn(z +1)/fˆn(z) is an interior point of the above in- 2 2 equality. Furthermore, on the event fˆn(z) > q V˜1 the above is a convex quadratic, and so { } ˆ 2 2 ˜ the set of c satisfying the inequality must be a closed interval. Since PG fn(z) > q V1 1 → as n , we restrict attention to that event. On that event, the distanceh between thei two roots→ of ∞ the quadratic is equal to:

2(z + 1)q 2 2 2 2 q V˜ V˜1V˜2 + V˜1fˆn(z + 1) + V˜2fˆn(z) 2fˆn(z)fˆn(z + 1)V˜12 . 2 2 12 fˆn(z) q V˜1 − − − r     Noting that nV˜ = fG(z)(1 fG(z)) + o (1), nV˜ = fG(z + 1)(1 fG(z + 1)) + o (1) and 1 − PG 2 − PG nV˜ = fG(z)fG(z + 1) + o (1), we conclude that the above is asymptotically equal to: 12 − PG 2(z + 1)q f(z)(1 f(z))f(z + 1)2 + f(z + 1)(1 f(z + 1))f(z)2 + 2f(z)2f(z + 1)2(1 + o (1)) √nf(z)2 − − PG 2(z + 1)q p = f(z)f(z + 1) (f(z) + f(z + 1))(1 + o (1)). √nf(z)2 PG p This is the confidence interval length claimed in the statement of the Proposition.

C.2 Bernoulli model (Section 7.2)

In this section we consider model (1) with Zi µi Bernoulli(µi), i.e., the Binomial model with a single (N = 1) trial. Furthermore, we do not∼ impose additional structure on , i.e., G we assume that G = ([0, 1]). Under the above model, Zi is supported on 0, 1 and ∈ G P z { } 1 z we can take λ = δ + δ to be the counting measure on 0, 1 and p(z µ) = µ (1 µ) − . 0 1 { } − The marginal distribution FG is fully determined by fG(1), since fG(0) = 1 fG(1). In this − case, the F -localizations we consider take the following simplified form. First, the DKW F -localization (16) is equal to:

DKW (α) = F ( 0, 1 ) with pmf f : f(1) fˆn(1) log (2/α) (2n) . (S68) Fn ∈ P { } − ≤  q  2 2 2  Second, for the χ -F -localization (22), write τ = χ1,1 α, then: −

2 χ (α) = F ( 0, 1 ) with pmf f : Fn ∈ P { }  ˆ 2 2 fn(1) + τ /(2n) τ /n 2 f(1) fˆn(1)(1 fˆn(1)) + τ /(4n) . − 1 + τ 2/n ≤ 1 + τ 2/n · − p q  (S69)

An important observation that we will use throughout the following proofs, is that any ˜ distribution F ( 0, 1 ) can be represented as FG˜ for some G ([0, 1]) in model (1) with the Bernoulli∈ P likelihood.{ } ∈ P

S28 C.2.1 Proof of Proposition8: Second moment Proof. We study the second moment of the prior. As already mentioned in the main text, this is an example of a linear functional that is partially identified. We discuss the partial identification aspect first. Suppose we know the marginal distribution of Zi exactly, that is, we know fG(1). Notice that EG [µ] = µ dG(µ) = fG(1). Then, the partial identification interval for L(G) is the following: R 2 2 L(G) µ dG(µ) , µ dG(µ) = fG(1) , fG(1) . (S70) ∈ "Z  Z #   For example,when fG(1) = 1/2, then L(G) [1/4, 1/2]. Why is the above the partial identification interval? First note that µ2 dG∈(µ) µ dG(µ) holds since G is supported on [0, 1] and µ2 dG(µ) ( µ dG(µ))2 holds by Jensen’s≤ inequality. Furthermore, there ≥ R R exist choices of G that make both inequalities tight. In particular, if G = δµ¯ for some R 2R 2 µ¯ then µ dG(µ) =µ, ¯ µ dG(µ) =µ ¯ , while for G = (1 µ¯)δ0 +µδ ¯ 1, it holds that µ dG(µ) = µ2 dG(µ) =µ. ¯ − We seekR to determineR the (asymptotic) length of the different confidence intervals we considerR in thisR work. We start with the F -localization approaches.

F -localization: By the above discussion on partial identification, we find that the F - 2 localization intervals take the form [infG fG(1) , sup fG(1) ] where the extrema are G { } taken over all G such that FG n(α). We further restrict attention to the event wherein ∈ F  infG fG(1) (0, 1) and supG fG(1) (0, 1). Since fG(1) (0, 1) under the assumptions of Proposition{ } ∈3, this event will occur{ with} ∈ asymptotic probability∈ 1 for both F -localizations. For the DKW-F -localization, in view of (S68), we then get the interval:

2 DKW = fˆn(1) log (2/α) (2n) , fˆn(1) + log (2/α) (2n) . I − "  # q  q  2 2 2 Similarly, for the χ -F -localization (S69) we get the interval (with τ = χ1,1 α): − 2 2 ˆ 2 2 χ fn(1) + τ /(2n) τ /n 2 = fˆn(1)(1 fˆn(1)) + τ /(4n) , I 1 + τ 2/n − 1 + τ 2/n · − " p q ! ˆ 2 2 fn(1) + τ /(2n) τ /n 2 + fˆn(1)(1 fˆn(1)) + τ /(4n) . 1 + τ 2/n 1 + τ 2/n · − p q #

DKW χ2 Since fˆn(1) = fG(1) + o (1), as n , it follows for both = and = , that: P → ∞ I I I I 2 = fG(1) fG(1) + o (1) = fG(1)(1 fG(1)) + o (1), |I| − P − P as claimed.

AMARI: The modulus problem (28) at δ > 0 takes the form:

δ ¯ ¯ maximize L(G1) L(G 1) s.t. fG1 (1) fG−1 (1) fn(1)(1 fn(1)). (S71) G1,G−1 n − − − ≤ √n · − ∈G q

S29 By (S70), it may be simplified as:

2 δ ¯ ¯ maximize fG1 (1) fG−1 (1) s.t. fG1 (1) fG−1 (1) fn(1)(1 fn(1)). (S72) G1,G−1 n − − ≤ √n · − ∈G q

We write p = fG−1 (1) and fG1 (1) = p + ε, for a choice of ε 0 that we will make below. Then we seek to find (feasible choices of p, ε) so that (p + ε) ≥p2 is maximized. We seek to ` − solve this problem for some δ = δn δ > 0. Throughout the rest of the proof we assume ≥ that fG(1) [1/2, 1); the case fG(1) (0, 1/2) being analogous. We define: ∈ ∈ ˜ pn = max 1/2, inf fG˜ (1) G n . ¯ | ∈ G n n oo 2 We note that pn = fG(1) + oP(1). Since p (p + ε) p is decreasing in p for p 1/2, it ¯ 7→ − ≥ 2 follows that (S72) is optimized for the choice fG1 (1) = p = pn. Furthermore, ε (p+ε) p is increasing in ε, and so it is maximized for the largest¯ admissible value of7→ ε. By− the constraint in (S72), we see that

ε (δ/√n) f¯n(1)(1 f¯n(1)). ≤ · − q The above constraint may be replaced by an equality, since the RHS above is of order

OP(1/√n) and so, the distribution F with f(1) = pn + ε would be included in both F - localizations (S68) and (S69) for n large enough. We¯ conclude that

δ ωn(δ) = pn(1 pn) + f¯n(1)(1 f¯n(1)), ¯ − ¯ √n · − q which is differentiable in δ with: 1 ω0 (δ) = f¯n(1)(1 f¯n(1)). n √n · − q Let us plug the above into (30) to find the optimal Q( ). First, we consider the part of Q that is a function of z. For z = 1 we get · ¯ ¯ n ωn0 (δ) fG1 (1) fG−1 (1) fn(1)(1 fn(1)) · − = − = 1 f¯n(1). δ · f¯n(1) f¯n(1) −

Similarly, for z = 0, we get f¯n(1). It remains to evaluate the additive component in (30) − that does not depend on z. Let G0 = (G1 + G 1)/2, where G 1,G1 are any priors that − − have marginal pmf at z = 1 equal to pn, respectively pn + ε. Then ¯ ¯ p (1 + p ) 1 2 n n δ ¯ ¯ L(G0) = (pn + ε) + pn = ¯ ¯ + fn(1)(1 fn(1)). 2 ¯ ¯ 2 2√n · −  q ε δ ¯ ¯ fG0 (1) = pn + = pn + fn(1)(1 fn(1)). ¯ 2 ¯ 2√n · − q

S30 Using the above two results, we find that the constant term is equal to:

1 n ω0 (δ) fG1 (z) fG−1 (z) fG0 (z) · n − + L(G ) − δ · f¯ (z) 0 z=0 n
X = f¯n(1) fG (1) + L(G ) − 0 0 pn(1 + pn) = f¯n(1) pn + ¯ ¯ − ¯ 2 pn(1 pn) = f¯n(1) ¯ − ¯ . − 2 Hence we now have an explicit expression for Q(z) in (30) for z 0, 1 : ∈ { } pn(1 pn) Q(z) = 1(z = 1) ¯ − ¯ . − 2

pn(1 pn) This means that L = fˆn(1) ¯ −¯ , where fˆn(1) = # Zi = 1 /n. The worst case − 2 { } absolute bias of L is given by: b b 1 pn(1 pn) B = (ωn(δ) δω0 (δ)) = ¯ − ¯ . 2 − n 2 With V as in (13), we finallyb get the confidence interval (15):

b pn(1 pn) α = L tα(B, V ) = fˆn(1) ¯ − ¯ tα(B, V ). I ± − 2 ±

Under the given asymptoticsb V b= bo (1), B = fG(1)(1 fG(1))/2 +b ob(1) and so it follows P − P that for α (0, 1), tα(B, V ) = fG(1)(1 fG(1))/2 + oP(1). We conclude that the left endpoint of∈ the confidence intervalb convergesb− in probability to: b b 2 fG(1) fG(1)(1 fG(1))/2 fG(1)(1 fG(1))/2 = fG(1) . − − − − The right point of the AMARI confidence interval converges in probability to:

fG(1) fG(1)(1 fG(1))/2 + fG(1)(1 fG(1))/2 = fG(1). − − − Hence we conclude that the length of theAMARI confidence intervals converges to the length of the partial identification interval.

C.2.2 Proof of Proposition8: Posterior mean Proof. We now turn to study,

µ2 dG(µ) θG(1) = EG µ Z = 1 = . f (1) R G   For a fixed value of the denominator fG(µ )(1), we derived partial identification intervals for the numerator in (S70). It directly follows that the partial identification intervals for θG(1) are equal to: θG(1) [fG(1), 1] . ∈

S31 F -localization: The argument now is very similar to that for the second moment. With the DKW-F -localization, in view of (S68), we get the interval

DKW (1) = fˆn(1) log (2/α) (2n), 1 [0, 1]. I −   q  \ 2 2 2 Similarly, for the χ -F -localization (S69) we get the interval (with τ = χ1,1 α): −

2 2 2 fˆ (1) + τ /(2n) τ /n χ n ˆ ˆ 2 (1) = 2 2 fn(1)(1 fn(1)) + τ /(4n), 1 [0, 1]. I " 1 + τ /n − 1p + τ /n · − # q \ ˆ DKW Since fn(1) = fG(1) + oP(1), as n , it follows for both (1) = (1) and (1) = 2 → ∞ I I I χ (1), that: I (1) = 1 fG(1) + o (1). |I | − P

AMARI: For fixed c [0, 1], we start by studying the modulus problem (S71) for the linear functional ∈ L(G) = θlin(z; c) = µ2 dG(µ) c µ dG(µ). G − Z Z Following precisely the derivation in the proof for AMARI in Supplement C.2.1 and the notation used therein, we find that on an event with asymptotic probability 1, it holds that the optimal Qc( )(30) for all c [0, 1] takes the form: · ∈

pn(1 pn) Qc(z) = (1 c)1(z = 1) ¯ − ¯ , − − 2 with worst-case bias B = pn(1 pn)/2 and V = o (1). Write α(z; c) for the confidence − P I lin ˆlin¯ ¯ ˆlin interval for θG (z; c) and θ (z; c), resp. θ+ (z; c) for its left and right endpoints. All oP(1) terms above are uniformb with− respect to c, andb so arguing againe as in Supplement C.2.1, it follows that:

ˆlin ˆlin sup θ (z; c) fG(1)(fG(1) c) = oP(1), sup θ+ (z; c) fG(1)(1 c) = oP(1). c [0,1] − − − c [0,1] − − ∈ ∈

` u Recall that in Algorithm2 we seek to find all c [c , c ] such that 0 α(z; c), where [c`, cu] [0, 1] is the pilot interval. Fix ζ > 0 small,∈ then by the above uniform∈ I convergence, ⊂ we have that: e

P 0 α(z; c) for any 0 c fG(1) ζ 0 as n , ∈ I ≤ ≤ − → → ∞ h i and that: e P 0 α(z; c) for all 1 c fG(1) + ζ 1 as n . ∈ I ≥ ≥ → → ∞ h i Since ζ > 0 was arbitrary, we thus we find that the left-most endpoint of α(z) converges e I in probability to fG(1) and the right-most endpoint converges to 1, i.e., the asymptotic confidence interval length is equal to 1 fG(1). −

S32 D Computational aspects for F -localization D.1 Parametric convex programming for F -localization intervals + We explain how to compute θˆ (z) in (7) (the steps for θˆ−(z) being analogous) when α α G and n are convex, but not necessarily representable through linear constraints. Recall F that θG(z) = aG(z)/fG(z). We first compute confidence intervals for fG(z) using the same F -localization, i.e.,

+ fˆ−(z) = inf fG(z) G ( n(α)) , fˆ (z) = sup fG(z) G ( n(α)) . α { | ∈ G F } α { | ∈ G F } The objective here is linear and the constraints are convex, and so the above is a convex programming problem. We then observe that

+ θˆ (z) = sup θG(z) G ( n(α)) α { | ∈ G F } + = sup sup aG(z)/t G ( n(α)) , fG(z) = t t [fˆ−(z), fˆ (z)] . { | ∈ G F } | ∈ α α h i ˆ ˆ+ Hence, we can proceed as follows. Let be a fine discretization of [fα−(z), fα (z)], say with 100 equidistant points. Then, for each Tt , compute: ∈ T + θˆ (z; t) = sup aG(z)/t G ( n(α)) , fG(z) = t . α { | ∈ G F } Note that this problem also has an objective that is linear in G, and specifies convex ˆ+ constraints on G, i.e., it is a convex programming problem. Finally, we report θα (z) = ˆ+ maxt θα (z; t). ∈T D.2 Considerations for discretization of G If an infinite-dimensional is specified, then it is important to guarantee that the error incurred when solving (7)G or (28) with a discretized class instead of , is negligible compared to e.g., the confidence interval width. This typicallyG requires toG be tight, and G in our applications we used the prior classes (4), (34) and (35)e with chosen as a compact set. K The following proposition can be used to verify the accuracy of a discretization . G Proposition S17. Consider the linear functional L(G) = ψ(µ)dG(µ) for a function ψ( ). e · a) If , then, ˜ R ˜ , where ψ(µ) Cψ infG˜ e L(G) L(G) Cψ/2 infG˜ e TV(G, G) | | ≤ ∈G − ≤ ∈G{ } ˜ ˜ is the total variation distance between and ˜. TV(G, G) = supA G(A) G( A) G G | − | b) If is -Lipschitz continuous, then, ˜ ˜ ψ(µ) Cψ infG˜ e L(G) L(G) Cψ infG˜ e W1(G, G) , ˜ ∈G − ≤ ∈˜G{ } with W1(G, G) = inf E[ µ µ˜ ]:(µ, µ˜) random variables s.t. µ G, µ˜ G the Wasser- { | − | ∼ ∼ } stein distance between G and G˜ (cf. Panaretos and Zemel[2019 ] and references therein).

Proof. a) Recall that TV(G, G˜) = 1 dG(µ) dG˜(µ) . Thus, 2 | − | R ˜ L(G) L(G) = ψ(µ) dG(µ) dG˜(µ) Cψ dG(µ) dG˜(µ) 2Cψ TV(G, G˜). − − ≤ − ≤ Z Z  

S33 b) Letting µ G, µ˜ G˜, the optimal Wasserstein coupling, we get ∼ ∼ ˜ ˜ L(G) L(G) = E [ψ(µ)] E [ψ(˜µ)] E [ ψ(µ) ψ(˜µ) ] CψE [ µ µ˜ ] = CψW1(G, G). − | − | ≤ | − | ≤ | − |

For example, when part b) of the Proposition is applicable, then it suffices for to be a cover of in terms of the Wasserstein distance. In some cases, part b) is not appli-G G cable. For example, when constructing intervals for the local false sign rate in the stan-e dard Gaussian empirical Bayes problem, then the numerator aG(z) in (10) takes the form aG(z) = ψ(µ)dG(µ) with ψ(µ) = 1(µ 0)ϕ(z µ), and so ψ( ) is not Lipschitz continuous. Instead, part a) of the Proposition is applicable,≥ − and so a cover· in total variation suffices. BelowR we provide details for the discretization of ( )(4) and (τ, )(34), when is a compact interval. P K LN K K

D.2.1 Discretization of compactly supported distributions Consider ([L, U]) (4), the class of all distributions supported on the compact interval = [L, U].P We first discretize = [L, U] as the finite grid K K U L U L (p, L, U) = L, L + − ,L + 2 − ,...,U , p N. (S73) K p p ∈   Then ([L, U]) may be discretized by considering ( (p, L, U)), the class of all distributions supportedP on the grid (p, L, U). This class isP amenableK to our optimization tasks. By K enumerating the grid elements as (p, L, U) = µ , . . . , µp , we may represent every K { 1 +1} G ( (p, L, U)) by the probabilities πj = PG [µ = µj] assigned to µj, and so we may identify∈ P K( (p, L, U)) with the probability simplex: P K p+1 p+1 p+1 S = (π , . . . , πp ) [0, 1] πj = 1 . (S74)  1 +1 ∈  j=1  X 

Sp+1 is a linear polytope. See (17) for an explicit example of how it is used for computations. In Section 5.1 we discretized ([0, 1]) as above with p + 1 = 300. Finally, we note that the discretizationP ( (p, L, U)) provides a O((U L)/p) covering of ([L, U]) in Wasserstein distance, but notP inK total variation distance. For− this discretization scheme,P Proposition S17 justifies inference for functionals satisfying b) of its statement. The implication for inference on empirical Bayes estimands is that we may use the above discretization to conduct inference for the posterior mean, e.g., in the Gaussian empirical Bayes problem.

D.2.2 Discretization of Gaussian location mixtures Consider the class of distributions (τ, ) from (34) in the special case where = [L, U] is a compact interval. Then lettingLN(p, L,K U)(S73) the equidistant discretizationK of [L, U], we use (τ, (p, L, U)) as a discretizationK of (τ, ) that is amenable to efficient computation.LN WeK have the following numerical representationLN K of (τ, (p, L, U)): LN K p+1 2 p+1 G (τ, (p, L, U)) G = πj (µj, τ ), (π , . . . , πp ) S . (S75) ∈ LN K ⇐⇒ N 1 +1 ∈ j=1 X

S34 Here (S75) refers to the probability simplex (S74) and so (τ, (p, L, U)) can also be represented in term of Sp+1. Furthermore in the GaussianLN empiricalK Bayes problem (1) with Z µ (µ, σ2), and G discretized as in (S75) we typically do not need to resort to numerical quadrature.∼ N To see this, note that for G (τ, (p, L, U)): ∈ LN K

2 L(G) = πjL( (µj, τ )), N j X 2 and for many functionals of interest there exist explicit expressions for L( (µj, τ )). A special case of the above result is marginalization: N

2 2 Z πj µj, τ + σ . ∼ N j X
Finally, we note that by a direct calculation it follows that (τ, (p, L, U)) provides a O((U L)/p) covering of (τ, [L, U]) in both total variationLN and WassersteinK distance. Inference− based on the discretizedLN class will thus be valid for linear functionals satisfying a) or b) of Proposition S17 as long as p is large enough.

E Computational aspects for AMARI E.1 Discretization Discretization of : Here the same considerations apply as in Supplement D.2. G Discretization of Z: Computing (28) for a continuous likelihood, such as (µ, σ2), re- M M 2 ¯M N M quires numerical integration, e.g., to compute (f δ (z) f δ (z)) fn (z) dλ (z). In G1 G this section we explain how to conduct the discretization− rigorously.−1 Our goal is to allow  an arbitrary discretization (that may be coarse)R by accounting for the discretization in the calculation of the worst-case bias. In particular, even if the discretization is too coarse, our intervals will have correct coverage, although they may be overly wide. Fix M > 0 as in (27), and consider the grid,S29

= n = M = t1,n < t2,n < . . . < tKn 1,n = M , (S76) I I {− − } where the grid may depend on n. Also let us define t ,n = , tK ,n = + and Ik,n = 0 −∞ n ∞ [tk 1,n, tk,n) for k 1,...,Kn and Kn N. In analogy to (27), we define: − ∈ { } ∈ Kn ZI := k1(Zi Ik,n) 1,...,Kn . (S77) i ∈ ∈ { } k X=1 Also let λI the counting measure on 1,...,Kn and analogously to the development af- { } ter (27), define fGI to be the marginal density of ZiI with respect to λI , i.e., fGI (k) = fG(z)1(z Ik,n)dλ(z) for k 1,...,Kn and also define f¯I (k) analogously. ∈ ∈ { } R S29In practice, to guarantee shorter confidence intervals, the grid should be made as dense as possible, subject to computational constraints, and should also become denser as n increases. In the Gaussian problem for example, we discretize [−M,M] as a dense equidistant grid, with step size  σ. While we do not pursue this further here, existing theory for discretization in statistical inverse problems [Johnstone and Silverman, 1991] suggests that even relatively coarse griding suffices to maintain the minimax risk. The result of Theorem2 allows for an arbitrary discretization of [ −M,M] (by applying the results of that theorem to the ‘discretized’ likelihood) that can also change with n.

S35 In view of the above considerations, the modulus problem (28) takes on the following discrete form:

Kn (f (k) f (k))2 GI1 GI−1 2 sup L(G1) L(G 1) G1,G 1 n, n − δ . (S78) − − | − ∈ G · f¯ (k) ≤ ( k I ) X=1 Below we discuss the solution of this discretized form of the modulus.

E.2 Computing the affine minimax estimator E.2.1 Direct form of the modulus problem To solve (S78) with modern convex optimization solvers, it is convenient to represent it as follows

sup L(G1) L(G 1) (S79a) G1,G−1 − −

Km 2 (fGI (k) fGI (k)) δ s.t. 1 − −1 (S79b) v f¯ (k) ≤ √n uk=1 I uX t G1,G 1 (S79c) − ∈ G G1,G 1 are F-localized, i.e., FG1 ,FG−1 n (S79d) − ∈ F We make the following observations:

– The optimization variables are G1,G2 . With suitably discretized, as in Sec- tion D.2, these have finite-dimensional representations.∈ G G The choices of (discretized) considered in this work may be represented using a finite number of linear constraints.G – The objective (S79a) is linear in the optimization variables.

– The maps G` fGI` (k) are linear in G` and so (S79b) corresponds to a second order cone constraint.7→ – The localization constraints in (S79d) may be implemented as a finite number of constraints on G1, resp. G 1. These can be either linear (DKW and Gauss-F - localizations) or quadratic (χ−2-F -localization) in the optimization variables. To see

these two claims, first note that the maps G` FG` are also linear. Furthermore, in- specting the proof of Theorem2, we see that (327→) only needs to hold for the discretized distributions, F I ,F I ,F I , F¯I (with triangular array asymptotics accounting for G1 G−1 G I changing with n). As a consequence of the above observations, the discretized modulus problem may be repre- sented as a finite-dimensional second order conic program (SOCP) [Boyd and Vandenberghe, 2004], which in turn is efficiently solvable by modern convex optimization solvers such as Mosek [ApS, 2020] or Hypatia [Coey et al., 2020]. In our numerical examples we use Mosek; our implementation can also use Hypatia.

E.2.2 Superdifferential of the modulus problem and duality

Evaluation of the estimator (30) requires access to ωn0 (δ), an element of the superdifferen- tial of the modulus of continuity ωn(δ) at δ = δn. An element ω0 (δ) √n may be directly n ·

S36 extracted upon solving (S79) as the dual variable associated to the constraint (S79b), pro- vided that strong duality holds, and the primal and dual optima are attained. Many convex solvers, including Mosek and Hypatia, return the dual variables. Argument sketch. Define the Lagrangian of (S79) for λ 0: ≥ 1/2 Km 2 (fGI1 (k) fGI−1 (k)) δ (G1,G 1, λ; δ) = L(G1) L(G 1) λ − . L − − − −  f¯ (k) − √n k I ! X=1   Note that we parametrize the optimization problem and also the Lagrangian by δ. For any feasible G1,G 1 and λ 0 − ≥ (G1,G 1, λ; δ) L(G1) L(G 1). (S80) L − ≥ − − δ δ δ Let G1,G 1 be primal optimal solutions to (S79) and let λ be the optimal dual variable, then [Boyd− and Vandenberghe, 2004, Chapter 5.5.2]:

δ δ δ δ δ δ ωn(δ) = L(G1) L(G 1) = (G1,G 1, λ ; δ) = sup (G1,G 1, λ ; δ) (S81) − − L − G1,G−1 L −

δ+∆δ δ+∆δ Now fix δ > 0 and take any ∆δ such that ∆δ > δ. Also let G1 ,G 1 solutions to (S79) at δ + ∆δ. Putting all results together − −

δ+∆δ δ+∆δ ωn(δ + ∆δ) = L(G1 ) L(G 1 ) − − (S80) δ+∆δ δ+∆δ δ (G1 ,G 1 , λ ; δ + ∆δ) ≤ L − 2 1/2 Km (f δ+∆δ (k) f δ+∆δ (k)) GI GI δ+∆δ δ+∆δ δ 1 − −1 δ + ∆δ = L(G1 ) L(G 1 ) λ − − −  f¯ (k)  − √n  k I X=1     2 1/2  Km (f δ+∆δ (k) f δ+∆δ (k)) GI GI δ δ+∆δ δ+∆δ δ 1 − −1 δ λ ∆δ = L(G1 ) L(G 1 ) λ + − − −  f¯ (k)  − √n √n k=1 I  X  δ+∆δ δ+∆δ δ δ   = (G1 ,G 1 , λ ; δ) + (λ /√n)∆δ L − (S81) δ ωn(δ) + (λ /√n)∆δ ≤ δ δ Thus λ /√n ∂ωn(δ), that is, λ /√n is an element of the superdifferential of ωn( ) at δ. ∈ ·

E.3 Bias-aware Normal confidence interval Recall that for constructing the confidence intervals from (15), we need to calculate (with W (0, 1)): ∼ N 1/2 tα(B,V ) = inf t : P B + V W t 1 α for all b B ≤ ≥ − | | ≤ n h i o This is the same as:

1/2 1/2 tα(B,V ) = V inf t : P B/V + W t 1 α for all b B ≤ ≥ − | | ≤ n h i o

S37 It is not directly obvious how to calculate this, however here we will argue that the calcula- tion reduces to calculating the quantile of the absolute value of a Normal distribution (and hence can be efficiently computed); this expression is also given in Armstrong and Koles´ar [2018]: Proposition S18. Under the above setting it holds that:

1/2 1/2 tα(B,V ) = V cvα(B/V )

Here cvα(u) is the 1 α quantile of the absolute value of a (u, 1) distribution. − N Proof. For convenience of notation and without any loss of generality, let us assume V = 1. First let us note that b + W =D b + W for any b, hence: | | |− | tα(B,V ) = inf t : P [ b + W t] 1 α for all 0 b B { | | ≤ ≥ − ≤ ≤ } Next, observe that for b = B, B + W (B, 1) , and thus by definition: | | ∼ |N | inf t : P [ B + W t] 1 α = cvα(B) { | | ≤ ≥ − } We now just need to check what happens for 0 b B, and indeed we will need some stochastic dominance argument. It suffices to argue≤ that≤ for any fixed t > 0 and 0 b B: ≤ ≤ P [ B + W t] P [ b + W t] | | ≤ ≤ | | ≤ Thus, if we let h(b) = P [ b + W t] it suffices to show h0(b) 0 for all b 0, so that it is decreasing. A direct calculation| | ≤ yields (with Φ, ϕ the standard≤ Normal CDF≥ and pdf respectively): h(b) = Φ(t b) Φ( t b). − − − − So: h0(b) = ϕ(t b) + ϕ( t b) 0. − − − − ≤ The last inequality holds since t b t b for t, b 0. | − | ≤ |− − | ≥ F Exponential family (logspline) G-modeling

In this section we summarize the empirical Bayes approach introduced by Efron[2016] and Narasimhan and Efron[2020]. The key idea is to specify as a flexible exponential fam- H ily of effect size distributions with natural parameters α = (α1, . . . , αp), sufficient statistic Q(µ): R Rp and base measure H. Concretely, distributions G are parametrized by 7→ ∈ H α with Radon-Nikodym derivative gα(µ) = dG/dH(µ) defined as

gα(µ) = exp(Q(µ)>α A(α)). (S82) −

A(α) is such that gα(µ)dH(µ) = 1. It is worth pointing out, that in contrast to our setting, is not a convex class. α is estimated byα ˆ, the maximizer of the log (marginal) likelihoodH`(α) in modelR (1):

n

`(α) = log p(Zi µ)gα(µ)dH(µ) . (S83) | i=1 X Z 

S38 Efron[2016] further recommends to maximize the penalized likelihood `(α) s(α) instead, − where s(α) = c0 α 2 for some c0 > 0. The empirical Bayes quantity θG(z) = EG h(µi) Zi = z || || ˆ can then be estimated by the plug-in estimator θ(z) = E h(µi) Zi = z , where G is the Gb   prior with dH-density gα( ). Standard delta method calculations and maximum-likelihood ˆ ·   asymptotics can then be used to estimate standard errors and correct bias due tob the pe- nalization (but not due to misspecification). Efron[2016] demonstrates that even under misspecification, such a family of effect size distributions leads to practical (albeit biased) empirical Bayes point estimates. We use the following parameters for the method in our numerical results. – We take the base measure H to be the uniform measure U[ 4, 4], the sufficient statistic to be a natural spline with 5 degrees of freedom with equidistant− knots on the above grid and c0 = 0.001. – In Figure7 we use the same settings as above but vary the degrees of freedom from 2 to 12.

G Sensitivity analysis for the prostate dataset

In this Supplement, we explore some of the issues raised in Section 8.1 by revisiting the prostate data analysis of Section 5.2. There we we posited that G = (τ 2, [ 3.3]) (34) with τ = 0.25. The first question we ask, is whether a goodness-of-fit∈ G LN test can− guide the 2 choice of τ in a data-driven way. To test H0(τ): G (τ , [ 3.3]), we use the Split Likelihood-Ratio (SLR) test of Wasserman, Ramdas, and∈ LN Balakrishnan− [2020], of which we provide a brief explanation. First, we randomly split our observations into two folds, I0 and I1. Then, let G1 be the nonparametric maximum likelihood estimator of G in the class (R) using Zi, i I1 and P ∈ G the nonparametric maximum likelihood estimator of G in the class (τ 2, [ 3b.3]) using 0 LN − Zi, i I . The Split Likelihood-Ratio (SLR) is defined as: ∈ 0 b f (Zi) SLR = Gb1 . f (Zi) i I0 Gb0 Y∈ Wasserman, Ramdas, and Balakrishnan[2020] prove that the test SLR > 1/α is a finite- { } sample valid level α test for the null hypothesis H0(τ). The second column of Table S1 shows the SLR for τ 0.02, 0.1, 0.25, 0.5, 55 . The SLR test at level α = 0.05 only rejects the model with τ = 0∈.55, { and the SLR statistic} becomes smaller as τ decreases. Next, we consider inference for the local false sign rate PG µ 0 Z = 2 and poste- ≥ rior mean EG µ Z = 2 at Z = 2 using the Gauss-F -Localization approach. The last two   columns of Table S1 show the confidence intervals for each choice of τ (that was not re-   jected by the SLR test). We observe that as τ becomes smaller, the confidence intervals for PG µ 0 Z = 2 become substantially wider, while the confidence intervals for the pos- terior≥ mean are less sensitive. One way of determining how pessimistic a given choice of τ   may be, is to inspect the worst-case priors in (7). Figure S1 shows these for the local false sign rate PG µ 0 Z = 2 . ≥  

S39 τ = 0.02 τ = 0.1 τ = 0.25 τ = 0.5 1.5 0.8 15 3 0.6 1.0 ) ) ) ) 2 µ µ µ 10 µ 0.4 ( ( ( ( g g g g 0.5 5 1 0.2

0 0 0.0 0.0 3 2 1 0 1 2 3 3 2 1 0 1 2 3 3 2 1 0 1 2 3 3 2 1 0 1 2 3 − − − µ − − − µ − − − µ − − − µ

Figure S1: Lebesgue density of worst-case priors in the Gauss-F -Localization ap- proach (7) for the local false sign rate PG µ 0 Z = 2 in the Prostate dataset. Each panel corresponds to a different specification of the≥ prior class , namely = (τ 2, [ 3.3])   G G LN − where τ varies across panels.

τ SLR Goodness of fit rejected CI for PG µ 0 Z = 2 CI for EG µ Z = 2 ≥ 0.02 0.0030 X 0.1859 – 0.9996  0.0196 – 0.7156 

0.10 0.0032 X 0.4491 – 0.9746 0.0302 – 0.7127 0.25 0.0042 X 0.6396 – 0.8905 0.0922 – 0.6960 0.50 1.8847 X 0.8043 – 0.8325 0.3834 – 0.5291 0.55 74.106 X

Table S1: Goodness-of-fit testing and sensitivity analysis for the Prostate data.

S40