Statistical Considerations for the Design and Analysis of the Elispot Assay in HIV-1 Vaccine Trials

Journal of Immunological Methods 288 (2004) 19–34 www.elsevier.com/locate/jim Research Paper Statistical considerations for the design and analysis of the ELISpot assay in HIV-1 vaccine trials

Michael G. Hudgensa,*, Steven G. Selfa, Ya-Lin Chiua, Nina D. Russellb, Helen Hortonb, M. Juliana McElrathb,c

a Public Health Sciences Division, Fred Hutchinson Cancer Research Center, 1100 Fairview Ave. N, NW-500, PO Box 19024, Seattle 98109-1024, USA b Clinical Research Division, Fred Hutchinson Cancer Research Center, USA c Department of Medicine, University of Washington, Seattle, USA

Received 11 September 2003; received in revised form 23 December 2003; accepted 24 January 2004

Available online 5 March 2004

Abstract

Effector T lymphocyte responses are considered critical for controlling human immunodeficiency virus type-1 (HIV-1) infection. The enzyme-linked immunospot (ELISpot) assay has emerged as a primary means of assessing HIV-specific T cell responses, and the development of objective methods that distinguish positive and negative ELISpot responses while properly controlling the rate of false positives is critical. In this paper, we consider several statistical methods that are helpful in defining such a positive criterion. Simulation results under a variety of scenarios suggest that a permutation-based criterion using a resampling adjustment for multiple comparisons yields the desired false positive rate while remaining competitive with other potential criteria in terms of sensitivity. These results also provide guidance on the effect of the number of experimental and negative control replicate wells on assay sensitivity. Application of different potential positive criteria using ELISpot assay results from IFN-g-secreting T cells of HIV-1 seropositive and seronegative donors confirmed several of the results obtained under simulation. Our findings support the application of statistically-based positive criteria such as the permutation-based resampling approach in assessing HIV vaccine-induced T cell responses. Moreover, the proposed methods have potential utility in related HIV immunopathogenesis studies and in non-HIV clinical vaccine trials. D 2004 Elsevier B.V. All rights reserved.

Keywords: AIDS; Cytokines; Human; T lymphocytes; Vaccination

Abbreviations: CTL, cytolytic T lymphocyte; ELISpot, enzyme-linked immunospot; FDR, false discovery rate; FWE, family-wise error rate; HIV-1, human immunodeficiency virus type-1; HVTN, HIV Vaccine Trials Network; MCPs, multiple comparison procedures; PBMC, peripheral blood mononuclear cells; PHA, phytohemagglutinin; SFCs, spot forming cells; SIV, simian immunodeficiency virus. * Corresponding author. Department of Biostatistics, School of Public Health, University of North Carolina at Chapel Hill 3107-E McGavran-Greenberg Hall Chapel Hill, NC 27599. Tel.: +1-919-966-8872; fax: +1-919-966-3804. E-mail address: [email protected] (M.G. Hudgens).

1. Introduction Traditionally in phase I and II HIV-1 vaccine studies, CD8+ T cell responses have been identified Induction of strong and durable T lymphocytes as cytolytic T lymphocytes (CTLs) utilizing 51Cr capable of recognizing HIV-1-infected cells has be- release cytotoxicity assays (Evans et al., 1999; AIDS come an important immunogenicity outcome in vac- Vaccine Evaluation Group 022 Protocol Team, 2001; cine trials, given the increasing evidence that HIV-1- Belshe et al., 2001). Unfortunately, this assay requires specific cell-mediated immune responses are at least fresh peripheral blood mononuclear cells (PBMC), is partially protective against disease in macaques chal- extremely labor intensive, and does not afford a lenged with SHIV (Barouch et al., 2000; Amara et al., reliable quantitative response. As a result, many 2001), and against HIV-1 disease (Koup et al., 1994; researchers have turned to alternative techniques for Borrow et al., 1994; Rowland-Jones et al., 1995; measuring ex vivo viral-specific CD8+ and CD4+ T Mazzoli et al., 1997; Musey et al., 1997; Ogg et al., cells in cryopreserved PBMC such as flow cytometric 1998; Goh et al., 1999) in humans. With a consider- analysis of intracellular cytokines, MHC-peptide tet- able number of vaccine candidates being tested in ramer staining, and IFN-g ELISpot. Here we consider non-human primate models and in phase I and II how statistical methodology can be employed in clinical trials, it is imperative to utilize a standardized, defining an ELISpot positive criterion. While the sensitive assay to measure HIV-1-specific T cell ELISpot assay does provide quantitative information, responses. The need to design and conduct phase III assessing the ability to induce a detectable qualitative trials to establish correlates of immunity provides response is a natural first step in initial evaluation of a further motivation for the development of a sensitive, candidate vaccine. quantitative, and reproducible assay. Based on principles similar to the ELISA assay, the Both CD4+ and CD8+ virus-specific T cells are ELISpot allows enumeration of both CD4+ and CD8+ likely to contribute to control of viral replication. T cells that secrete cytokines in response to an Anti-viral activities of HIV-1-specific CD8+ T cells antigenic stimulus at the single cell level. Using 96- include cytolysis of infected cells and release of well plates coated with anti-cytokine antibodies and chemokines and cytokines, such as IFN-g. HIV-spe- an appropriate detection system, each cytokine-pro- cific CD4+ T cells are likely to be critical for ducing cell can be identified by an individual spot maintenance of effective virus-specific CD8 T cell within a well. Thus, the raw form of the data arising responses (Matloubian et al., 1994; Zajac et al., 1998). from the ELISpot assay is the number of spot forming While CD8+ T cells constitute the majority of HIV- cells (SFCs) for each experimental and control (neg- specific T cell responses in HIV-infected individuals ative and positive) well, each of which are often and SIV-infected rhesus macaques (Vogel et al., 2002; performed in duplicate or triplicate. An assay is Cao et al., 2003), the contributions of virus-specific subsequently categorized as positive or negative CD4+ versus CD8+ T cells elicited by vaccines are depending on whether or not the number of SFCs in likely to be dependent on the type of vaccine admin- the experimental wells is significantly greater than in istered. Protein subunit vaccines are more likely to the control wells. induce CD4+ T cell responses since they are pro- Current methods for making a positive determina- cessed and presented through the MHC class II tion often rely on reasonable but ad hoc approaches. processing pathway. Vaccine modalities that are based For example, responses to a particular antigen stimu- on viral vectors are, however, likely to induce CD8+ lation might be considered positive if on average there and CD4+ T cell responses since they will be pro- are (i) at least 10 more SFCs/2 105 PBMC in the cessed and presented through both the MHC class I experimental wells than control wells, and (ii) at least and II pathways. Our main interests lie in measure- double the number of SFCs in the experimental wells ment of HIV-specific CD8+ T cell responses since than in the control wells (Larsson et al., 1999; Sub- these have been shown to directly contribute to klewe et al., 1999; Alter et al., 2002). Similar criteria control of viral replication in rhesus macaque models appear throughout the immunology literature (Lalvani of simian immunodeficiency virus (SIV) infection (Jin et al., 1997; Kaul et al., 2000; Larsson et al., 2000; et al., 1999). Loing et al., 2000; Dhodapkar et al., 2000). In what M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34 21 follows, we refer to the criterion based on (i) and (ii) 2.2. ELISpot assay above as the empirical method. While the empirical method is reasonable, it is not clear how the false An IFN-g ELISpot assay was used to detect HIV-1- positive rate is being controlled. For example, it is specific IFN-g producing T cells from cryopreserved not known whether the criterion is too liberal or PBMC as previously described (Russell et al., 2003). conservative as the background frequencies from the Ninety-six-well hydrophobic polyvinylidene difluor- control wells of the assay vary, or if the strictness of the ide membrane-bottomed plates (Millipore) were coated criterion varies according to the assay format. Further- with anti-IFN-g mAb overnight at 4 jC. After washing, more, the empirical criterion gives no formal guidance plates were blocked at room temperature for 1 h. with regards to using results from multiple antigenic Cryopreserved PBMC were thawed and incubated stimulations to assess an overall response. The goal of overnight at 37 jC with 5% CO2. T cell subsets were this study was to employ statistical methods in devel- purified using anti-CD4+ Ab-coated immunomagnetic oping an alternative positive criterion that addresses beads (Microbeads, Miltenyi Biotec) according to the these issues. Consideration is given to optimizing manufacturer’s instructions. Cells (either whole sensitivity while controlling the false positive rate of PBMC, CD4+, or CD4-depleted) were then added to the assay and accommodating multiple comparisons. pre-coated plates at concentrations of 2 105 or The effect of assay format (i.e., the number of replicate 1 105 cells/well. Two hundred twelve HIV-1 Env experimental and control wells) is also considered. (MN strain), 248 Pol (HXB2), and 49 Nef (BRU) 15- mer peptides overlapping by 11 amino acids were synthesized by SynPep (Dublin, California) and 122 2. Materials and methods HIV-1 Gag (HXB2) 15-mer peptides overlapping by 11 amino acids were provided by the National Institutes of 2.1. Study population Health AIDS Research and Reference Reagent Pro- gram (Bethesda, MD). The peptides were used for anti- Assays were performed using cryopreserved genic stimulation in pools of 25 or 50 peptides, each at a PBMC from 36 HIV-1 seropositive and 45 HIV-1 final concentration of 1–2 Ag /ml. Negative control seronegative individuals. Of the seropositive popula- wells contained media alone (no peptide). Cells stimu- tion, there were 10 long-term non-progressors, 8 anti- lated with phytohemagglutinin (PHA) (1 Ag/ml) served retroviral treated chronically infected patients, 12 as a positive control. After antigenic stimulation over- untreated chronically infected patients, and 6 untreat- night (16-20 h), a secondary biotinylated anti-IFN- ed primary infected patients (see Russell et al., 2003 gmAb was added, followed by avidin biotinylated for details on a majority of these patients). Of the enzyme complex (VectastainABC Elite Kit, PK-6100, seronegative population, 35 were placebo recipients in Vector) and subsequently, AEC peroxidase substrate a recent HIV Vaccine Trials Network vaccine trial (Vectastain). Plates were developed for approximately 7 (HVTN 203), the results of which are forthcoming. min and colored SFCs were counted using an automated These volunteers received multiple doses of a placebo ELISpot reader (Immunospot, CellularTechnology). combination which included both an aluminum hy- droxide adjuvant (AIDSVAX placebo, VaxGen, Bris- 2.3. Univariate criteria bane, CA) and a mixture of 10 mM Tris–HCl buffer pH 9.0, virus stabilizer, and freeze drying medium In this section we discuss several potential positive (PLACEBO-ALVAC, Aventis Pasteur, Lyon, France). criteria. First we introduce some notation. Suppose The remaining 10 seronegative individuals were pre- that we have mE replicate wells for cells stimulated viously enrolled in an AIDS Vaccine Evaluation with a particular peptide pool and mC common repli- Group protocol (AVEG 201); PBMC from pretreat- cate wells containing cells serving as a negative ment time points were used for these experiments. All control, all at the same concentration (often 200,000 C C study participants provided written consent, and all cells/well). Let Y 1 ,..., YmCbe the number of SFCs/well E E aspects of the study were approved by the human for the mC control wells, and Y1 ,..., Y mE be the subjects institutional review board. number of SFCs/well for the mE peptide pool wells. 22 M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34

Let the distribution governing the number of SFCs/ the more general null (Eq. (2)) using a Satterthwaite control well be denoted by FC with mean lC. Similarly approximation (Rosner, 1995). The t-test assumes that define FE and lE for the peptide pool replicate wells. the underlying distribution is approximately normal, a E Let Y¯ E = SiYi /mE be the per-well average for a partic- potentially dubious assumption for count data. Thus, a ular peptide pool and let Y¯ C be the corresponding log transformation might be considered prior to average for the control wells. performing such a test. Alternatively, the Wilcoxon The empirical criterion described above for a ranksum (or Mann–Whitney) test, which is the non- positive response is then given by parametric analogue of the two sample t-test, has been employed in a similar fashion (McCutcheon et al., ¯ ¯ z ¯ z ¯ YE YC 10 and YE 2YC: ð1Þ 1997). That is, experiments satisfying Eq. (1) are assumed to provide sufficient evidence to conclude that the 2.3.2. Binomial peptide pool wells have significantly more SFCs/well An alternative approach is to employ standard than the control wells. Criterion (1) is typically used categorical data methods by considering whether the when the experiment is done at 200,000 cells/well proportion of total spots attributable to the experi- (Russell et al., 2003). Should a different concentration mental wells is significantly greater than to be be employed, Eq. (1) would be modified accordingly. expected given the assay format (Flanagan et al., For example, at 106 cells/well, we would require 1999; Self et al., 2001). Since the ELISpot assay produces count data, one might assume that Y C,..., Y¯ Y¯ z 50 and Y¯ z 2Y¯ . 1 E C E C C f l E ... E f l YmC Poisson( C)andY1 , , YmE Poisson( E). Alternatively, statistically based positive criteria E E C C E C might be employed by testing the null hypothesis Letting Y = Ri Yi , Y = Ri Yi , and N = Y + Y ,it is easy to show that given N, Y E is distributed z H : lC lE; ð2Þ Binomial(N,p)wherep = mE/(mE + mC). Note that p = 1/2 for equal number of replicate wells for the versus the alternative HV: l < l ,or C E experiment and the control. Because the conditional distribution of Y E is binomial, an exact one-sided test H : FCðyÞVFEðyÞ for all y; ð3Þ for the null hypothesis given by Eq. (3) can be versus the alternative HV: FC( y) z FE( y) for all y with employed. Alternatively, a normal approximation can at least one strict inequality. To avoid statistically be applied wherein the quantity significant but biologically or clinically irrelevant E results, one might choose more conservative null Y pN Zbin ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ; hypotheses. For example, we might require statistical pð1 pÞN evidence that the mean SFCs/well for stimulated wells is at least twice background, i.e., H:2lC z lE. Below is treated as a standard normal deviate. we list several statistically based positive criteria, some of which are currently employed in the ELISpot 2.3.3. Severini literature and others of which are new. While the exact binomial test discussed above is optimal under the Poisson distribution (Lehmann, 2.3.1. t and Wilcoxon ranksum 1959), it relies on the simple but potentially unreal- Two statistically based criteria previously employed istic assumption that the number of SFCs/well is in the analysis of ELISpot data are the t and Wilcoxon governed by a Poisson distribution. It is not uncom- ranksum tests. The two-sample (or unpaired) t-test has mon, however, for count data to fail to satisfy the been applied to determine if the difference in the mean assumption of the Poisson distribution that the var- number of SFCs in the experimental wells is signifi- iance equals the mean. In general, the existence of cantly greater than in the control wells (Herr et al., variability in the data that exceeds the nominal 1996, 1997, 2000). In particular, a one-sided two- variance expected under the assumed model is called sample t-test can be employed to test the null hypoth- overdispersion. For count data, heterogeneity in the esis (Eq. (3)) under the assumption of equal variance or average event rates across the population under study M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34 23 is often a cause of overdispersion. In the presence of by ELISpot to at least one HIV-1 peptide pool (e.g., overdispersion, an alternative positive criterion is HIV Vaccine Trials Network protocol 203(NIAID needed since the Poisson assumption of the statistical Office of Communications and Public Liaison, 2002; positive criterion does not hold, which in turn can Cohen, 2002)). In this case, it is desirable to control the result in drawing incorrect inferences. Recently, family-wise error rate (FWE), i.e., the probability of Severini (1999) proposed an extension of the bino- having any false positives among the multiple hypoth- mial test to be applied in case of overdispersion. eses being tested (Costigan, 1998). On the other hand, Severini’s test entails a slight modification to the if the setting is of an exploratory nature (e.g., epitope normal approximationpffiffiffiffiffiffiffiffiffiffiffi to the binomial test wherein mapping), it may be more appropriate to control the Zser ¼ Zbin= c 1 is treated as a standard normal false discovery rate (FDR), i.e., the expected propor- deviate with c equal to the ratio of the sample tion of rejected hypotheses that are true (Benjamini and variance to the sample mean of combined counts Hochberg, 1995). Methods that control the FDR can be C C E E Y1 ,. . ., YmC, Y1 ,. . ., YmE. more powerful than those controlling the FWE, but are only valid when the overall conclusion of a set of 2.3.4. Resampling hypothesis tests is not erroneous, even if some of the Bootstrap or permutation-based methods (Efron individual null hypotheses are falsely rejected. Moti- and Tibshirani, 1993) offer two alternative positive vated by the use of the ELISpot assay as an endpoint in criteria for comparing the distributions of SFCs/well HIV-1 vaccine trials, the focus of this paper concerns without making the parametric assumptions of the controlling the FWE. For further discussion regarding two-sample t-test or the exact binomial test. The different types of FWE (e.g., strong or weak), see permutation test corresponds to testing hypothesis Westfall and Young (1993). (3) whereas the bootstrap approach may test either Among multiple comparison procedures (MCPs) hypothesis (2) or (3) depending on the algorithm that control the FWE, one seeks to maximize sensitivity employed (Efron and Tibshirani, 1993). The permu- or power, i.e., the probability of correctly rejecting false tation and Wilcoxon ranksum tests are closely related, hypotheses. In the multiple comparisons setting, there with the main difference being that the permutation are several definitions of power, two of which we will test permutes the actual counts, whereas the ranksum consider. Complete power is the probability of rejecting test permutes the ranks of the counts. all false hypotheses; minimal power is the probability of rejecting at least one false hypothesis. For the 2.4. Multiple comparisons clinical trials setting motivating this work, minimal power is important if the primary endpoint is defined as Often in application of the ELISpot assay several having a positive response to at least one peptide pool. peptides or pools of peptides are used in an effort to On the other hand, if the focus is on the breadth of determine the breadth of response. Therefore, when response (i.e., the number of positive responses across determining whether a subject has a positive response, all peptide pools), then complete power might be of a multiple comparison adjustment may be appropriate greater interest. to control the false positive rate (Huang et al., 2000; Given that a multiple comparisons adjustment is Nagorsen et al., 2000). Below we give a brief outline needed to maintain control of the FWE, there are of the main issues of multiple comparisons germane to several methods to consider (Hochberg and Tamhane, the analysis of data arising from the ELISpot assay in 1987; Westfall and Young, 1993; Hsu, 1996; D’Agos- a clinical trials setting. tino and Russell, 1998). By comparing different sets of In general, the approach employed for a multiple experimental wells for each distinct peptide pool to the comparisons problem depends on the scientific goals same set of control wells, the ELISpot assay is an of the study or trial. In clinical trials it is often excellent example of the many-to-one problem (Dun- important to control the false positive rate at the nett, 1955; Hochberg and Tamhane, 1987; Westfall and participant level. For example, consider a Phase II Young, 1993; Hsu, 1996; Hochberg and Westfall, HIV-1 vaccine trial where the primary endpoint is the 2000). The challenge in choosing the appropriate proportion of participants who have a positive response MCP in the many-to-one problem arises from the 24 M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34 dependence among the multiple test statistics induced (iv) underlying distributions governing the number of by using the same control for each comparison. Bon- SFCs/well. ferroni-type (Holm, 1979), normal-based (Dunnett, 1955; Hsu, 1996), and resampling (Westfall and Young, To address (i), we considered values of K = 1 (i.e., no 1993; Westfall et al., 1999) MCPs have been advocated multiple comparisons), 10, and 25. To address (ii), for either general situations entailing dependent test several combinations of mC and mE were considered. statistics or the many-to-one problem in particular. Duplicate (mC =2, mE = 2) and triplicate (mC =3, mE = 3) formats are common combinations found in 2.5. Simulation study the ELISpot literature; however, statistical theory suggests that choosing mC>mE may be more optimal for A set of simulation studies were employed to assess many-to-one problems such that (mC =6, mE =2), the performance of the different univariate positive (mC = 10, mE = 2), and (mC =6,mE = 3) formats are also criteria and multiple comparison procedures discussed considered. To address (iii), the mean number of SFCs/ in the previous sections. The goal was to find at least control well was varied over four values: lC = 3, 10, 30, one MCP that maintains the FWE for various 100. To address (iv), count data were generated by assuming that the counts follow a mixture model in (i) numbers of distinct peptide pools (K), which the mean of the Poisson distribution is itself a (ii) assay formats (i.e., mC and mE), random variable with a gamma distribution having (iii) levels of background (i.e., mean number of SFCs/ mean l and variance (/ 1)l. The resulting counts negative control well, lC), and are then negative binomial random variables with mean

Fig. 1. Histogram of spot forming cells (SFCs) per well from simulated control wells under each distributional assumption. The mean number of SFCs is given by l. The ratio of the variance to the mean is denoted by /. M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34 25

Fig. 2. Family wise error rate (FWE) for various univariate positive criteria (as described in Section 2.3) based on 10,000 simulated data sets for assay format (mC, mE)=(6, 2). The shaded bars correspond to settings where the observed false positive rate was less than or equal to the nominal a = 0.05 level. The mean number of SFCs for the negative control wells is given by lC (muc). The ratio of the variance to the mean is denoted by / (phi). l and variance /l. For / = 1 the mean of the Poisson distribution is fixed at l such that the resulting counts Table 1 are simply Poisson with mean and variance l. Simu- Multiple comparison procedures considered lations were done for a variety of levels of overdisper- Name Multiple comparison sion: none (/ = 1), mild (/ = 2), moderate (/ = 10), and procedure severe (/ = 25). Illustrative histograms of simulated Binomial-Holm Normal approximation to control wells are given in Fig. 1. Looking across the Binomial with Holm (1979) multiplicity adjustment columns of Fig. 1 corresponds to an increase in the Severini-Holm Severini’s (1999) Binomial mean SFCs/well (l), while looking down the rows adjustment with Holm (1979) corresponds to an increase in overdispersion (/). For multiplicity adjustment each combination of (i)–(iv), 10,000 simulated data t-test-Holm Two-sample t-test with Holm sets were computed; for the resampling-based methods, (1979) multiplicity adjustment Empirical Empirical criterion (Eq. (1)) bootstrap and permutation samples of size 500 were Bootstrap-WY Westfall and Young (1993) drawn for each simulation. step-down bootstrap For each set of simulations, both control of the FWE Permutation-WY Westfall and Young (1993) and power (i.e., sensitivity) were considered. Alterna- step-down permutation tive hypotheses were simulated by assuming that the Dunnett Dunnett (1955) 26 M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34 number of SFCs in experimental wells can be decom- environment. On the other hand, many of the posed into the sum of background SFCs and ‘‘true’’ multiple comparison procedures discussed here are SFCs corresponding to IFN-g emitting T cells. The relatively recent developments. SAS (Cary, NC, background SFCs were assumed to have the same USA) offers two procedures, PROC GLM and distribution as SFCs in the negative control wells (with PROC MULTTEST, which allow implementation mean lC = 3, 10, 30, or 100). The antigen-recognizing of several of the tests described above. More cell populations were assumed to be governed by the recently, the Bioconductor project (www.bioconduc- same distribution, but with a mean of 20 SFCs/well. tor.org) has developed a multtest package for use in Under the Poisson background, this implies that the R (Ihaka and Gentleman, 1996) which implements SFCs in the experimental wells have a Poisson dis- many of the same multiple comparisons procedures tribution with mean lE = 23, 30, 50, or 120. For K =10 available in SAS. Other statistical packages offer and 25, both minimal and complete power were limited multiple comparison capabilities. All simu- considered. lations and analyses in this paper were implemented in SAS (Westfall et al., 1999). Sample code for 2.5.1. Implementation implementing these procedures in SAS is available Each of univariate positive criteria considered is upon request. Code for the permutation criterion is simple to implement in today’s statistical computing given in Appendix A.

Fig. 3. Minimal power for various multiple comparison procedures (as described in Section 2.4 and Table 1) based on 10,000 simulated data sets with K = 25 peptide pools and two false null hypotheses for assay format (mC, mE)=(6, 2). The shaded bars correspond to settings where the observed FWE was less than or equal to the nominal a = 0.05 level. The mean number of SFCs for the negative control wells is given by lC (muc). The ratio of the variance to the mean is denoted by / (phi). M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34 27

3. Results can give especially high FWEs in the presence of overdispersion (e.g., / = 25). Simulations under the 3.1. Simulation study alternative hypothesis (results not shown) demonstrate (i) superior power of the Binomial test for the Poisson 3.1.1. Univariate results simulations (which is consistent with statistical theory False positive rates for the various univariate posi- (Lehmann, 1959), and (ii) a general decrease in power tive criteria (i.e., for K = 1 peptide pool) are displayed with an increase in either overdispersion or background by the trellis plot (Venables and Ripley, 1999) given in for all criteria. The Permutation and Wilcoxon ranksum Fig. 2. The figure is composed of 16 panels tests gave nearly identical results (not shown) with corresponding to each combination of background regard to the FWE and power. level (lC) and dispersion (/). Within each panel, the FWE is given for each positive criterion. Results are 3.1.2. Multiple comparison results shown for the (mC, mE)=(6, 2) format only; similar Given the considerations of the section above, the trends were observed for other formats. Comparisons univariate simulation results, and implementation com- with the dotted vertical line depicting the nominal FWE plexity, only the MCPs given in Table 1 were consi- of 0.05 show that only the Permutation test consistently dered in the simulation study. Fig. 3 shows the minimal controls the FWE. The Binomial and Empirical criteria power (i.e., the probability that at least one false null

Fig. 4. Minimal and complete power of the Permutation-WY criterion for K = 25 peptide pools with two false null hyphotheses based on 10,000 simulated data sets. The mean number of SFCs for the negative control wells is given by lC. The ratio of the variance to the mean is denoted by /. Assay format is denoted by (mC, mE) where mC is the number of replicate control wells and mE is the number of replicate experimental wells, by the five different line types. 28 M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34 hypothesis is rejected) when two of K = 25 null hy- sensitivity in the presence of high background (i.e., potheses are false for the (mC, mE)=(6, 2) format. The lC = 30, 100) and lacks specificity in the presence of shaded bars indicate scenarios where the observed overdispersion (i.e., / = 10, 25). Similar trends were FWE was less than 5%. Generally, most MCPs de- found for complete power, other assay formats, and monstrate excellent control of the FWE under the K = 10 peptide pools (results not shown). Poisson distribution. However, as in the univariate Given the results above, we also explored the effect simulations, only the Permutation-WY MCP consis- of assay format on sensitivity using the Permutation- tently controls the FWE under all scenarios considered. WY criterion (Fig. 4). Not surprisingly, power generally Additionally, the Permutation-WY approach performs increases with the total number of wells (mC + KmE). well with respect to power compared to other MCPs For example, in Fig. 4 the highest sensitivity is achieved which control the FWE. For example, consider the under the (mC, mE)=(6, 3) format, which requires 81 results in Fig. 3 for lC = 30 and / = 2 where only the total wells when there are K = 25 peptide pools; in Permutation-WY, Severini-Holm, Dunnett, and Empir- contrast, the duplicate format has the lowest sensitivity ical criteria have FWE less than or equal to 5%; of these and requires only 52 total wells. Note that reasonable four criteria in this scenario, the Permutation-WY is the gains in power can be achieved over the duplicate most powerful. In general, the Empirical criterion lacks format by simply increasing the number of control

Fig. 5. Left panel: Scatterplot and least-squares regression line of mean SFCs/well for each peptide pool versus negative control in seronegative individuals. Right panel: Scatterplot of between replicate variances versus mean for all peptive pool and negative control wells in seronegative and seropositive individuals. The three lines denote observed overdispersion (i.e., variance to mean ratio) of / = 1, 10, and 100. M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34 29 wells. For example, in the absence of overdispersion Table 2 a (i.e., upper left panel of Fig. 4) and a mean background Response rates by different positive criteria Study population of lC = 40, minimal power increases from 0.61 for (mC, mE)=(2, 2) to 0.75 for (mC, mE)=(6, 2) and to 0.82 for Seropositive Seronegative (mC, mE)=(10, 2). Given operational constraints of Effector cells CD4+ CD4 PBMC CD4+ CD4 performing each experimental peptide pool in triplicate, (n=36) (n=36) (n=24) (n=74) (n=74) increasing the number of control wells provides a Empirical 19 (53%) 35 (97%) 5 (21%) 2 (3%) 0 (0%) simple method for increasing assay sensitivity. a = 0.10 3.2. Application Permutation- 22 (61%) 35 (97%) 1 (4%) 10 (14%) 7 (9%) WY Bootstrap- 24 (67%) 35 (97%) 4 (17%) 18 (24%) 14 (19%) To illustrate application of the proposed statistical WY criteria, we considered ELISpot assay results from 45 t-test-Holm 23 (64%) 35 (97%) 2 (8%) 15 (20%) 9 (12%) HIV-1-seronegative and 36 HIV-1-seropositive volun- Dunnett 27 (75%) 35 (97%) 2 (8%) 22 (30%) 11 (15%) teers. Several of the seronegative volunteers were Severini- 25 (69%) 35 (97%) 4 (17%) 19 (26%) 7 (9%) Holm tested at more than one time point, but for purposes Binomial- 25 (69%) 35 (97%) 12 (50%) 10 (14%) 11 (15%) of this analysis, these results are treated as indepen- Holm dent observations. For each volunteer, up to three different cell populations (PBMC, CD4+ T cells, or a = 0.05 CD8+ T cells) may have been assayed. For nearly all Permutation- 20 (56%) 33 (92%) 1 (4%) 7 (9%) 4 (5%) WY assays, experimental wells were tested in duplicate Bootstrap- 23 (64%) 35 (97%) 4 (17%) 14 (19%) 9 (12%) (mE = 2) while six wells were used for each negative WY control well (mC = 6). The number of HIV-1 peptide t-test-Holm 23 (64%) 35 (97%) 2 (8%) 11 (15%) 6 (8%) pools tested ranged from K =3 to K = 26. Two dis- Dunnett 26 (72%) 35 (97%) 2 (8%) 12 (16%) 6 (8%) plays of the data are given in Fig. 5. Using data from Severini- 23 (64%) 35 (97%) 3 (13%) 19 (26%) 6 (8%) Holm seronegative individuals only, the left panel shows a Binomial- 25 (69%) 35 (97%) 11 (46%) 9 (12%) 9 (12%) strong linear association between the level of back- Holm ground and the mean SFCs/well for HIV-1 peptide pools. Since no response to these pools is expected a = 0.01 from seronegative samples, this plot indicates that the Permutation- 8 (22%) 22 (61%) 1 (4%) 1 (1%) 1 (1%) WY negative control wells are predictive of the back- Bootstrap- 19 (53%) 33 (92%) 1 (4%) 5 (7%) 6 (8%) ground in the experimental wells and thus supports WY the assumption that a positive criterion should entail t-test-Holm 16 (44%) 35 (97%) 1 (4%) 5 (7%) 3 (4%) contrasts between experimental and negative controls Dunnett 21 (58%) 35 (97%) 1 (4%) 8 (11%) 3 (4%) wells. The right panel of Fig. 5 indicates moderate Severini- 18 (50%) 35 (97%) 3 (13%) 18 (24%) 6 (8%) Holm levels of overdispersion in these data. Binomial- 23 (64%) 35 (97%) 10 (42%) 4 (5%) 6 (8%) Response rates, shown in Table 2, were calculated Holm separately for each cell population. All statistically a Number and percent of positive responses for 36 HIV- based positive criteria were employed using one-sided seropositive and 45 HIV-seronegative volunteers. A volunteer’s tests performed at 0.10, 0.05, and 0.01 significance response was categorized as positive if at least one peptide pool levels. For assays from the HIV-1 seronegative vol- stimulated a positive IFN-g secreting response. unteers, the Permutation-WY criterion yields results closest to the nominal false positive rate. In general, categorizes PBMC from 21% (5/24) of seronegative for the other criteria, the observed false positive rate individuals as positive. These results are generally substantially exceeds the nominal significance level consistent with the simulation study from the previous for at least one of the cell populations. For example, section when there is extra-Poisson variability. While the Empirical criterion yields reasonable false positive the Permutation-WY criterion gives rise to lower rates for CD4+ and CD4 cells, but incorrectly observed sensitivity in the HIV-1 seropositive individ- 30 M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34 uals (especially at a = 0.01), the higher rates given by determining the specificity of the assay based on the other criteria likely include a substantial portion of response rates in seronegative, unvaccinated indivi- false positives. Moreover, based on the results in Fig. duals. As a result, these empirical methods combine 4, one would expect a greater observed sensitivity both statistical and biological false positive rates. with the Permutation-WY criterion had the assay Employing such an approach either requires popula- employed triplicate experimental wells (i.e., mE = 3). tion-specific positivity criteria based on large control samples from each population or the development of a common, conservative cross-population criterion that 4. Discussion will necessarily lose sensitivity in those populations with low biological false positive rates. Defining Establishing a unified positive criterion for the population-specific criteria may not be operationally IFN-g ELISpot assay is critical to enabling valid or economically feasible, whereas a conservative comparisons across labs, studies, and populations. omnibus criterion will be a clear disadvantage when Our collective experience at the HIV Vaccine Trials assessing low frequency vaccine-induced responses. Network suggests that the distribution of background Ultimately, either approach will complicate interpre- SFCs/well can vary between labs and within labs, tation of results because one cannot clearly separate either over time or across protocols, and in individual the two types of false (i.e., not vaccine-induced) responses over multiple time points. Until the sources positive responses. of this variation are identified and controlled, it is In summary, we advocate employing a statisti- important that a positive criterion be robust to such cally-based positive criterion such as the Permuta- variability. To this end, we recommend adopting the tion approach especially combined with study permutation-based criterion using a resampling ad- designs that can distinguish positive responses that justment for multiple comparisons. Our simulation are vaccine-induced from those that are not. This results suggest that in the multiple comparisons set- approach has the potential to more accurately assess ting, this method consistently preserves the nominal T cell responses in diverse geographic populations false positive rate while remaining competitive with in the clinical trial setting. Its utility is likely to other methods in terms of sensitivity. These results extend to clinical evaluation of immunogens and also suggest that assay format is an important consid- immunotherapeutics for other pathogens of public eration given the significant between well heteroge- health importance. neity that can occur with the ELISpot assay. It is important to note that the statistical criteria presented here will not control the ‘‘biological’’ false Acknowledgements positive rate. In the context of a vaccine trial, such positive responses are not induced by vaccination but The authors are grateful for the financial support may be due to cross-reactivity with other pre-existing from NIH U01-AI46703, R201 AI-29168, AI-47086, immune effector cells. It is easy to imagine how the rate AI-46725, AI-48017. of such responses may vary considerably across populations, especially in the developing world where the breadth of the infectious disease burden is considerable. Appendix A Use of concurrent placebo control groups and/or base- line, pre-immunization assays remains an important In this section we provide a sketch of the algorithm component of trial design to clearly separate the vac- for computing permutation adjusted p-values. Sample cine-induced positive responses, the ‘‘biological’’ false SAS code is also provided. For further details, see positive responses and the false positive responses. Westfall and Young, 1993 and Westfall et al., 1999. The often employed empirical criteria offer an The single-step permutation algorithm is given by: alternative strategy for determining the critical value for positivity (Jung et al., 1999; Lewis et al., 2000; 1. For each of the K distinct peptide pools, compute an Dhodapkar et al., 2000).Thisapproachrelieson unadjusted p-value comparing the observed SFCs/ M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34 31

well from the peptide pool replicate wells to the 5. For each peptide pool, the permutation adjusted negative control replicate wells. These unadjusted p- p-value is then computed as the proportion of values can be computed using any univariate times that the unadjusted p-value is greater than statistically-based positive criterion, e.g., a two- or equal to the B minimum permutation p-values. sample t-test. 2. Permute the observed SFCs/well across all peptide A slightly more complicated (and powerful) step- pool and negative control wells. down permutation algorithm is given in Algorithm 2.8 3. Compute unadjusted p-values as in Step 1 using the of Westfall and Young (1993). The step-down permu- permuted data set from Step 2. Take the minimum tation algorithm, denoted Permutation-WY, was of these K unadjusted p-values. employed in computing the results for this paper. 4. Repeat Steps 2 and 3 B times. B z 10,000 is SAS code to compute single-step and step-down recommended. permutation p-values is given below: 32 M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34

References Armitage, P., Colton, T. (Eds.), Encyclopedia of Biostatistics. Wiley, New York, p. 421. AIDS Vaccine Evaluation Group 022 Protocol Team, 2001. Cellular D’Agostino, R.B., Russell, H.K., 1998. Multiple endpoints, P level and humoral immune responses to a canarypox vaccine contain- procedures. In: Armitage, P., Colton, T. (Eds.), Encyclopedia of ing human immunodeficiency virus type 1 env, gag, and pro in Biostatistics. Wiley, New York, p. 2762. combination with rgp120. Journal of Infectious Diseases 183, Dhodapkar, M.V., Young, J.W., Chapman, P.B., Cox, W.I., Fonte- 563. neau, J.F., Amigorena, S., Houghton, A.N., Steinman, R.M., Alter, G., Merchant, A., Tsoukas, C.M., Rouleau, D., LeBlanc, Bhardwaj, N., 2000. Paucity of functional T-cell memory to R.P., Cote, P., Baril, J.G., Thomas, R., Nguyen, V.K., melanoma antigens in healthy donors and melanoma patients. Sekaly, R.P., Routy, J.P., Bernard, N.F., 2002. Human immu- Clinical Cancer Research 6, 4831. nodeficiency virus (HIV)-specific effector CD8 T cell activity Dunnett, C.W., 1955. A multiple comparison procedure for com- in patients with primary HIV infection. Journal of Infectious paring several treatments with a control. Journal of the Ameri- Diseases 185, 755. can Statistical Association 50, 1096. Amara, R.R., Villinger, F., Altman, J.D., Lydy, S.L., O’Neil, S.P., Efron, B., Tibshirani, R.J., 1993. An Introduction to the Bootstrap Staprans, S.I., Montefiori, D.C., Xu, Y., Herndon, J.G., Wyatt, Chapman & Hall, New York. L.S., Candido, M.A., Kozyr, N.L., Earl, P.L., Smith, J.M., Ma, Evans, T.G., Keefer, M.C., Weinhold, K.J., Wolff, M., Montefiori, H.L., Grimm, B.D., Hulsey, M.L., Miller, J., McClure, H.M., D., Gorse, G.J., Graham, B.S., McElrath, M.J., Clements-Mann, McNicholl, J.M., Moss, B., Robinson, H.L., 2001. Control of a M.L., Mulligan, M.J., Fast, P., Walker, M.C., Excler, J., Duliege, mucosal challenge and prevention of AIDS by a multiprotein A., Tartaglia, J., The NIAID AIDS Vaccine Evaluation Group, DNA/MVA vaccine. Science 292 (5514), 69. 1999. A canarypox vaccine expressing multiple human immu- Barouch, D.H., Santra, S., Schmitz, J.E., Kuroda, M.J., Fu, T.M., nodeficiency virus type 1 genes given alone or with rgp120 Wagner, W., Bilska, M., Craiu, A., Zheng, X.X., Krivulka, G.R., elicits broad and durable CD8+ cytotoxic T lymphocyte Beaudry, K., Lifton, M.A., Nickerson, C.E., Trigona, W.L., responses in seronegative volunteers. Journal of Infectious Dis- Punt, K., Freed, D.C., Guan, L., Dubey, S., Casimiro, D., Si- eases 180, 290. mon, A., Davies, M.E., Chastain, M., Strom, T.B., Gelman, Flanagan, K.L., Plebanski, M., Akinwunmi, P., Lee, E.A., Reece, R.S., Montefiori, D.C., Lewis, M.G., Emini, E.A., Shiver, W.H., Robson, K.J., Hill, A.V., Pinder, M., 1999. Broadly J.W., Letvin, N.L., 2000. Control of viremia and prevention of distributed T cell reactivity, with no immunodominant loci, clinical AIDS in rhesus monkeys by cytokine-augmented DNA to the pre-erythrocytic antigen thrombospondin-related adhe- vaccination. Science 290, 486. sive protein of Plasmodium falciparum in West Africans. Eu- Belshe, R.B., Stevens, C., Gorse, G.J., Buchbinder, S., Weinhold, ropean Journal of Immunology 29, 1943. K., Sheppard, H., Stablein, D., Self, S., McNamara, J., Frey, Goh, W.C., Markee, J., Akridge, R.E., et al., 1999. Protection S., Flores, J., Excler, J.L., Klein, M., El Habib, R., Duliege, against human immunodeficiency virus type 1 infection in per- A., Harro, C., Corey, L., Keefer, M., Mulligan, M., Wright, P., sons with repeated exposure: evidence for T cell immunity in the Celum, C., Judson, F., Mayer, K., McKirnan, D., Marmor, M., absence of inherited CCR5 coreceptor defects. Journal of Infec- Woody, G., for the National Institute of Allergy and Infectious tious Diseases 179, 548. Diseases AIDS Vaccine Evaluation Group, and HIV Network Herr, W., Schneider, J., Lohse, A.W., Meyer zum Buschenfelde, K., for Prevention Trials (HIVNET), 2001. Safety and immunoge- Wolfel, T., 1996. Detection and quantification of blood-derived nicity of a canarypox-vectored human immunodeficiency virus CD8+ T lymphocytes secreting tumor necrosis factor a in res- type 1 vaccine with or without gp120: a phase 2 study in ponse to HLA-A2.1-binding melanoma and viral peptide anti- higher- and lower-risk volunteers. Journal of Infectious Dis- gens. Journal of Immunological Methods 191, 131. eases 183, 1343. Herr, W., Linn, B., Leister, N., Wandel, E., Meyer zum Buschen- Benjamini, Y., Hochberg, Y., 1995. Controlling the false discovery felde, K., Wolfel, T., 1997. The use of computer-assisted video rate: a practical and powerful approach to multiple testing. Jour- image analysis for the quantification of CD8+ T lymphocytes nal of the Royal Statistical Society. Series B. Methodological producing tumor necrosis factor spots in response to peptide 57, 289. antigens. Journal of Immunological Methods 203, 141. Borrow, P., Lewicki, H., Hahn, B.H., Shaw, G.M., Oldstone, M.B., Herr, W., Ranieri, E., Olson, W., Zarour, H., Gesualdo, L., Storkus, 1994. Virus-specific CD8+ cytotoxic T-lymphocyte activity as- W.J., 2000. Mature dendritic cells pulsed with freeze-thaw cell sociated with control of viremia in primary human immunode- lysates define an effective in vitro vaccine designed to elicit ficiency virus type 1 infection. Journal of Virology 68, 6103. EBV-specific CD4+ and CD8+ T lymphocyte responses. Blood Cao, J., McNevin, J., Holte, S., Fink, L., Corey, L., McElrath, M.J., 96, 1857. 2003. Comprehensive analysis of human immunodeficiency Hochberg, Y., Tamhane, A.C., 1987. Multiple Comparison Proce- virus type 1 (HIV-1)-specific gamma interferon-secreting dures. Wiley, New York. CD8+ T cells in primary HIV-1 infection. Journal of Virology Hochberg, Y., Westfall, P.H., 2000. On some multiplicity problems 77, 6867. and multiple comparison procedures in biostatistics. Handbook Cohen, J., 2002. Disappointing data scuttle plans for large-scale of Statistics, vol. 18. Elsevier, New York, p. 75. AIDS vaccine trial. Science 295, 1616. Holm, S., 1979. A simple sequentially rejective multiple test pro- Costigan, T., 1998. Bonferroni inequalitites and intervals. In: cedure. Scandinavian Journal of Statistics 6, 65. M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34 33

Hsu, J.C., 1996. Multiple Comparisons. Theory and Methods. Matloubian, M., Concepcion, R.J., Ahmed, R., 1994. CD4+ T cells Chapman & Hall, New York. are required to sustain CD8+ cytotoxic T-cell responses during Huang, X., Fan, Z., Kalinyak, C., Mellors, J.W., Rinaldo, C.R., 2000. chronic viral infection. Journal of Virology 68, 8056. CD8+ T-cell gamma interferon production specific for human Mazzoli, S., Trabattoni, D., Lo Caputo, S., et al., 1997. HIV-specific immunodeficiency virus type 1 (HIV-1) in HIV-1 infected sub- mucosal and cellular immunity in HIV-seronegative partners of jects. Clinical and Diagnostic Laboratory Immunology 7, 279. HIV-seropositive individuals. Nature Medicine 3, 1250. Ihaka, R., Gentleman, R., 1996. R: a language for data analysis and McCutcheon, M., Wehner, N., Wensky, A., Kushner, M., Doan, S., graphics. Journal of Computational and Graphical Statistics 5, Hsiao, L., Calabresi, P., Ha, T., Tran, T.V., Tate, K.M., Winkel- 299. hake, J., Spack, E.G., 1997. A sensitive ELISPOT assay to Jin, X., Bauer, D.E., Tuttleton, S.E., Lewin, S., Gettie, A., Blan- detect low-frequency human T lymphocytes. Journal of Immu- chard, J., Irwin, C.E., Safrit, J.T., Mittler, J., Weinberger, L., nological Methods 210, 149. Kostrikis, L.G., Zhang, L., Perelson, A.S., Ho, D.D., 1999. Musey, L., Hughes, J., Schacker, T., Shea, T., Corey, L., McElrath, Dramatic rise in plasam viremia after CD8(+) T cell depletion M.J., 1997. Cytotoxic-T-cell responses, viral load, and disease in simian immunodeficiency virus-infected macaques. Journal progression in early human immunodeficiency virus type 1 in- of Experimental Medicine 189, 991. fection. New England Journal of Medicine 337, 1267. Jung, M.-C., Hartmann, B., Gerlach, J.-T., Diepolder, H., Gruber, Nagorsen, D., Keilholz, U., Rivoltini, L., Schmittel, A., Letsch, A., R., Schraut, W., Gruner, N., Zachoval, R., Hoffman, R., San- Asemissen, A.M., Berger, G., Buhr, H.-J., Thiel, E., Scheiben- tantonio, T., Wchtler, M., Pape, G.R., 1999. Virus-specific bogen, C., 2000. Natural T-cell response against MHC class I lymphokine production differs quantitatively but not qualita- epitopes of epithelial cell adhesion molecule, her-2/neu, and tively in acute and chronic hepatitis B infection. Virology carcinoembryonic antigen in patients with colorectal cancer. 261, 165. Cancer Research 60, 4850. Kaul, R., Plummer, F.A., Kimani, J., Dong, T., Kiama, P., Rostron, NIAID Office of Communications and Public Liaison, 2002. T., Njagi, E., MacDonald, K.S., Bwayo, J.J., McMichael, A.J., NIAID Phase III HIV vaccine trial to determine correlate of Rowland-Jones, S.L., 2000. HIV-1-specific mucosal CD8+ lym- protection will not proceed. NIAID News. phocyte responses in the cervix of HIV-1-resistant prostitutes in Ogg, G.S., Jin, X., Bonhoeffer, S., Dunbar, P.R., Nowak, M.A., Nairobi. Journal of Immunology 164, 1602. Monard, S., Segal, J.P.,Cao, Y., Rowland-Jones, S.L., Cerundolo, Koup, R.A., Safrit, J.T., Cao, Y., Andrews, C.A., McLeod, G., V., Hurley, A., Markowitz, M., Ho, D.D., Nixon, D.F., McMi- Borkowsky, W., Farthing, C., Ho, D.D., 1994. Temporal asso- chael, A.J., 1998. Quantitation of HIV-1-specific cytotoxic T ciation of cellular immune responses with the initial control of lymphocytes and plasma load of viral RNA. Science 279, 2103. viremia in primary human immunodeficiency virus type 1 syn- Rosner, B., 1995. Fundamentals of Biostatistics, Fourth ed. Dux- drome. Journal of Virology 68, 4650. bury Press, North Scituate, MA. Lalvani, A., Brookes, R., Hambleton, S., Britton, W.J., Hill, A.V., Rowland-Jones, S., Suttone, J., Ariyoshi, K., et al., 1995. HIV- McMichael, A.J., 1997. Rapid effector function in CD8+ me- specific cytotoxic T-cells in HIV-exposed but uninfected Gam- mory T cells. Journal of Experimental Medicine 186, 859. bian women. Nature Medicine 1, 59. Larsson, M., Jin, X., Ramratnam, B., Ogg, G.S., Engelmayer, J., Russell, N., Hudgens, M.G., Ha, R., Havenar-Daughton, C., McEl- Demoitie, M.A., McMichael, A.J., Cox, W.I., Steinman, R.M., rath, M.J., 2003. Moving to HIV-1 vaccine efficacy trials: de- Nixon, D., Bhardwaj, N., 1999. A recombinant vaccinia virus fining T cell responses as potential correlates of immunity. based ELISPOT assay detects high frequencies of Pol-specific Journal of Infectious Diseases 187, 226. CD8 T cells in HIV-1-positive individuals. AIDS 13, 767. Self, S.G., Hudgens, M.G., Chiu, Y., Rossini, A.J., 2001. Statistical Larsson, M., Messmer, D., Somersan, S., Fonteneau, J.F., Dona- methods for evaluating ELISPOT assays in HIV vaccine trials. hoe, S.M., Lee, M., Dunbar, P.R., Cerundolo, V., Julkunen, I., AIDS Vaccine 2001, Philadelphia September Poster no. 36. Nixon, D.F., Bhardwaj, N., 2000. Requirement of mature den- Severini, T.A., 1999. On the effect of overdispersion on exact con- dritic cells for efficient activation of Influenza A-specific ditional tests. Journal of the Royal Statistical Society. Series B. Memory CD8+ T cells. Journal of Immunology 165, 1182. Methodological 61, 115. Lehmann, E.L., 1959. Testing Statistical Hypotheses. Wiley, New Subklewe, M., Chahroudi, A., Bickham, K., Larsson, M., Kurilla, York. M.G., Bhardwaj, N., Steinman, R.M., Steinman, M., 1999. Lewis, J.J., Janetzki, S., Schaed, S., Panageas, K.S., Wang, S., Presentation of Epstein-Barr virus latency antigens to CD8+, Williams, L., Meyers, M., Butterworth, L., Livingston, P.O., interferon-g-secreting, T lymphocytes. European Journal of Chapman, P.B., Houghton, A.N., 2000. Evaluation of CD8(+) Immunology 29, 3995. T-cell frequencies by the Elispot assay in healthy individuals Venables, W.N., Ripley, B.D., 1999. Modern Applied Statistics with and in patients with metastatic melanoma immunized with ty- S-PLUS. Springer-Verlag, New York. rosinase peptide. International Journal of Cancer 3, 391. Vogel, T.U., Horton, H., Fuller, D.H., Carter, D.K., Vielhuber, K., Loing, E., Andrieu, M., Thiam, K., Schorner, D., Wiesmller, K.-H., O’Connor, D.H., Shipley, T., Fuller, J., Sutter, G., Erfle, V., Hosmalin, A., Jung, G., Gras-Masse, H., 2000. Extension of Wilson, N., Picker, L.J., Watkins, D.I., 2002. Differences bet- HLA-A*0201-restricted minimal epitope by N-palmitoyl-lysine ween T cell epitopes recognized after immunization and after increases the life span of functional presentation to cytotoxic T infection. Journal of Immunology 169, 4511. cells. Journal of Immunology 164, 900. Westfall, P.H., Young, S.S., 1993. Resampling-based Multiple Tes- 34 M.G. Hudgens et al. / Journal of Immunological Methods 288 (2004) 19–34

ting: Examples and Methods for p-value Adjustment. Wiley, Zajac, A.J., Blattman, J.N., Murali-Krishna, K., Sourdive, D.J., New York. Suresh, M., Altman, J.D., Ahmed, R., 1998. Viral immune Westfall, P.H., Tobias, R.D., Rom, D., Wolfinger, D., Hochberg, Y., evasion due to persistence of activated T cells without ef- 1999. Multiple Comparisons and Multiple Tests Using the SAS fector function. Journal of Experimental Medicine 188, System SAS Institute, Cary, NC. 2205. 本文献由“学霸图书馆-文献云下载”收集自网络，仅供学习交流使用。

学霸图书馆（www.xuebalib.com）是一个“整合众多图书馆数据库资源，

提供一站式文献检索和下载服务”的24 小时在线不限IP 图书馆。图书馆致力于便利、促进学习与科研，提供最强文献下载服务。

图书馆导航：

图书馆首页文献云下载图书馆入口外文数据库大全疑难文献辅助工具