Zoonoses and Public Health REVIEW ARTICLE Statistical Evaluation of Test Accuracy Studies for Toxoplasma gondii in Food Animal Intermediate Hosts I. A. Gardner1, M. Greiner2 and J. P. Dubey3 1 Department of Medicine and Epidemiology, School of Veterinary Medicine, University of California, Davis, CA, USA 2 Federal Institute for Risk Assessment, Epidemiology, Biostatistics and Mathematical Modelling Unit, Berlin, Germany 3 United States Department of Agriculture, Agricultural Research Service, Animal and Natural Resources Institute, Animal Parasitic Diseases Laboratory, Beltsville, MD, USA Impacts • Receiver-operating curve (ROC) analysis and area under the ROC curve should be used to compare tests for Toxoplasma gondii against a highly accurate reference standard. • In the absence of a perfect reference standard, sensitivity and specificity of tests under evaluation can be estimated by maximum likelihood or Bayesian latent class methods. These methods are being increasingly accepted as a legitimate approach in chronic infectious diseases. • Recommendations are made regarding methods for improved analysis and reporting of studies for T. gondii. Keywords: Summary Diagnostic test evaluation; test accuracy; receiver-operating characteristic analysis; The availability of accurate diagnostic tests is essential for the detection and likelihood ratios; latent class analysis; control of Toxoplasma gondii infections in both definitive and intermediate Toxoplasma gondii hosts. Sensitivity, specificity and the area under the receiver-operating character- istic (ROC) curve are commonly used measures of test accuracy for infectious Correspondence: diseases such as toxoplasmosis. These test performance characteristics are I. A. Gardner. Department of Medicine and important considerations when selecting from among a group of tests for a spe- Epidemiology, School of Veterinary Medicine, University of California, One Shields Ave, cific testing purpose. In this study, we reviewed statistical approaches to evalua- Davis, CA 95616, USA. Tel.: 530-752-6992; tion of tests for toxoplasmosis with and without a gold-standard (reference) Fax: 530-752-0414; test, including use of ROC analysis and likelihood ratios which retain the diag- E-mail: [email protected] nostic information inherent in a quantitative test result. We use previously pub- lished data from a comparison of the accuracy of serological tests for swine Received for publication August 5, 2008 toxoplasmosis to demonstrate suggested methods of data analysis. We make rec- ommendations for statistical analysis and reporting of test evaluation studies for doi: 10.1111/j.1863-2378.2009.01281.x T. gondii in food animals based on our own experiences and those of others. intermediate hosts and to minimize the risk of human Introduction infection associated with ingestion of undercooked meat Livestock, poultry and wild game are commonly harbouring T. gondii cysts. Serological tests are used infected with Toxoplasma gondii following ingestion of mostly for detection of infected animals because they are pasture, feed or water contaminated with cat faeces inexpensive and results can be obtained rapidly. Compre- containing oocysts. The occurrence of T. gondii cysts in hensive studies comparing the accuracy of multiple sero- edible muscles of these species poses a potential risk to logical tests with bioassay-based reference standards have humans, if the meat is consumed without thorough been reported in pigs (Dubey et al., 1995a) but not in cooking to temperatures of at least 67°C (Dubey et al., other livestock hosts such as goats and sheep, in wild 1990). game or in backyard poultry. Accurate and reliable diagnostic tests are essential for Design and reporting standards for test accuracy stud- the detection, surveillance and control of infections in ies have been developed for human diseases (Bossuyt 82 ª 2009 Blackwell Verlag GmbH • Zoonoses Public Health. 57 (2010) 82–94 I. A. Gardner et al. Statistical Evaluation of Test Accuracy Studies for T. gondii et al., 2003a,b) and these guidelines have been refined for 2005). In addition, latent class methods do not correct infectious diseases especially in the developing world for other biases such as verification (work-up) bias that (Peeling et al., 2006; TDR Diagnostics Evaluation Expert occur in diagnostic test evaluation studies because of fail- Panel, 2006). In addition, quality assessment criteria for ure to consider important design aspects (Bossuyt et al., inclusion of studies in systematic reviews of diagnostic 2003b). test accuracy are now published (Whiting et al., 2003, In this study, we describe statistical approaches to the 2004). In contrast, standards have not been developed for evaluation of tests for T. gondii in food animal intermedi- infectious animal diseases although two of the authors ate hosts. We use data from a previously published evalu- (IG, MG) have made general recommendations about ation study of serological tests for swine toxoplasmosis epidemiological (population-based) approaches to test (Dubey et al., 1995a) to demonstrate methods that evaluation (Greiner and Gardner, 2000a). In addition, the depend or do not depend on the availability of a gold World Organization for Animal Health provides guide- standard reference test. Finally, we make recommenda- lines for international certification of tests for trade and tions regarding the analysis and reporting of test evalua- non-trade applications and has a registry of tests that are tion studies for toxoplasmosis with the overall goal of validated as fit for specific purposes (Office International improving their quality. des Epizooties [OIE], 2008). Evaluation of the accuracy of diagnostic tests should be Example: Evaluation of the Accuracy of based on representative field samples from naturally Serological Testing for Detection of T. gondii infected animals rather than experimental challenge stud- Infection in Sows ies alone (Greiner and Gardner, 2000a). Experimental challenge can provide estimates of time to detect infection Serological tests are a practical and inexpensive method and duration of detectable infection, but the doses of for estimating the prevalence of T. gondii infection in organisms, challenge routes and experimental conditions food animals and wild game (reviewed by Tenter et al., (environment, husbandry, lack of concurrent infections) 2000; Dubey and Jones, 2008). Toxoplasma infections may not be representative of field situations. If study may cause reproductive failure in pregnant females but design recommendations are followed to minimize biases most infections are subclinical and hence, tests usually are (Greiner and Gardner, 2000a; Bossuyt et al., 2003a,b), use validated for detection of subclinical infection. of appropriate field samples mitigates inferential problems To demonstrate recommended approaches to evalua- associated with the generalization of findings from chal- tion of test accuracy, we re-analyzed data from a study lenge studies and allows for estimation of the sensitivity conducted by one of us (JPD) to evaluate the sensitivity and specificity of the test under evaluation (termed the and specificity of five serological tests (MAT = modified index test in Bossuyt et al., 2003a) to those of a reference agglutination test; ELISA = enzyme-linked immunoassay; standard for a specific testing purpose. The reference LAT = latex agglutination test; IHAT = indirect hemag- standard is often termed a ‘gold standard’ if it provides glutination inhibition test; DT = Sabin-Feldman dye test) perfect classification of infection status. For chronic infec- for toxoplasmosis in 1000 naturally exposed sows (Dubey tious diseases such as toxoplasmosis, it is often possible et al., 1995a). Hearts were collected in batches from a to establish an animal’s infection status definitively by swine slaughterhouse in Iowa and serological tests were post-mortem examination followed by additional tests of performed on clotted heart blood in collaborators’ labora- tissues that are considered predilection sites for cysts. tories. Only 893 sera were suitable for DT because of bac- Toxoplasma gondii cysts not only have a high affinity for terial contamination and anti-complementary substances. skeletal and cardiac muscles and brain in most species One serum was missing an ELISA result. All serological but can also be found in visceral organs such as lungs, tests were conducted in a blinded fashion. liver and kidneys (Dubey et al., 1998). Mouse bioassay, which is often used as a reference Latent-class statistical methods that do not require des- standard in test evaluation studies, was performed on all ignation of a gold standard provide a flexible approach to 1000 samples using 100 g of homogenized, acidic pepsin- estimate test accuracy (Hui and Walter, 1980; Enøe et al., digested heart tissue, and a subset of samples (n = 183) 2000) and potentially can prevent the bias that occurs in yielding low titres on the MAT was selectively followed estimates of sensitivity and specificity if the test under up by bioassay in T. gondii-free cats using five times more evaluation is compared with an imperfect reference stan- heart muscle (500 g). For bioassay, hearts were allocated dard. However, to obtain confidence intervals (CI) of the into two groups: group 1 consisted of the first 463 hearts same width for sensitivity and specificity, sample sizes for of which 42 (9.1%) were bioassayed in cats and group 2 a latent class analysis are typically much larger than those consisted of the remaining 537