UvA-DARE (Digital Academic Repository)
Beyond diagnostic accuracy. Applying and extending methods for diagnostic test research
Glas, A.S.
Publication date 2003
Link to publication
Citation for published version (APA): Glas, A. S. (2003). Beyond diagnostic accuracy. Applying and extending methods for diagnostic test research.
General rights It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons).
Disclaimer/Complaints regulations If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible.
UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)
Download date:30 Sep 2021 11 The Diagnostic Odds Ratio: A single indicator of test performance e
Afinaa S. Glas, Jeroen G. Lijmer, Martin H. Prins, Gouke J. Bonsel, Patrickk M.M. Bossuyt
Journall of Clinical Epidemiology, accepted
Abstract t Diagnosticc testing can be used to discriminate subjects with a target disorder from subjectss without it. Several indicators of diagnostic performance have been proposed, suchh as sensitivity and specificity. Using paired indicators can be a disadvantage in comparingg the performance of competing tests, especially if one test does not outperformm the other on both indicators. Here we propose the use of the odds ratioo as a single indicator of diagnostic performance. The diagnostic odds ratio is closelyy linked to existing indicators, it facilitates formal meta-analysis of studies on diagnosticc test performance, and it is derived from logistic models, which allow forr the inclusion of additional variables to correct for heterogeneity. A disadvantage iss the impossibility of weighing the true positive-and false positive rate separately. Inn this paper the application of the diagnostic odds ratio in test evaluation is illustrated. .
13 3 11 The Diagnostic Odds Ratio: A single indicator of test performance e
Introduction n Inn an era of evidence-based medicine, decision makers need high-quality data to supportt decisions about whether or not to use a diagnostic test in a specific clinical situationn and, if so, which test. Many quantitative indicators of test performance havee been introduced, comprising sensitivity and specificity, predictive values, chance-correctedd measures of agreement, likelihood ratios, area under the receiver operatingg characteristic curve, and many more. All are quantitative indicators of thee test's ability to discriminate patients with the target condition (usually the diseasee of interest) from those without it, resulting from a comparison of the test's resultss with those from the reference standard in a series of representative patients. Inn most applications, the reference standard is the best available method to decide onn the presence or absence of the target condition. Less well known is the odds ratioo as a single indicator of test performance. The odds ratio is a familiar statistic inn epidemiology expressing the strength of association between exposure and disease.. As such it can also be applied to express the strength of the association betweenn test result and disease. Thiss paper offers an introduction to the understanding and use of the odds ratioo in diagnostic applications. In brief, we will refer to it as the diagnostic odds ratioo (DOR). First we will point out the usefulness of the odds ratio in dichotomous andd polychotomous tests. We will then discuss the use of the DOR in meta- analysiss and the application of conditional logistic regression techniques to enhance thee information resulting from such analysis.
Dichotomouss test outcomes .Althoughh most diagnostic tests have multiple or continuous outcomes, either groupingg of categories or application of a cut-off value is frequently applied to classifyy results into positive or negative. Such a dichotomization enables to representt the comparison between a diagnostic test and its reference standard in onee 2x2 contingency table, as depicted in table 1.
Tablee 1 2x2 contingency table.
Referencee Test
Targett disorder Noo target disorder
test t positive e TP P FP P
negative e FN N TN N
Thee abbreviations TP, FP, FN and TN denote the number of respectivelyy true positives, false positives, false negatives and true negatives.. Same definitions are used throughout the text and table 2.
14 4 Commonn indicators of test performance derived from such a 2x2 table are thee sensitivity of the test, its specificity, the positive and negative predictive values andd the positive and negative likelihood ratios.1 See table 2 for a definition of thesee indicators. Unfortunately,, none of these indicators in itself validly represent the test's discriminatoryy performance. Sensitivity is only part of the discriminatory evidence ass high sensitivity may be accompanied by low specificity. Additionally, no simple aggregationn rule exists to combine sensitivity and specificity into one measure of performance. . Tablee 3 shows the performance of three different radiological diagnostic tests too stage ovarian cancer as an illustration of the need for combined judgment. All threee tests were performed in a group of 280 patients suspected of ovarian cancer.2 Surgicall and histo-pathological findings were used as the reference standard. The sensitivityy of the ultrasound was worse than that of the computer tomography (CT)) scan in detecting peritoneal metastases but for the specificity the reverse held.. Likelihood ratios and the predictive values are also not decisive. Also the combinedd evidence of the pairs of indicators cannot simply be ranked. Forr this, a single indicator of test performance like the test's accuracy is required. Inn addition to its global meaning of agreement between test and reference standard, accuracyy in its specific sense refers to the percentage of patients correctly classified byy the test under evaluation. This percentage depends on the prevalence of the targett disorder in the study group whenever sensitivity and specificity are not equal andd it weights false positive and false negative findings equally. Another single indicatorr is Youden's index.3 4 It can be derived from sensitivity and specificity and ass such it is independent of prevalence, but since it is a linear transformation of thee mean sensitivity and specificity its values are difficult to interpret.5
15 5 11 The Diagnostic Odds Ratio: A single indicator of test performance e
133 16 6 Thee odds ratio used as single indicator of test performance is a third option. Itt is not prevalence dependent and may be easier to understand, as it is a familiar epidemiologicall measure. The diagnostic odds ratio of a test is the ratio of the oddss of positivity in disease relative to the odds of positivity in the non-diseased.67 Thiss means that the following relations hold: TPP /FP _ sens /(I - spec) (1) ) DORR = FN/TNN (1-sens)/ spec Alternatively,, the DOR can be read as the ratio of the odds of disease in test positivess relative to the odds of disease in test negatives. TP/FN__ PPV /(1-NPV) (2) ) DORR = FP// TN (1-PPV)// NPV Theree is also a close relation between the DOR and likelihood ratios: TP P'R M M LR(+) ) (3)) DOR = FP/ / TN N LR(-) ) Thee value of a DOR ranges from 0 to infinity, with higher values indicating betterr discriminatory test performance. A value of 1 means that a test does not discriminatee between patients with the disorder and those without it. Values lower thann 1 point to improper test interpretation (more negative tests among the diseased). Thee inverse of the DOR can be interpreted as the ratio of negativity odds within thee diseased relative to the odds of negativity within the non-diseased. The DOR risess steeply when sensitivity or specificity becomes near perfect, as is illustrated in figurefigure l.6 o o Q Q Figuree 1 Behaviour of the odds ratio with changingg sensitivity and specificity. Specificity:: = 0.99, —*— = 0.95, == 0.80, = 0.50 0.44 0.6 Sensitivity y 17 7 11 The Diagnostic Odds Ratio: A single indicator of test performance e Ass can be concluded from above formulas the DOR does not depend on the prevalencee of the disease that the test is used for. Nevertheless, across clinical applicationss it is likely to depend on the spectrum of disease severity, as is the case forr all other indicators of test performance 89. Another point to consider is that, thee DOR, as a global measure, cannot be used to judge a test's error rates, at particularr prevalence's. Two tests with an identical DOR can have very different sensitivityy and specificity, with distinct clinical consequences. If a 2x2 table contains zeroes,, the DOR will be undefined. Adding 0.5 to all counts in the table is a commonlyy used method to calculate an approximation of the DOR.1011 Confidence intervalss for range estimates and significance testing can be conventionally calculatedd with the following formula.1213 (4)) SE(logDOR)= — +— +— + — \TPP TN FP FN AA 95 % confidence interval of the logDOR can then be obtained by: (5)) ) Calculatingg the antilog of this expression (back-transformation) provides the confidencee interval of the DOR. Inn the example of the radiological tests comparison, represented in table 3, thee estimated DOR for the ultrasound in detecting peritoneal metastases is 31 (0.69/(l-0.69))/((l-0.93)/0.93).. This means that for the ultrasound the odds for positivityy among subjects with peritoneal metastases is 31 times higher than the oddss for positivity among subjects without peritoneal metastases. In the same way thee DOR's for the CT scan and magnetic resonance imaging (MRI) can be calculatedd (table 3). MR imaging has the highest DOR in detecting peritoneal metastasess compared to the ultrasound and CT scan (77 versus 31 and 51 respectively).. In contrast, ultrasound has the highest DOR in detecting liver metastasess and lymph nodes: 54 versus 17 for CT and 15 for MRI (table 4). Iff DOR's had been presented in the original article, a quick comparison wouldd have led to the conclusion that MR imaging performs best in diagnosing peritoneall metastases whereas ultrasound does better in diagnosing lymph node andd liver metastases in this population. 18 8 „„ »- O <-- *- r» \r>\r> «- T- 5 8:: 8: 8: «ttEE 3< <** S* ** — ,- njj t- 0> > MM * TJ J oii m E£ £ j= = o>> -^ a a cc * E E =>> TJ ww c §jj rara ™ U U == U"> ra ra MM ww 3e ™™ tf) aii JB JJJ CL «« o> CÉÉ E 19 9 Thee Diagnostic Odds Ratio: A single indicator of test performance e Polychotomouss and continuous test outcomes Thee performance of a test for which several cut-offs are available can be expressed byy means of ROC analysis.1415 A receiver operating characteristic (ROC) curve plotss the true positive rate on the Y-axis as a function of the false positive rate on thee X-axis for all possible cut-off values of the test under evaluation. The area underr the curve obtained (AUC) can subsequently be calculated as an alternative singlee indicator of test performance.16 Thee AUC takes values between 0 en 1, with higher values indicating better testt performance. These can be interpreted as an estimate of the probability that thee test correctly ranks two individuals of which one has the disease and one does nott have the disease.16 It can alternatively be interpreted as the average sensitivity acrosss all possible specificities. (6)) AUC = dx x 11 + DOR-- —^— 11 - x 0 0 Iff the DOR is constant for all possible cut-off values, the ROC curve will be symmetricc (in relation to the diagonal y=-x+l) and concave. In that case, a mathematicall relation exists between the AUC and the DOR of a test (see formula 6). Thee higher the value of the DOR the higher the AUC. AUC's of 0.85 and 0.90, forr example, correspond to DOR's of 13 and 24 respectively. An increase in AUC off 5% will almost double the DOR which is a direct consequence of scale differences: thee DOR has an upper limit of infinity whereas the AUC takes values in the 0 to 11 range. For non-symmetrical ROC curves the DOR is not constant over all cut- offoff points. In these cases the AUC cannot be calculated from the DOR associated withh a single (or a few) cutoff values. Thee shape of the ROC curve and the cut-off independence of the DOR dependd on the underlying distribution of test results in patients with and without thee target condition. Figure 2 shows several probability densities distributions of testt results in diseased and non-diseased populations. It can be observed that the DORR is reasonably constant for a large range off cut-off points on the ROC curve, butt for the extremes of sensitivity and specificity the DOR rises steeply. If the originall or transformed results in both diseased and non-diseased follow a logistic distributionn with equal SD, the DOR is constant for all possible cut-off values (figuree 2D).17 20 0 A)A) Normal, symmetrical 6 0.06-,0.06-, r10 D D 0.03 0.03 O O 0.05 0.05 R R QQ Logistic, assymetrical 0.06 0.06 0.10-,0.10-, rlO21 D D ,011X00 0 R R Figuree 2 Value of the DOR for all thresholds and distributions of test results in the non- diseaseddiseased and diseased. :: DOR .non-diseasedd population, .-diseasedd population Thee DOR in meta-analysis Thee DOR offers considerable advantages in meta-analysis of diagnostic studies whichh combines results from different studies into summary estimates with increasedd precision. Meta-analysis of diagnostic tests offers statistical challenges, becausee of the bivariate nature of the conventional expressions of test performance. Simplee pooling of sensitivity and specificity usually is inappropriate as this approach ignoress threshold differences."18 In addition, heterogeneity may lead to an underestimationn of a test's performance. The current strategy for meta-analysis, ass endorsed by the Methods Working Group of the Cochrane Collaboration19, buildss on the methods described by Kardaun and Kardaun and Littenberg and Moses.1120211 The approach by Littenberg and Moses relies on the linear regression off the logarithm of the DOR of a study (dependent variable) on an expression of thee positivity threshold of that study (independent variable). If the regression line 21 1 11 The Diagnostic Odds Ratio: A single indicator of test performance e hass a zero slope, the DOR is constant across studies. A summary ROC (sROC) cann be produced after back transforming the regression line. The resulting sROC willl be symmetric and concave. In other words, study heterogeneity can be attributedd to threshold differences. In the context of the DOR, the summary DOR off the study under evaluation can be obtained from the intercept (e'mcrcrp') of the regressionn line.11 n Additional heterogeneity owing to variation in study characteristics (e.g.. cohort versus case-control) or clinical characteristics (e.g. heterogeneous prior therapy)) can be evaluated simultaneously by adding these variables as covariates to thee regression model, leaving a corrected estimated value for the pooled DOR. Thee resulting parameter estimates can be (back)transformed to relative diagnostic oddss ratio's (rDOR). A rDOR of 1 indicates that the particular covariate does not affectt the overall DOR. A rDOR>l means that studies, study centres or patient subgroupss with a particular characteristic have a higher DOR than studies without thiss characteristic. For a rDOR <1 the reverse holds.22 When the DOR is homogeneouss across studies, the DOR's of different studies can also be pooled directly.. Homogeneity can be tested by using the Q test statistic or the H statistic.23 Wee will illustrate the usefulness of the DOR in meta-analysis by re-analysing aa meta-analysis on the diagnostic performance of two magnetic resonance angiographyy techniques (3D gadolinium-enhanced (3D-GD) and 2D time-of- flightflight (2D-TOF)) detecting peripheral arteriosclerotic occlusive disease.24 The separatee meta-regression analysis yielded an intercept of 4.13 and a slope of 0.41 forr 2D-TOF. For 3D-GD these values were respectively 5.93 and -0.37. From the interceptt the summary DOR's can be calculated, respectively e413=62 and e5 93=376. Thee non-zero slopes, indicated heterogeneity apart from threshold differences, whichh in turn limits a direct comparison of summary odds ratio's. To explore additionall variation all studies of the two techniques were put together in one regressionn model. Then each available covariate was examined for its effect on the diagnosticc performance. Most effect had the covariates 3D-GD versus 2D-TOF techniquee and a covariate dealing with the post-processing technique (maximum intensityy projections (MIP) in addition to transverse source images or multiplanar reformationn (MIP +) versus MIP alone). Subsequently these 2 covariates were selectedd for the final multivariate model. The adjusted rDOR estimated was 7.5 (confidencee interval: 2.8-22) for the 3D-GD and 4.5 (confidence interval: 1.5-14) forr the use of MIP+. The confidence intervals did not contain the value 1. As suchh one can conclude that - after correction for heterogeneity - 3D-GD and the usee of MIP+ have a better diagnostic performance compared to 2D-TOF and the usee of MIPs alone. 22 2 Thee methodology of systematic reviews and meta-analysis of diagnostic tests iss still evolving, with new and potentially better methods being developed, better adaptedd to the inherently bivariate nature of the problem. Yet the convenience of thee odds ratio in statistical modelling guarantees its future role in the meta-analysis off diagnostic tests. Logisticc regression Thee DOR offers advantages when logistic regression is used with diagnostic problems.. Logistic regression can be used to construct decision rules, reflecting thee combined diagnostic value of a number of diagnostic variables.25 Another applicationn is the study of the added value of diagnostic tests.26 With a single dichotomouss test the logistic regression equation reads: (7)) P(D|x) = 1 1+exp(a+px) ) wheree x stands for the test result and the coefficients and have to be estimated. Iff a positive test result is coded as x = 1 and a negative as x = 0, we have 1 (8)) P(D|poSitive)=i ^^ 1+expp (a pxJ and d (9)) P(D| negative) = 11 +expa Next,, one derives from expression 1,8 and 9 (10)) DOR = [P(D|positive) / (1 -P(D|positive)) J / [P(D| negative) / (1 -P(D| negative))] = exp(G). Inn other words, the DOR equals the regression coefficient, after exponentiation. Logisticc regression modelling has been proposed as the preferred statistical method too obtain a post-test probability of disease when results from multiple tests are available.. History taking and physical examination can also be considered as individuall diagnostic tests. The post test probability after having obtained test resultss xl,x2, ...xk is expressed as (11)) P(D|X1,X2...Xk) = + x + x ++ 1+eXp-<« Pl 1 P2 2 +Pk*kY 23 3 Thee Diagnostic Odds Ratio: A single indicator of test performance e Withh multiple dichotomous tests of which the results x}, x2 ... x^ are coded as e ua tne presentt (1) or absent (0), the corresponding coefficients Sl5 fê2 ^k q l conditionall logDOR.7 These DOR's are conditional: they depend on the other variabless that have been used in the model.27 If more information becomes availablee a new regression equation has to be constructed to obtain the proper conditionall DOR. This application is illustrated by a study that aimed to assess thee value of symptoms in diagnosing arrhythmias in general practice.28 The followingg equation from the logistic model was created in order to estimate the probabilityy of arrhythmia in a patients with specific signs and symptoms: P(arrhythmia)= = MM i /-i , - (- 4.40+0.051*age+0.47*gender+1.11*palpitations+0 78*dyspnoea+0.45*useofcardiovas.med.) \ Agee is a continuous variable expressed in years. The odds ratio (e°-O51=1.05) calculatedd from the respective coefficient does not express the diagnostic performance off the variable age, but the OR for the increase in age per year. Gender is coded 1 forr males and 0 for females. The use of cardiovascular medication, palpitations andd dyspnoea as recorded during consultation are coded as positive (1) or negative (0).. Subsequently the conditional DOR of each dichotomous variable, adjusted forr the other variables, can be estimated. For gender the DOR is e047=1.6, meaning thatt the odds for having arrhythmias is 1.6 times larger in males than in females. Thee adjusted odds ratio's for palpitations during consultation, dyspnoea during consultationn and the use of cardiovascular medication are respectively 3.0, 2.2 andd 1.6. Discussion n Thee diagnostic odds ratio as a measure of test performance combines the strengths off sensitivity and specificity, as prevalence independent indicators, with the advantagee of accuracy as a single indicator. These characteristics lend the DOR particularlyy useful for comparing tests whenever the balance between false negative andd false positive rates is not of immediate importance. These features are also highlyy convenient in systematic reviews and meta-analyses. Inn decisions on the introduction of a test in clinical practice, we are aware thatt the actual balance between the true positive rate and false positive rate often matters.299 Whenever false positives and false negatives are weighted differentially bothh the prevalence and the conditional error rates of the test have to be taken intoo consideration to make a balanced decision. In these cases the DOR is less useful,, as it does not distinguish between the two types of diagnostic mistake. If 24 4 ruling-outt or ruling-in of the target condition is the primary intended use of a test,, conditional indicators such as sensitivity and specificity still have to be used. Ass all available measures of test performance, the DOR of a test is unlikely to bee a test-specific constant. Its magnitude likely depends on the spectrum of disease ass well as on pre-selection through the use of other tests.630 Despite this universal caveatt for indicators of diagnostic tests, we feel that a more systematic use of the oddss ratio in diagnostic research can contribute to more consistent applications of diagnosticc knowledge. Somee may object that there are already too many indicators of test performance. Withh such an abundance of choices, there is little need for yet another statistic. Thiss may be true, but it is hard to see how the selection can or should be produced. Eachh of the indicators serves a different purpose. Sensitivity and specificity are expressionss of the conditional hit rates of the test. Predictive values or posterior probabilitiess are the numbers that are most salient for clinical practice. The so- calledd likelihood ratios come in handy for comparing the diagnostic content of multiplee possible test results and for transforming those into post-test probabilities. Amidstt those helpful indicators, the diagnostic odds ratio has a place as a single statisticc with a long history and useful statistical properties. Acknowledgement t Thee authors thank Dr. Guus Hart for critically reviewing and giving useful suggestionss on earlier drafts. References s 1.. Sackett DL, Haynes RB, Guyatt GH, Tugwell P. Clinical epidemiology: a basic science for clinicalclinical medicine. Boston:Little, Brown and Co, 1991. 2.. Tempany CM, Zou KH, Silverman SG, Brown DL, Kurtz AB, McNeil BJ. Staging of advanced ovariann cancer: comparison of imaging modalities-report from the Radiological Diagnostic Oncologyy Group. Rad/o/ogy 2000;215(3)761-7. 3.. Youden WJ. Index for rating diagnostic tests. Cancer 1950;3:32-35. 4.. Hilden J, Glasziou P. Regret graphs, diagnostic uncertainty and Youden's Index. Stat Med 1996;15(10):969-86. . 5.. Linnet K. A review on the methodology for assessing diagnostic tests. Clin Chem 1988;34(7):: 1379-1386. 6.. Kraemer HC. Risk Ratio's, Odds Ratio, and the test QROC. Evaluating Medical Tests. Newburyy Park: SAGE Publications, Inc., 1992:103-113. 7.. Korte de PJ. Probability analysis in diagnosing Coronary Artery Disease, 1993. 25 5 11 The Diagnostic Odds Ratio: A single indicator of test performance e 8.. Hlatky MA, Pryor DB, Harrell FE, Jr., Califf RM, Mark DB, Rosati RA. Factors affecting sensitivityy and specificity of exercise electrocardiography. Multivariable analysis. A J Med 1984;77(1):64-71. . 9.. Moons KG, van Es GA, Deckers JW, Habbema JD, Grobbee DE. Limitations of sensitivity, specificity,, likelihood ratio, and bayes' theorem in assessing diagnostic probabilities: a clinicall example. Epidemiology 1997;8(1):12-17. 10.. Haldane J. The estimation and significance of the logarithm of a ratio of frequencies. Ann Humm Gen 1955;20:309-314. 11.. Littenberg B, Moses LE. Estimating diagnostic accuracy from multiple conflicting reports: aa new meta-analytic method. Med Dec Making 1993;13(4):313-21. 12.. Altman D. Comparing groups-categorical data. In: Altman D, editor. Prac Stat Med Res. Unitedd States of America: Chapman&Hall/CRC, 1997:229-276. 13.. Bland JM, Altman DG. The odds ratio. BMJ 2000;320:1468. 14.. Choi BCK. Slopes of a receiver operating characteristic curve and likelihood ratios for a diagnosticc lest. Am J Epidemiol 1998;148(11):1127-1132. 15.. Swets JA. The Relative Operating Characteristic in Psychology. Science 1973;182:990-1000. . 16.. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristicc (ROC) curve. Radiology 1982;143(1):29-36. 17.. Hasselblad V, Hedges LV. Meta-analysis of screening and diagnostic tests. Psychol Bull 1995;117(1):167-178. . 18.. Irwig L, Macaskill P, Glasziou P, Fahey M. Meta-analytic methods for diagnostic test accuracy.. J Clin Epidemiol 1995;48(1):119-30; discussion 131-2. 19.. The Cochrane Methods Working Group. On Systematic Review Of Screening And Diagnosticc Tests: Recommended Methods: 1995; http://www.cochrane.org.au/. 20.. Kardaun JW, Kardaun OJ. Comparative diagnostic performance of three radiological proceduress for the detection of lumbar disk herniation. Meth Inf Med 1990;29(1):12-22. 21.. Moses LE, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test intoo a summary ROC curve: data-analytic approaches and some additional considerations. StatStat Med 1993;12(14):1293-1316. 22.. Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meuten JHP, et al. Empirical evidencee of design-related bias in studies of diagnostic tests. JAMA 1999;282{11): 1061-1066. . 23.. Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med 2002;21(11):1539-1558. . 24.. Nelemans PJ, Leiner T, de Vet HC, van Engelshoven JM. Peripheral arterial disease: meta- analysiss of the diagnostic performance of MR angiography. Rad/o/ogy 2000;217(1):105-14. 25.. Wells PS, Anderson DR, Rodger M, Ginsberg JS, Kearon C, Gent M, et al. Derivation of a simplee clinical model to categorize patients probability of pulmonary embolism: increasing thee models utility with the SimpliRED D-dimer. Thromb Haemost 2000;83(3):416-20. 26 6 26.. Clark TJ, Bakour SH, Gupta JK, Khan KS. Evaluation of outpatient hysteroscopy and ultrasonographyy in the diagnosis of endometrial disease. Ofastet Gynecol 2002;99(6): 1001-7. . 27.. Knottnerus J A. Application of logistic regression to the analysis of diagnostic data: exact modelingg of a probability tree of multiple binary variables. Med Dec Making 1992; 12(2): 93-108. . 28.. Zwietering PJ, Knottnerus JA, Rinkens PE, Kleijne MA, Gorgels AR Arrhythmias in general practice:: diagnostic value of patient characteristics, medical history and symptoms. Fam Practt 1998;15(4):343-53. 29.. Glasziou P, Hilden J. Test selection measures. Med Dec Making 1989;9(2):133-141. 30.. Hlatky MA, Lee KL, Botvinick EH, Brundage BH. Diagnostic test use in different practice settings.. A controlled comparison. Arch Int Med 1983; 143(10): 1886-1889. 27 7 28 8