Springer.Biostatistics.For.Radiologists

Biostatistics for Radiologists

Planning, Performing, and Writing a Radiologic Study Francesco Sardanelli • Giovanni Di Leo

Biostatistics for Radiologists

Planning, Performing, and Writing a Radiologic Study

Foreword by Adrian K. Dixon FRANCESCO SARDANELLI, MD GIOVANNI DI LEO, Dr.Sci. Associate Professor of Radiology Researcher University of Milan School of Medicine Radiology Unit Department of Medical and Surgical Sciences IRCCS Policlinico San Donato Director of Radiology Unit San Donato Milanese, Italy IRCCS Policlinico San Donato e-mail: [email protected] San Donato Milanese, Italy e-mail: [email protected]

Originally published as: Biostatistica in Radiologia Progettare, realizzare e scrivere un lavoro scientifico radiologico Francesco Sardanelli, Giovanni Di Leo © Springer-Verlag Italia 2008 All rights reserved

Library of Congress Control Number: 2008938920

ISBN 978-88-470-1132-8 Springer Milan Berlin Heidelberg New York e-ISBN 978-88-470-1133-5

Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Italia 2009

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illu- strations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the Italian Copyright Law in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the Italian Copyright Law.

The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Product liability: The publishers cannot guarantee the accuracy of any information about dosage and application contained in this book. In every individual case the user must check such information by consulting the relevant literature.

Typesetting: Ferrari – Studio editoriale, Cologno Monzese (MI), Italy Printing and binding: Printer Trento S.r.l., Trento, Italy

Printed in Italy Springer-Verlag Italia S.r.l., Via Decembrio 28, I-20137 Milan, Italy To Genny, Francesca, and Federica F.S.

To my father, who taught me how to distinguish significant from nonsignificant things G.D.L. Foreword

It is a pleasure to write the foreword to this excellent book on the evidence behind radiological investigations and biostatistics in radiology. This is an area which is not widely appreciated by radiologists and it would be an invaluable book for those in training who should become fully versed about terminology such as technical performance, diagnostic performance, diagnostic impact, therapeutic impact, patient impact, patient outcomes and societal impact. They should also know that the widely (and often erroneously) used term ‘accuracy’ may not be the best assessment! This very readable book should not only be useful for radiologists but also administrators who are now beginning to realise that an effective imaging department underpins all high quality cost-efficient modern medicine. The history and development of evidence-based radiology (from Fryback and Thornbury’s original paper right up to date with recent contributions from Hollingworth, Hunink, Jarvik and Malone) is very well presented. Lorenzo Mannelli, an excellent Italian Radiologist working in Cambridge, says of the Italian Edition: “The book is easy to read and the short paragraph titles on the side of the pages make it easy to use for future reviewing of “hot topics” when needed. All the examples and vocabulary are from the radiological world, making the statistics easier to understand. Although after reading this book, you will still need statistical advice, at least you will be able to understand what the statistician is speaking about! The final chapter on impact factors is interesting and helps the reader to understand the dynamics of journals. I definitely recommend this book as an easy reading for residents in radiology”. I am sure that this new English language edition will fill a very major void in the radiological literature. The authors have done us a great service.

Cambridge, UK, October 2008 Adrian K. Dixon, MD

Preface to the Italian Edition

Better to light a candle than to curse the darkness.

CONFUCIUS

Science deals with discovery but also with communication. It’s hard to say you have an idea if you are not able to evoke the same idea in the mind of your listener.

MARCUS DU SATOY

For many years “Biostatistics for Radiologists” was an unrealized dream of the senior author. Since that dream is now coming true, I have taken on the task of writing this preface, which offers the opportunity for an appraisal of the years leading up to the genesis of this book. I hope it can be useful to young colleagues who intend to devote themselves to radiologic research. More than twenty-five years ago, I was a resident at the Postgraduate School in Radiodiagnostics of the University of Genoa, directed by Professor Luigi Oliva. My supervisor was Professor Giorgio Cittadini, Director of the Chair “R” of Radiology of the University. He was the chairman of my medical grad- uation thesis, entitled “Colonic hypotonic effect of fenoverine and hyoscine N-butyl bromide: analysis of variance with nonparametric tests”. Already then there was an attention to statistical methods predicting the events of my future before me. In 1984, after several years mainly dedicated to gastrointestinal double- contrast studies, I was included in a small team made up of physicians, physicists, and engineers who had the good fortune to work on one of the first magnetic resonance imaging scanners installed in Italy. It was a proto- type with a resistive magnet operating at only 0.15 T. For the best use of this new diagnostic technology, the radiologist had to understand the NMR phenomenon, the radiofrequency pulse sequences, and the role of the field gra- dients which generate the images. At that time, physicists were teaching magnetic resonance in courses and congresses using formal demonstrations based on Bloch’s equations, combining both classic and quantum models. These lessons were very hard to follow. Only when formulas and equations were translated into a different language, positively associated with the clinical meaning of the images, did the attending radiologists open their eyes and grasp the practical sense of that theory. For the first time I understood that scientific communication was a crucial process requiring intellect, fan- tasy, and creativity. X Preface to the Italian Edition

In the same year, I was involved as author in a small paper entitled “Sensitivity, specificity, overall accuracy. What is the meaning of these three words commonly used in scientific radiologic language?”1. This paper was the result of an interesting discussion which had begun in Cittadini’s room late one evening and lasted for two-three hours. The topic – how to quantify diagnostic performance – seemed highly intriguing to me. I promised myself to gain a deeper insight into the matter. It was a new world waiting to be explored: how to evaluate the uncertainty intrinsic to biologic phenomena and measurements and, as a consequence, medical diagnosis. At the time I was only a resident cooperating towards writing an article, but I began to add some substance to the immaterial dream of writing a book. The chapter dedicated to “Indices of Diagnostic Performance” included in the Italian textbook “Diagnostic Imaging and Radiotherapy”, recently published in its sixth edition, would be the embry- onic stage of “Biostatistics for Radiologists”. Some years later, in 1987, I became a staff radiologist of the Chair R of Radiology at the Genoa University and San Martino Hospital. I began to add a planned research activity to the clinical routine. The areas of research involv- ing high-level cooperation with clinicians produced the most interesting results. This was the case with Giuseppe Molinari (cardiologist at the Genoa University) and Giuseppe Canavese, breast surgeon at the Genoa Cancer Research Institute. In this period I submitted my first manuscripts to peer- reviewed journals. Immediately, I understood that good technical knowledge and updated clinical experience are not enough for writing an article enough good to be accepted for publication. The crux of the matter is given by study design, data presentation and analysis, and, in particular, statistical methods which demonstrate the significance of the results. Around this time I began to interact with statisticians. Once again communication was stifled. Radiologists were on one side of a wall and statisticians on the other, as it had been earlier with the physicists for magnetic resonance. In my personal experience, however, this wall crumbled thanks to research con- duced on multiple sclerosis. In this field, magnetic resonance imaging was playing an increasingly important role. My relations with Gianluigi Mancardi (Department of Neurology, University of Genoa) and Paolo Bruzzi (Clinical Epidemiology, Genoa Cancer Research Institute) were a turning point. Each of us wanted to learn what the other two colleagues already knew and would spend hours and hours to reach… understanding. At the same time, another apparently impersonal factor was in action – the reviewers analyzing the manuscripts I submitted to the journals. Their criticisms were sometimes very harsh. However, the higher the rank of the journal, the greater the knowledge in methodology I could obtain by interacting with the reviewers, even though the manuscript was rejected. This was another way I begun to accumulate a limited know-how in biostatistics and research methodology applied to radiology. While I was (and still am) learn-

1 Sardanelli F, Garlaschi G, Cittadini G (1984) Sensibilità, specificità, accuratezza diagnostica. Quale significato attribuire a queste tre parole così spesso usate nel linguaggio scientifico radiologico? Il Radiologo 23:58-59. Preface to the Italian Edition XI ing from my errors, a long line of textbooks on medical statistics began to grow in my bookcase. At the beginning of the 1990s, I became head of Diagnostic Imaging at the Breast Unit of the San Martino University Hospital and the Genoa Research Cancer Institute. By that time, Italian breast radiologists were involved in a flourishing debate: clinical mammography on the one hand, organized screening mammography on the other. Which was the key-point? The majority of Italian women who asked for a mammographic examination in radiology departments were asymptomatic. Their request was one of spontaneous period- ic control. This gave rise to a kind of oxymoron: clinical mammography (with physical examination and frequent accessorial ultrasound examination) in an asymptomatic population. Radiologists with lengthy clinical experience of breast imaging on patients with symptoms were conditioned by a practice of “first of all, sensitivity” which attained acceptable levels of specificity with further work-up in a relevant fraction of patients. The application of this logic to asymptomatic women resulted in the medicalization of a healthy population. It was a typical problem generated by low disease prevalence. In asymptomatic women sensitivity must be combined with a sufficiently high specificity and positive predictive value. On the other hand, we knew that ultrasound enabled us to detect breast cancers in women with high breast density and negative mammography and that periodical mammography is also useful in women under 50 and over 70. However, once again there was a wall. Clinical mammography on one side, screening mammography on the other. This experience gave me new incentive to flesh out that dream. In the meantime, I begun to serve as a reviewer for international journals. This gave me the opportunity to compare my evaluation of a manuscript with those of other reviewers. Moreover, at the end of that decade I started to coop- erate with Franca Podo from the Istituto Superiore di Sanità (Rome), a physicist and world-renowned expert in magnetic resonance imaging and spectroscopy. Working together we conducted the HIBCRIT study for multimodal- ity surveillance of women at high genetic-familial risk of breast cancer. Here the high disease prevalence justified an intensive surveillance including physical examination, mammography, ultrasound, and contrast-enhanced magnetic resonance imaging. It was a fantastic experience from which I learned a lot, especially on the management of multicenter trials, an intense and effective cooperation without the need of breaking down walls, with a follow-on now extending to new topics. From 1999 to 2000 I was Director of the Department of Radiology at the Biomedical Institute in Genoa. This role broadened the spectrum of my experience. The higher levels of productivity in clinical radiologic activity was a preparation for the upcoming events. In fact, in 2001 I was assigned the Direction of the Department of Radiology at the Policlinico San Donato near Milan. In my opinion, the principal aim was to have a radiologic team with high levels of clinical efficiency and scientific research. The administrators gave me free reign to start a process of training and selection of young colleagues. Some of the fruits of this process can already be appreciated. I have to thank several persons who have been and con- tinue to be keystones in the day-to-day operation of the system: the radiologists Alberto Aliprandi, Bijan Babaei and Pietro Bertolotti and the coordinators of XII Preface to the Italian Edition

radiographers Francesco Gerra and Eleonora Norma Lupo. Recently we were joined by Carlo Ottonello, who was resident in Radiodiagnostics at the Genoa University at the beginning of the 1990s. Our younger colleagues have the opportunity to show their abilities in clinics and research, in part thanks to many projects we have in cooperation with the clinical departments of our institution. In recent years, I combined the Direction of the Unit of Radiology of the Policlinico San Donato (from 2006, appointed as Istituto di Ricovero e Cura a Carattere Scientifico, IRCCS, by the Ministry of Health) with the position of Associate Professor of Radiology at the University of Milan School of Medicine. This new context favored my study on research methodology. The last chapter of this book arose from a lesson entitled “How to Write a Scientific Paper?”. I held with the residents of the Postgraduate School in Radiodiagnostics on the express request of the Director of the School, Professor Gianpaolo Cornalba, in a framework of close cooperation and common rationale. At the same time I served on the National Board of the Councilors of the Italian Society of Medical Radiology as a President’s Delegate for Scientific Research. Over the past four years, Alessandro Del Maschio (at that time President’s Delegate for Scientific Research) promoted a course on Methodology of Scientific Research. It was held by Irene Floriani and Valter Torri, from the “Mario Negri” Institute (Milan) and then repeated in several Italian cities. The aim was to increase the level of knowledge in research methodology among Italian radiologists, a need which had already emerged during the first multicenter studies promoted by the Italian Society of Medical Radiology (SIRM) on breast MR imaging. My involvement only enlarged the scale of the audience by introducing several radiologists and a young physicist (the second author of this book) to the faculty. We all worked together in the preparation of the lessons during multiple meetings and long discussions, in particular with Irene Floriani and Valter Torri. This was a new stimulus for real- izing my dream. So they too deserve my heartfelt thanks. However, something was still missing: I had no solid mathematical background. Giving prominence to logics over computing could not exempt me from formal correctness. I therefore decided to associate a clever physicist from the Naples School and full-time researcher at the Radiology Unit of the IRCCS Policlinico San Donato, Giovanni Di Leo, with the project of this book. Both of us worked on all the chapters, even though he drafted the first version of the chapters with a higher mathematical content while I drafted the first version of the chapters with a higher logical and methodologic content, with each of us providing the other with constructive criticism. Lastly, I would like to thank Antonella Cerri from Springer. She enthusiastically latched on to the idea of this book when I described the project to her several years ago during a friendly chat at the end of a meeting of the editorial board of European Radiology. We really hope to communicate to radiologists the methodologic know-how which is taking on an increasingly important role. Many years ago a surgeon asked me: “Do you know what the difference is between a radiologist and a surgeon?”. Before I could answer, he said: “You say ‘my CT scanner’, I say ‘my patient’”. It was true. We must give clear demonstrations that the images Preface to the Italian Edition XIII of higher and higher quality we are able to produce have a significant impact on patient outcome and the population health status. This book is a small contribution towards this challenge. As I stated at the beginning of this preface, younger colleagues could heed the advice from this personal history. When you wake up in the morning, keep on dreaming. Then sooner or later, flesh out those dreams and bring them to life.

San Donato Milanese, April 2008 Francesco Sardanelli Preface to the English Edition

We enthusiastically accepted the proposal from Springer to do an English version of this book, based on the advantage that radiologists (and more generally experts in medical imaging) were unable to find a volume where the basics of research methodology were presented as applied to diagnostic imaging. Conversely, we had the large disadvantage related to the difficulty of explain- ing complex matters such as biostatistics which have been extensively developed in many other splendid books written by real experts in the field. However, insofar as we went ahead in rewriting the text in English, we realized that not only was the meaning retained, but the message also became clearer and less redundant. This was probably due not only to the effect of the different language, but also the result of a rethinking of the content of chapters and paragraphs several months after publishing the Italian edition. Now, our general impression is that the Italian version has been written for ourselves (to hone our thinking, to render it more analytic and detailed, to better understand the subject matter) and that the English version has been written for the reader (to provide her/him with a clearer message). We hope that this is true. Obviously, small errors and imperfections have been corrected and some points specifically written for Italian radiologists have been omitted. The major change made in this English version is an expanded Introduction with more emphasis on evidence-based medicine and evidence based radiology. At any rate, we would like to emphasize that this book is nothing more than an introduction to the topic, a portal to the realm of research methodology, with the words “radiology and medical imaging” emblazoned upon it. A sincere word of thanks to Alexander Cormack, the English copyeditor who had the patience to transform our text into real English.

San Donato Milanese, October 2008 Francesco Sardanelli Giovanni Di Leo

Acknowledgements

A sincere word of thanks to: – all the faculty members of the SIRM Course on Scientific Methodology, who cooperated with Irene Floriani, Valter Torri, and the two authors: Giuseppe Brancatelli, Laura Crocetti, Antonella Filippone and Roberto Carlo Parodi; – Lorna Easton who searched for the impact factors of the medical journals reported in the tables in Chapter 10; – Dr. Francesco Secchi for the systematic evaluation of the instructions for authors of the radiologic journals; – Dr. Myriam G. Hunink (Erasmus University Medical Center, Rotterdam, The Netherlands) for the suggestions used in the Introduction to this English version; – the authors and publishers who gave permission for the reproduction of tables and figures from previously published papers; – Elisabetta Ostini for the layout of text, tables, and figures.

Contents

Introduction ...... 1 Evidence-Based Medicine (EBM) ...... 1 Delayed Diffusion of EBM in Radiology and Peculiar Features of Evidence-Based Radiology ...... 5 Health Technology Assessment in Radiology and Hierarchy of Studies on Diagnostic Tests ...... 7 Why do we Need Biostatistics? ...... 11 The Structure of this Book ...... 13 References ...... 15

1. Diagnostic Performance ...... 19 1.1. The Results of an Examination Compared to a Reference Standard ...... 20 1.2. Measures of Diagnostic Performance ...... 21 1.3. Sensitivity, Specificity, FN Rate and FP Rate ...... 22 1.4. Predictive Values, Diagnostic Accuracy and Disease Prevalence ...... 25 1.5. Bayes’ Theorem, Likelihood Ratios and Graphs of Conditional Probability ...... 32 1.6. Cutoff and ROC Curves ...... 36 References ...... 40

2. Variables and Measurement Scales, Normal Distribution, and Confidence Intervals ...... 41 2.1. Variables and Measurement Scales ...... 42 2.1.1. Categorical Variables ...... 42 XX Contents

2.1.2. Discrete Numerical Variables ...... 43 2.1.3. Continuous Numerical Variables ...... 43 2.1.4. Measurement Scales ...... 44 2.2. Gaussian Distribution ...... 45 2.3. Basics of Descriptive Statistics ...... 51 2.3.1. Measures of Central Tendency ...... 51 2.3.2. Data Spread about the Measurement of Central Tendency: Variance and Standard Deviation ...... 54 2.4. Standard Error of the Mean ...... 56 2.5. Standard Error of the Difference between Two Sample Means . . 59 2.5.1. Paired Data ...... 61 2.6. Confidence Intervals ...... 61 2.7. Confidence Interval of a Proportion ...... 63 References ...... 64

3. Null Hypothesis, Statistical Significance and Power ...... 67 3.1. Null Hypothesis and Principle of Falsification ...... 68 3.2. Cutoff for Significance, Type I or α Error and Type II or β Error ...... 70 3.3. Statistical Power ...... 71 3.4. Why 0.05? ...... 74 3.5. How to Read a p Value ...... 75 References ...... 76

4. Parametric Statistics ...... 77 4.1. The Foundations of Parametric Statistics ...... 78 4.2. Comparison between Two Sample Means: Student’s t Test . . . 80 4.2.1. The Link with Confidence Intervals ...... 85 4.3. Comparing Three or More Sample Means: the Analysis of Variance ...... 86 4.3.1. ANOVA for Independent Groups ...... 87 4.3.2. ANOVA for Paired Data ...... 89 4.4. Parametric Statistics in Radiology ...... 91 References ...... 92

5. Non-Parametric Statistics ...... 93 5.1. One Sample with Two Paired Measurements ...... 94 5.1.1. Variables Measured with Dichotomous Scale ...... 94 5.1.2. Variables Measured with Ordinal Scales ...... 98 5.1.3. Variables Measured with Interval or Rational Scales . . . 100 5.2. Two Independent Samples ...... 101 5.2.1. Variables Measured with Nominal or Ordinal Scales . . . 101 5.2.2. Variables Measured with Interval or Rational Scales . . . 102 5.3. Three or More (k) Dependent Samples ...... 103 5.3.1. Variable Measured with Dichotomous Scale ...... 103 Contents XXI

5.3.2. Variables Measured with Ordinal. Interval or Rational Scale ...... 104 5.4. Three or More (k) Independent Samples ...... 104 5.4.1. Variables Measured with Nominal or Ordinal Scale . . . 104 5.4.2. Variables Measured with Interval or Rational Scale . . 105 5.5. Some Considerations Regarding Non-Parametric Tests ...... 106 References ...... 107

6. Linear Correlation and Regression ...... 109 6.1. Association and Causation ...... 109 6.2. Correlation between Continuous Variables ...... 111 6.3. Interpreting the Correlation Coefficient ...... 113 6.4. Test for Significance ...... 115 6.5. Rank Correlation ...... 116 6.6. Linear Regression ...... 118 6.6.1. Coefficients for Linear Regression ...... 119 6.7. Interpreting the Regression Line ...... 122 6.8. Limitations of the Use of the Regression Line ...... 124 References ...... 124

7. Reproducibility: Intraobserver and Interobserver Variability ...... 125 7.1. Sources of Variability ...... 125 7.2. Why do we Need to Know the Variability of Measurements? ...... 128 7.3. Intraobserver and Interobserver Variability for Continuous Variables: the Bland-Altman Analysis ...... 129 7.4. Interpreting the Results of Bland-Altman Analysis ...... 134 7.5. Intra- and Interobserver Variability for Categorical Variables: the Cohen k ...... 136 References ...... 140

8. Study Design, Systematic Reviews and Levels of Evidence ...... 141 8.1. Phases 1, 2, 3, and 4 of Pharmacologic Research ...... 142 8.2. Study Classification ...... 144 8.3. Experimental Studies and Control Group ...... 145 8.4. Observational Studies ...... 148 8.5. Randomized Controlled Studies: Alternative Approaches . . . 149 8.6. Studies on Diagnostic Performance: Classification ...... 150 8.7. Randomization and Minimization ...... 153 8.8. Sample Size ...... 155 8.9. Systematic Reviews (Meta-analyses) ...... 159 8.10. Levels of Evidence ...... 160 References ...... 163 XXII Contents

9. Bias in Studies on Diagnostic Performance ...... 165 9.1. Classification ...... 166 9.2. Bias Affecting External Validity ...... 167 9.2.1. Study Design ...... 168 9.2.2. Subject Selection ...... 170 9.2.3. Radiologic Methods and Reference Standard ...... 173 9.2.3.1. Diagnostic Technology (Technologic Obsolescence) ...... 173 9.2.3.2. Imaging Protocol ...... 174 9.2.3.3. Imaging Analysis ...... 174 9.2.3.4. Reader Training and Experience ...... 174 9.2.3.5. Reference Standard ...... 174 9.2.4. Statistical Analysis ...... 175 9.3. Bias Affecting Internal Validity ...... 175 9.3.1. Protocol Application ...... 175 9.3.2. Reference Standard Application ...... 176 9.3.3. Data Measurement ...... 176 9.3.4. Reader Independence ...... 178 9.4. A Lot of Work to Be Done ...... 178 References ...... 179

10. How to Write a Radiologic Paper ...... 181 10.1. Major Papers, Minor Papers, Invited Papers ...... 182 10.2. Which Medical Journal? ...... 184 10.3. Do we Always Need Institutional Review Board Approval and Informed Consent? ...... 201 10.4. Title, Running Title and Title Page ...... 202 10.5. Four-section Scheme, Section Size and Editing Sequence . . . 203 10.6. «Introduction»: Why did you do it? ...... 204 10.7. «Materials and Methods»: What did you do and how did you do it? ...... 205 10.8. «Results»: What did you Find? ...... 209 10.9. «Discussion»: What is the Meaning of your Findings? . . . . 210 10.10. «References» ...... 211 10.11. «Abstract» and «Keywords» ...... 212 10.12. Shared Rules ...... 213 10.13. Other Recommendations ...... 213 10.14. Dealing with the Editor’s Response and the Reviewers’ Opinions ...... 215 10.15. To Conclude ...... 218 References ...... 219

Subject and Noun Index ...... 221 Introduction

The practice of evidence-based medicine means integrating individual clinical expertise with the best available external evidence from systematic research.

DAVE L. SACKETT

The creative principle of science resides in mathematics.

ALBERT EINSTEIN

After all, to understand is the intrinsic purpose of science, and science is really much more than mechanical computing.

ROGER PENROSE

Evidence-Based Medicine (EBM)

Over the past three decades, the following view has gained increasing favor throughout the medical community: clinical practice should be based on the critical evaluation of the results obtained from medical scientific research. Today this evaluation is greatly favored by Internet which provides instanta- neous online access to the most recent studies even before they appear in print form. The possibility of instantaneously accessing quality-filtered and relevance-filtered secondary publications (meta-analyses, systematic reviews, and guidelines) has become real in routine practice. This notion – a clinical practice based on the results (the evidence) given by Evidence-based medicine (EBM) the research – has engendered a discipline: evidence-based medicine (EBM), also referred to as evidence-based healthcare,orevidence-based practice [MALONE, 2007]. In this context the term evidence is more closely associated with the concepts of proof, demonstration, or testability than simply with visi- bility or clarity. In fact, the general meaning of the new discipline suggests a clinical practice no longer based on bequeathed knowledge, on opinions, impressions, and perceptions, but on demonstrable proofs. EBM has been defined as “the systematic application of the best evidence to evaluate the available options and decision making in clinical management and policy settings”, i.e. “integrating clinical expertise with the best available external clinical evidence from research” [EVIDENCE-BASED RADIOLOGY WORKING GROUP, 2001].

F. Sardanelli, G. Di Leo, Biostatistics for Radiologists, © Springer-Verlag Italia 2009. 2 Biostatistics for Radiologists

Origins of EBM This concept is not new. The basis for this way of thinking was developed in the 19th century (Pierre C.A. Luis) and during the 20th century (Ronald A. Fisher, Austin Bradford Hill, Richard Doll, and Archie Cochrane). However, it was not until the second half of the last century that the Canadian School led by Gordon Guyatt and Dave L. Sackett at McMaster University (Hamilton, Ontario, Canada) promoted the tendency to guide clinical practice using the best results − the evidence − produced by scientific research [EVIDENCE-BASED RADIOLOGY WORKING GROUP, 2001; Greenhalgh, 2006a]. This approach was subsequently refined also by the Center for Evidence-Based Medicine (CEBM) at University of Oxford, England [CENTRE FOR EVIDENCE-BASED MEDICINE (http://cebm.net); MALONE, 2007]. Dave L. Sackett and coworkers stated that: EBM definitions Evidence based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence-based medicine means integrating individual clinical expertise with the best available external evidence from systematic research [SACKETT ET AL, 1996]. A highly attractive alternative but more technical definition, explicitly including diagnosis and investigation, has been proposed by Anna Donald and Trisha Greenhalgh: Evidence-based medicine is the use of mathematical estimates of the risk, of benefit and harm, derived from high-quality research on population samples, to inform clinical decision making in the diagnosis, investigation or management of individual patients [GREENHALGH, 2006b]. Patient’s values and choice However, EBM is not only the combination of current best available external evidence and individual clinical expertise. A third factor must be included in EBM: the patient’s values and choice. “It cannot result in slavish, cookbook approaches to individual patient care” [SACKETT ET AL, 1996]. Thus, EBM is the integration of: (i) research evidence; (ii) clinical expertise; and (iii) patient’s values and preferences [SACKETT ET AL, 1996; HUNINK ET AL, 2001; MALONE AND STAUNTON, 2007]. Clinical expertise “decides whether the external evidence applies to the individual patient”, evaluating “how it matches the patient’s clinical state, predicament, and preferences” [SACKETT ET AL, 1996]. A synopsis of this process is given in Figure 0.1.

Two general methods are generally proposed for applying EBM [DODD, 2007; MALONE AND STAUNTON, 2007; VAN BEEK AND MALONE, 2007] (Figure 0.2):

Top-down EBM – the top-down method, when academic centers, special groups of experts on behalf of medical bodies, or specialized organizations (e.g. the Cochrane col- laboration; http://www.cochrane.org) provide high-quality primary studies (original research studies), systematic reviews and meta-analyses, applications of decision analysis, or issue evidence-based guidelines and make efforts to put them into practice; Bottom-up EBM – the bottom-up method, when practitioners or other physicians working in routine practice are able “to ask a question, search and appraise the literature, and then apply best current evidence in a local setting”, opening a so-called audit cycle. Introduction 3

Figure 0.1. The general scheme of evidence based medicine. See Figure 0.2 for the top-down and bottom-up approaches to the best external evidence.

Figure 0.2. Top-down and bottom-up processes for evidence based medicine. *Appropriateness criteria are not included in the top-down EBM method since they are based on expert opinion, even though formalized procedures (such as the Delphi protocol) are frequently used and experts commonly base their opinion on systematic reviews and meta-analyses [MEDINA AND BLACKMORE, 2007]. EBM = Evidence Based Medicine. 4 Biostatistics for Radiologists

We should note that the top-down method involves a small number of peo- ple considered experts and does not involve physicians acting at the local level. However, there is a difference between the production of systematic reviews and meta-analyses (which are welcome as an important source of information by local physicians who want to practice the bottom-up model) and the production of guidelines which could be considered as an external cookbook (mistaken for a mandatory standard of practice) by physicians who feel themselves removed from the decision-making process [VAN BEEK AND MALONE, 2007]. On the other hand, the bottom-up method (which was considered an EBM method before the top-down method [HOLLINGWORTH AND JARVIK, 2007]) implies a higher level of knowledge of medical research methodology and EBM techniques by local physicians than that demanded by the top-down method. In either case, a qualitative improvement in patient care is expected. At any rate, clinical expertise must play a pivotal role as integra- tor of external evidence and patient’s values and choice. When decision analyses, meta-analyses and guidelines provide only part of the external evidence found by the local physicians, the two models act together, as hopefully should happen in practice. Moreover, a particular aim of the top-down method is the identification of gaps in knowledge to be filled by future research. In this way, EBM becomes a method for redirecting medical research towards improved medical practice [HOLLINGWORTH AND JARVIK, 2007].

EBM limitations However, EBM is burdened by limitations and beset by criticisms. It has been judged as unproven, very time-consuming (and therefore expensive), narrowing the research agenda and patients’ options, facilitating cost cutting, threatening pro- fessional autonomy and clinical freedom [SACKETT ET AL, 1996; TRINDER, 2000; MALONE AND STAUNTON, 2007]. At an objective evaluation, these criticisms seem to be substantially weak due to the pivotal role attributed to “individual clinical expertise” by EBM and to the general EBM aim “to maximize the quality and quantity of life for the individual patient” which “may raise rather than lower the cost of their care” as pointed out by Dave L. Sackett in 1996 [SACKETT ET AL, 1996]. Other limitations seem to be more relevant. On the one hand, large clinical areas – radiology being one of them – have not been sufficiently explored by studies according to EBM criteria. On the other hand, real patients can be total- ly different from those described in the literature, especially due to the presence of comorbidities, making the conclusions of clinical trials not directly applicable. This event is the day-to-day reality in geriatric medicine. The ageing population in Western countries has created a hard benchmark for EBM. These limitations may be related to a general criticism which suggests that the central feature in the EBM perspective is the patient population and not the individual patient [TONELLI, 1998; RAYMOND AND TROP, 2007]. Lastly, we should avoid unbridled enthusiasm for clinical guidelines, especially if they are issued with questionable methods [WOOLF ET AL, 1999]. However, all these limitations appear more as problems due to a still limited development and application of EBM than intrinsic EBM limitations. Basically, the correctness of EBM should be borne in mind, in that EBM aims to provide the best choice for the individual real patient with the use of proba- bilistic reasoning. EBM is investing significant effort towards improving con- temporary medicine. Introduction 5

Delayed Diffusion of EBM in Radiology and Peculiar Features of Evidence-Based Radiology Radiology is not outside of EBM, as stated by David L. Sackett and coworkers Evidence-based in 1996: “EBM is not restricted to randomised trials and meta-analyses […]. To radiology (EBR) find out about the accuracy of a diagnostic test, we need to find proper cross sectional studies of patients clinically suspected of harboring the relevant dis- order, not a randomised trial” [SACKETT ET AL, 1996]. Evidence-based radiology (EBR), also called evidence-based imaging, first appeared in the literature only in recent years. Until 2000, few papers on EBR were published in nonradiologic journals EBR delay [ACHESON AND MITCHELL, 1993; NO AUTHORS LISTED (British Columbia Office of Health Technology Assessment), 1997; NO AUTHORS LISTED, Int J Assess Health Care, 1997; DIXON, 1997; MUKERJEE, 1999] and in one journal specialized in dentomaxillofacial radiology [LIEDBERG ET AL, 1996]. From 2001 to 2005, several papers introduced the EBM approach in radiology [EVIDENCE- BASED RADIOLOGY WORKING GROUP, 2001; TAÏEB AND VENNIN, 2001; ARRIVÉ AND TUBIANA, 2002; BUI ET AL, 2002; GUILLERMAN ET AL, 2002; KAINBERGER ET AL, 2002; BENNETT, 2003; BLACKMORE, 2003; COHEN ET AL, 2003; GOERGEN ET AL, 2003; MEDINA ET AL, 2003; BLACKMORE, 2004; DODD ET AL, 2004; ERDEN, 2004; GILBERT ET AL, 2004; MATOWE AND GILBERT, 2004; GIOVAGNONI ET AL, 2005]. Not until 2006 was the first edition of the book entitled Evidence- Based Imaging published by L. Santiago Medina and C. Craig Blackmore [MEDINA AND BLACKMORE, 2006]. The diffusion of EBM in radiology was delayed. From this viewpoint, radiology is “behind other specialties” [MEDINA AND BLACKMORE, 2007]. As a matter of fact, according to L. Santiago Medina and C. Craig Blackmore, “only around 30% of what constitutes ‘imaging knowledge’ is substantiated by reliable scientific inquiry” [MEDINA AND BLACKMORE, 2006]. Other authors esti- mate that less than 10% of standard imaging procedures is supported by suffi- cient randomized controlled trials, meta-analyses or systematic reviews [DIXON, 1997; RCR WORKING PARTY, 1998; KAINBERGER ET AL, 2002]. The EBR delay may also be linked to several particular traits of our discipline. Particular traits of radiology In fact, the comparison between two diagnostic imaging modalities is markedly different from the well-known comparison between two treatments, typically between a new drug and a placebo or standard care. Thus, the classic design of the randomized controlled trial is not the standard for radiologic studies. What are the peculiar features of radiology which need to be considered? First of all, the evaluation of the diagnostic performance of imaging modal- Tecnical expertise ities must be based on a deep insight of the technologies used for image gener- ation and postprocessing. Technical expertise has to be combined with clinical expertise in judging when and how the best available external evidence can be applied in clinical practice. This aspect is just as important as “clinical expertise” (knowledge of indications for an imaging procedure, imaging interpretation and reporting, etc). Dodd and coworkers showed the consequences of ignoring a technical detail such as slice thickness in evaluating the diagnostic performance of magnetic resonance (MR) cholangiopancreatography: using a 3-mm instead of a 5-mm thickness, the diagnostic performance for the detection of choledocholithiasis changed from 0.57 sensitivity and 1.0 specificity to 6 Biostatistics for Radiologists

0.92 sensitivity and 0.97 specificity [DODD ET AL, 2004]. If the results of tech- nically inadequate imaging protocols are included in a meta-analysis, the consequence will be the underestimation of diagnostic performance. At times progress in clinical imaging is essentially driven by the development of new technology, as was the case for imaging at the beginning of the 1980s. However, more frequently, an important gain in spatial or temporal resolution, in signal-to-noise or contrast-to-noise ratio is attained through hard- ware and/or software innovations in pre-existing technology. This new step broadens the clinical applicability of the technology, as was the case for computed tomography (CT) which evolved from helical single-slice to multidetec- tor row scanners, thus opening the way to cardiac CT and CT angiography of the coronary arteries. Keeping up to date with technologic development is a hard task for radiologists, and a relevant part of the time not spent with imaging interpretation should be dedicated to the study of new imaging modalities or techniques. For radiologic research, each new technology appearing on the market should be tested with studies on its technical performance (image resolution, etc.). Reproducibility Second, we need to perform studies on the reproducibility of the results of imaging modalities (intraobserver, interobserver, and interstudy variability), an emerging research area which requires dedicated study design and statistical methods (e.g. Cohen k statistics and Bland-Altman analysis). In fact, if a test shows poor reproducibility, it will never provide good diagnostic performance, i.e. sensitivity and specificity. Good reproducibility is a necessary (but not suf- ficient) condition for a test to be useful. High speed of technologic Third, the increasing availability of multiple options in diagnostic imaging evolution should be taken into consideration along with their continuous and sometimes unexpected technologic development and sophistication. Thus, the high speed of technologic evolution has created not only the need to study theory and practical applications of new tools, but also to repeatedly start with studies on technical performance, reproducibility, and diagnostic performance. The faster the advances in technical development, the more difficult it is to do the job in time. This development is often much more rapid than the time required for performing clinical studies for the basic evaluation of diagnostic performance. From this viewpoint, we are always too late with our assessment studies. From images to patients However, the most important problem to be considered with new diagnostic technology is that “a balance must be struck between apparent (e.g. diagnostic) benefit and real benefit to the patient” [DIXON, 1997]. In fact, a qualitative leap in radiologic research is now expected: from the demonstration of the increasing ability to see more and better, to the demonstration of a significant change in treatment planning or, at best, a significant gain in patient health and/or quality of life – the patient outcome. The ALARA principle Lastly, we should specifically integrate a new aspect in EBR, i.e. the need to avoid unnecessary exposure to ionizing radiation, according to the as low as reasonably achievable (ALARA) principle [NO AUTHORS LISTED, Proceedings of the Second ALARA Conference, 2004; PRASAD ET AL, 2004; SEMELKA ET AL, 2007] and to governmental regulations [COUNCIL OF THE EUROPEAN UNION, 1997; BARR ET AL, 2006; FDA RADIOLOGICAL HEALTH PROGRAM, 2008]. The ALARA principle might be considered as embedded in radiologic “technical and clinical expertise”. However, in our opinion, it should be regarded as a Introduction 7

Figure 0.3. The process of evidence based radiology. ALARA = “as low as reasonably achievable”, with reference to ionizing radiation exposure.

fourth dimension of EBR, due to the increasing relevance of radioprotection issues in radiologic thinking and practice. A graphical representation of the EBR process, including the ALARA principle, is provided in Figure 0.3.

Health Technology Assessment in Radiology and Hierarchy of Studies on Diagnostic Tests

In the framework described above, EBM and EBR are based on the possibility Health technology assessment of obtaining the best external evidence for a specific clinical question. Now the (HTA) in radiology question is: how is this evidence produced? In other words, which methods should be used to demonstrate the value of a diagnostic imaging technology? This field is what we name health technology assessment (HTA) and particular features of HTA are important in radiology. Thus, EBR may exist only if a good radiologic HTA is available. As stated by William Hollingworth and Jeffry G. Jarvik, “the tricky part, as with boring a tunnel through a mountain, is making sure that the two ends meet in the middle” [HOLLINGWORTH AND JARVIK, 2007]. According to the United Kingdom HTA Programme, HTA should answer the following four fundamental questions on a given technology [WHITE ET AL, 2000; HOLLINGWORTH AND JARVIK, 2007]:

1. does it work? 2. for whom? 3. at what cost? 4. how does it compare with alternatives? 8 Biostatistics for Radiologists

Efficacy, effectiveness In this context, increasing importance has been gained by the use of three dif- and efficiency ferent terms. While efficacy reflects the performance of medical technology under ideal conditions, effectiveness evaluates the same performance under ordinary conditions, and efficiency measures the cost-effectiveness [HILLMAN AND GATSONIS, 2008]. In this way the development of a procedure in specialized or academic centers is distinguished by its application to routine clinical practice and from the inevitable role played by the economic costs associated with implementation of a procedure. Hierarchy of studies To evaluate the impact of the results of studies, i.e. the level at which the on diagnostic tests HTA was performed, we need a hierarchy of values. Such a hierarchy has been proposed for diagnostic tests and also accepted for diagnostic imaging modalities. During the 1970s, the first classification proposed five levels for the analysis of the diagnostic and therapeutic impact of cranial CT [FINEBERG ET AL, 1977]. By the 1990s [FRYBACK AND THORNBURY, 1991], this classification had evolved into a six-level scale, thanks to the addition of a top level called societal impact [THORNBURY, 1994; MACKENZIE AND DIXON, 1995; THORNBURY, 1999]. A description of this scale was presented more recently in the radiologic literature [EVIDENCE-BASED RADIOLOGY WORKING GROUP, 2001; SUNSHINE AND APPLEGATE, 2004]. The six-level scale This six-level hierarchy scale (Table 0.1) is currently widely accepted as a foundation for HTA of diagnostic tools. This framework provides an opportunity to assess a technology from differing viewpoints. Studies on technical performance (level 1) are of key importance to the imaging community and the evaluation of diagnostic performance and reproducibility (level 2) are the basis for adopting a new technique by radiologists and clinicians. However, radiologists and clinicians are also interested in how an imaging technique impacts patient management (levels 3 and 4) and patient outcomes (level 5) while healthcare providers wish to ascertain the costs and benefits of reimbursing a new technique, from a societal perspective (level 6). Governments are mainly concerned about the societal impact of new technologies in comparison to that of other initiatives they may be considering.

A one-way logical chain Note that this hierarchical order is a one-way logical chain. A positive effect at any level generally implies a positive effect at all preceding levels but not vice versa [HOLLINGWORTH AND JARVIK, 2007]. While a new diagnostic technology with a positive impact on patient outcome probably has a better technical performance, higher diagnostic accuracy, etc. compared with the standard technology, there is no certainty that a radiologic test with a higher diagnostic accuracy results in a better patient outcome. If we have demonstrated an effective diagnostic performance of a new test (level 2), the impact on a higher level depends on the clinical setting and frequently also on conditions external to radiology. This must be demonstrated with specifically designed studies. As a matter of fact, we might have a fantastic test for the early diagnosis of disease X but, if no therapy exists for that disease, no impact on patient outcomes can be obtained. HTA should examine the link between each level and the next in the chain of this hierarchy to establish the clinical value of a radiologic test. Cost-effectiveness in HTA Cost-effectiveness should be included in HTA at any level of the hierarchic scale as cost per examination (level 1), per correct diagnosis (level 2), per invasive test avoided (level 3), per changed therapeutic plan (level 4), per gained Introduction 9

Table 0.1. Hierarchy of studies on diagnostic tests Level Parameters under investigation

6. Societal impact Cost-benefit and cost-effectiveness analysis from a social perspective 5. Patient outcomes Fraction of patients improved with the test compared with fraction improved without the test; difference in morbidity between the patients with the test and those without the test; gain in quality-adjusted life years (QALYs) obtained by the patients with the test compared with those without the test 4. Therapeutic impact Fraction of patients for whom the test is judged useful for treatment planning or for whom the treatment planning is modified on the basis of the information supplied by the test 3. Diagnostic impact Fraction of patients for whom the test is judged useful for reaching the diagnosis or for whom the diagnosis is substantially modified after the test; positive and negative likelihood ratios 2. Diagnostic performance Sensitivity, specificity, accuracy, positive predictive value, negative predictive value and receiver operator characteristic (ROC) analysis; intraobserver, interobserver and interstudy reproducibility 1. Technical performance Gray scale range; modulation transfer function change; sharpness; spatial resolution, in-plane (line pairs per mm, pixel size) and through-the- plane (slice thickness), integrated in voxel size; signal-to-noise ratio; contrast resolution (contrast-to-noise ratio); time resolution (images/sec) etc

Sources: THORNBURY, 1994; SUNSHINE AND APPLEGATE, 2004; with modifications. In particular, reproducibility studies were added at level 2.

quality-adjusted life expectancy or per saved life (levels 5-6) [HOLLINGWORTH AND JARVIK, 2007]. New equipment or a new imaging procedure should have extensive HTA assessment before it is adopted in routine practice. Thereafter a period of clinical evaluation follows where diagnostic accuracy is assessed against a known gold standard. Indeed, the radiologic literature is mainly composed of level 1 (technical performance) and level 2 (diagnostic performance) studies. This is partly inevitable. The evaluation of the technical and diagnostic performance of medical imaging is a typical function of radiologic research. However, radiologists less frequently study the diagnostic impact (level 3) or therapeutic impact (level 4) of medical imaging, while outcome (level 5) and societal impact (level 6) analysis is positively rare in radiologic research. A “shortage Shortage of scientific of coherent and consistent scientific evidence in the radiology literature” to be evidence in radiology used for a wide application of EBR was noted in 2001 [EVIDENCE-BASED RADIOLOGY WORKING GROUP, 2001]. In recent years, several papers have appeared exploring levels higher than those concerning technical and diagnostic performance, such as the Scottish Low Back Pain Trial, the DAMASK study, and others [GILBERT FJ ET AL, 2004; BREALEY ET AL, for the DAMASK TRIAL TEAM, 2007; OEI ET AL, 2008; OUWENDIJK ET AL, 2008]. This lack of evidence on patient outcomes is a void also for well established technologies. This is the case for cranial CT for head injuries, even though the diagnostic information yielded by CT was “obviously so much better than [that] of alternative strategies that equipoise (genuine uncertainty about the efficacy of 10 Biostatistics for Radiologists

a new medical technology) was never present” and “there was an effective treatment for patients with subdural or epidural hematomas – i.e. neurosurgical evac- uation” [HOLLINGWORTH AND JARVIK, 2007]. However, cases like this are very rare, and “in general, new imaging modalities and interventional procedures should be viewed with a degree of healthy skepticism to preserve equipoise until evidence dictates otherwise” [HOLLINGWORTH AND JARVIK, 2007]. This urgent problem was recently highlighted by Christiane K. Kuhl and coworkers for the clinical value of 3.0-T MR imaging. They state: “Although for most neurologic and angiographic applications 3.0 T yields technical advantages compared to 1.5 T, the evidence regarding the added clinical value of high-field strength MR is very limited. There is no paucity of articles that focus on the technical evaluation of neurologic and angiographic applications at 3.0 T. This technology-driven science absorbs a lot of time and energy – energy that is not available for research on the actual clinical utility of high- field MR imaging” [KUHL ET AL, 2008]. The same can be said for MR spectroscopy of brain tumors [JORDAN ET AL, 2003; HOLLINGWORTH AND JARVIK, 2007], with only one [MÖLLER-HARTMANN ET AL, 2002] of 96 reviewed articles evaluating the additional value of MR spectroscopy which compared this technology with MR imaging alone. Reasons for shortage of high There are genuine reasons for rarely attaining the highest impact levels of level radiologic studies efficacy by radiologic research. On the one hand, increasingly rapid technologic development forces an endless return to low impact levels. Radiology was judged as the most rapidly evolving specialty in medicine [DIXON, 1997]. On the other hand, level 5 and 6 studies entail long performance times, huge economic costs, a high degree of organization and management for longitudinal data gathering on patient outcomes, and often require a randomized study design (by way of example, the average time for 59 studies in radiation oncology up to publication of the results reviewed in 2005 was about 11 years [SOARES ET AL, 2005]). In this setting, there are two essential needs: full cooperation with clinicians who manage the patient before and after a diagnostic examination, and methodologic and statistical expertise regarding randomized controlled trials. Radiologists should not be afraid of this, as it is not unfamil- iar territory for radiology. More than three decades ago, mammographic screening created a scenario in which the early diagnosis by imaging con- tributed to a worldwide reduction in mortality from breast cancer, with a high societal impact. Lastly, alternatives to clinical trials and meta-analyses exist. They are the so- called “pragmatic” or “quasi-experimental” studies and “decision analysis”. Pragmatic studies A pragmatic study proposes the concurrent development, assessment, and implementation of new diagnostic technologies [HUNINK AND KRESTIN, 2002]. An empirically based study, preferably using controlled randomization, inte- grates research aims in clinical practice, using outcome measures reflecting the clinical decision-making process and acceptance of the new test. Outcome measures include: additional imaging studies requested; costs of diagnostic work-up and treatments; confidence in therapeutic decision-making; recruit- ment rate; and patient outcome measures. Importantly, time is used as a fundamental dimension, as an explanatory variable in data analysis to model the learning curve, technical developments, and interpretation skill. Limitations of this approach can be the need of dedicated and specifically trained personnel Introduction 11 and the related economic costs to be covered by presumably governmental agencies [JARVIK, 2002]. However, this seems to demonstrate the potential for responding to the dual demand of the increasing pace of technologic development in radiology and the need to attain higher levels of radiologic studies, thus in a single approach obtaining data on diagnostic confidence, effect on therapy planning, patient outcome measures and cost-effectiveness analysis. Decision analysis, based on deductive reasoning, tries to overcome the lim- Decision analysis ited external validity associated with clinical trials [HUNINK ET AL, 2001; LAUNOIS, 2003]. It is a tool for evaluating a diagnostic test on the basis of patient outcome using intermediate outcome measures such as sensitivity and specificity obtained by already published studies. Different diagnostic or therapeutic alternatives are visually represented by means of a decision tree and dedicated statistical methods are used (e.g. Markov model, Monte Carlo simulation) [PLEVRITIS, 2005; HUNINK ET AL, 2001]. This method is typically used for cost-effectiveness analysis. For instance, it was recently used for simulat- ing the effectiveness of mammography, MR imaging, or both for screening of breast cancer in women carriers of BRCA1 mutations [LEE ET AL, 2008]. A simple way to appraise the intrinsic difficulty in HTA of radiologic proce- Looking at the pharmacologic dures is to compare radiologic with pharmacologic research (see Chapter 8). research After the chemical discovery of an active molecule, its development, cell and animal testing, the phase I and phase II studies are carried out by the industry with the participation of very few clinicians (for phase I and II studies). Very few academic institutions and large hospitals are involved in this long phase (commonly about ten years). When clinicians become involved in phase III studies, i.e. large randomized trials for registration, the study aims have already reached level 5 (outcome impact). Radiologists have to climb 4 levels of impact before reaching the outcome level. Of course it is possible to imagine a world in which even radiologic procedures are tested for outcome endpoints before entering clinical practice, but the real world is different, such that we have much more technology-driven research from radiologists than radiologist-driven research on technology.

Why do we Need Biostatistics?

The application of EBM implies a fundamental difficulty. Not only producing scientific evidence but also reading and correctly understanding the medical literature, in particular summaries of the best results such as systematic reviews and meta-analyses, requires a basic knowledge of and confidence with the principles and techniques of biostatistics. In fact, this is the only way to quantify the uncertainty associated with biological variability and the changes brought about by the patient’s disease. This theoretical background is now emerging as very important expertise to be acquired by any physician of the new millennium. Quantification of data variability and its presentation comprise the field of descriptive statistics. This branch of statistics enables us to describe the sam- Descriptive statistics ple under investigation by summarizing its features with diagrams or graphs and various parameters (mean, standard deviation, median, etc.). The quantification of uncertainty is needed to understand the probability we have if we apply the results of a study to the general population from which the study sub- 12 Biostatistics for Radiologists

Inferential statistics jects were drawn, i.e. when we use inferential statistics. This allows us to propose a general view and a theoretical model of the phenomenon under investigation. In this way, we can anticipate future events, namely we can make an inference. This is a deduction which evaluates whether the results of a study on a sample size can be applied to the general population, with a controlled error probability. As a consequence, there is a close proximity between inferential statistics and probability theory. Although biostatistics uses mathematical tools, which may be very simple or quite sophisticated, the problem is never a question of simple mechanical computing (today many software packages can adequately do the job). It is rather a question of understanding the meaning of the figures we obtain and the way we obtain them, both theoretically (what precisely do we mean by specificity or likelihood ratio?) and practically, for clinical decision-making. Difference between Note that while a statistically significant result can be lacking clinical rel- statistical significance evance, clinically relevant evidence should be based on statistical signifi- and clinical relevance cance. A study can produce very high statistical significance without having any clinical utility. Who would use an anti-hypertensive drug which system- atically (i.e. in all subjects) reduces arterial pressure by 1 mmHg compared with standard treatment? On the other hand, the considerable effect of a new drug against a form of cancer, if real, will be demonstrated in a controlled study (i.e. compared with standard treatment) which shows a significant increase in the disease-free interval or survival time. In other words, the size of a statistically significant difference needs to be evaluated to conclude that it is also clinically relevant, while a clinically relevant difference, to become evidence, must produce a statistically significant difference in a high-quality study. A particular aspect plays a role in clinical radiologic research. Even for simple studies on diagnostic performance (sensitivity, specificity etc.), the common lack of assumptions needed for applying parametric statistical methods (based on the direct computing of measured data) makes nonparametric statistical methods (based on qualitative classes or ranks, or other tools) frequently needed. However, understanding nonparametric statistics requires preliminary knowledge of basic parametric statistics. There are several reasons for the prevalent use of nonparametric statistical methods in radiology. The most important are as follows: the frequent use of nominal scales of measurement, often simply dichotomous (positive or negative) or ordinal (a typical example is the Breast Imaging Reporting and Data ® System, BI-RADS , scale [AMERICAN COLLEGE OF RADIOLOGY, 2003]); the limited possibility of demonstrating the normal distribution of continuous numerical data in a small sample size (a necessary assumption for using parametric statistical methods); and the high frequency of a small sample size. As a consequence, most books concerning general medical statistics appear barely appropriate for radiologists. These texts commonly dedicate numerous pages to parametric methods and very few pages to nonparametric methods, and even when nonparametric methods are extensively explained, no specific reference to their use in diagnostic imaging is available. An exception to this trend in Italy is Guida alla Statistica nelle Scienze Radiologiche by Professor Guido Galli, a nuclear physician from the University of Rome School of Medicine [GALLI, 2002]. Introduction 13

The Structure of this Book

All these reasons explain the need for radiologists to possess knowledge in applied biostatistics. In the following chapters we will propose such knowledge, giving greater priority to logic than to computing. In Chapter one we describe the classic tools for the quantification of diag- Chapter 1: nostic performance typically used in radiologic studies: sensitivity, specifici- diagnostic performance ty, predictive values, overall accuracy, and receiver operator characteristic (ROC) curve. Moreover, we introduce here the likelihood ratios which quantify the power of a diagnostic test, i.e. the ability of the test to modify the disease (or non-disease) probability, up to now rarely used in radiologic studies. In this context, we show some aspects of probability theory and present Bayes’ theorem. In Chapter two we define the concept of variable and the different types Chapter 2: variables, of variables with reference to their scales of measurement, as well as some scales of measurement, essential principles of descriptive statistics, normal distribution, and confi- normal distribution, dence intervals. Indeed, understanding the scales of measurement is essen- confidence intervals tial for choosing a statistical test applicable to the data to be analyzed. Being familiar with normal distribution is a must for the use of all tools in biostatistics. Confidence intervals can be thought of as a conceptual and practical bridge between descriptive and inferential statistics: they define a range of variability in the results in the event the same study were to be repeated for a sample with the same size of patients having the same characteristics. An important trend in recent times is the increasing emphasis radiologic journals have been giving to confidence intervals. The presentation of the 95% confidence intervals should be considered mandatory for all indices of diagnostic performance. Chapter three is dedicated to the theory of the scientific experiment, namely Chapter 3: null hypothesis to the null hypothesis and statistical significance. This topic has the greatest and statistical significance philosophic and methodologic implications. We explain why the demonstration of an experimental hypothesis (e.g. that two diagnostic options have a different sensitivity for a given disease) must be obtained by working on an antithetical hypothesis (i.e. that there is no difference in sensitivity between the two diagnostic options) which we name null hypothesis. The researcher’s aim is to demonstrate that the null hypothesis is sufficiently improbable to accept the indirect conclusion that the experimental hypothesis is probably true. However, this conclusion is never demonstrated in a direct and definitive way. While in Chapter four we provide some of the essentials of parametric sta- Chapter 4: parametric statistics tistics and the assumptions required for the application of parametric statistical tests, in Chapter five we describe the most important nonparametric statis- Chapter 5: nonparametric tical tests and the assumptions needed for their application. statistics In Chapter six we define the concepts of association, correlation, and regres- Chapter 6: association, sion and propose the techniques for their quantification. Particular attention is correlation and regression paid to the differentiation between association or correlation between two variables and the deduction of the cause-effect relationship, the latter never being provable on the basis of a statistical calculation alone. In Chapter seven we present the most important techniques for evaluating the Chapter 7: reproducibility reproducibility of the result of a diagnostic test, either for continuous variables (Bland-Altman analysis), or for nominal and ordinal variables (Cohen k). Here 14 Biostatistics for Radiologists

we introduce the concept of intraobserver and interobserver variability. These kinds of studies are currently highly appreciated for their ability to define the practical role of old and new imaging techniques. Chapter 8: study design, In Chapter eight the reader will find the principal types of study in relation sample size, systematic reviews to their design (observational or randomized experimental; prospective or ret- (meta-analysis), rospective; longitudinal or transversal; etc.) as well as a general description of levels of evidence the methods for calculating the sample size, i.e. the number of patients which need to be enrolled in a prospective study in order to obtain an acceptable probability of demonstrating the experimental hypothesis. Here we also include a short section on systematic reviews, namely those studies which gather together the information contained in already published studies on a given topic, conduct a critical appraisal of the methods used in those studies, select the studies according to predefined quality standards, and pool the results of the selected studies to provide a new and more reliable overall result, using dedicated statistical methods (meta-analysis). Afterwards, we define the so-called levels of evidence of radiologic studies. Chapter 9: bias In Chapter nine we present a list (without doubt incomplete) of the errors to be avoided in radiologic studies. In other words, we list the potential sources of bias that should be recognized as readers and avoided or, at least, limited and explicitly acknowledged as authors. Chapter 10: recommendations Finally, in Chapter ten we provide a series of practical recommendations for for writing a radiologic paper writing a radiologic study, with particular reference to the content of the four sections of the body of the paper and its logical structure (Introduction, Materials and Methods, Results, and Discussion) and the two essential accom- panying items (Abstract and References). The subject matter of this book clearly falls short of exhaustively treating biostatistics in radiology, in part because radiology is transversally cross-linked What you will not find with all medical subspecialties. However, a number of statistical techniques in this book which can be used in radiologic research are not considered in this book. For example, the reader will not find suggestions for statistical and graphical methods for describing data; moreover, logistic regression, multiple regression, the concept of absolute and relative risk, survival curves, and non-inferiority studies are not treated. We have avoided these topics in order to produce a book capable of introducing biostatistics to radiologists. While it is only a preliminary approach to biostatistics, the volume does have the advantage of present- ing the topic from the particular viewpoint of radiology. All the examples are progressively numbered in each chapter and drawn from the radiologic literature or invented ad hoc to facilitate the reader’s understanding of the theoretical concepts. We recommend that the reader who has Do not skip the examples grasped a theoretical definition should not skip the examples, as they could be a useful aid for committing the theoretical concept to memory. Similarly, we advise the reader who is having difficulty coming to grips with the theoretical definition to go straight to the following example as this could immediately shed light on the theoretical problem. A final word of advice. Throughout the book the reader will find several Formulas mathematical formulas. These have been included in their entirety for the readers willing to understand the mechanism of computing. However, a thorough understanding of the formulas is by no means required to grasp the general sense of the concepts and their practical use. Introduction 15

It is far from our intention to educate radiologists so that they can replace Radiologists interacting statisticians, as this appears neither possible nor useful. Instead it is our aim with statisticians educate radiologists so that they may interact with statisticians with proficien- cy and critical judgment.

References

Acheson L, Mitchell L (1993) The routine antenatal diagnostic imaging with ultrasound study. The challenge to practice evidence-based obstetrics. Arch Fam Med 2:1229-1231 American College of Radiology (2003) ACR breast imaging reporting and data system (BI-RADS): Breast Imaging Atlas. American College of Radiology, Reston, Va Arrivé L, Tubiana JM (2002) “Evidence-based” radiology. J Radiol 83:661 Barr HJ, Ohlhaber T, Finder C (2006) Focusing in on dose reduction: the FDA perspective. AJR Am J Roentgenol 186:1716-1717 Bennett JD (2003) Evidence-based radiology problems. Covered stent treatment of an axillary artery pseudoaneurysm: June 2003-June 2004. Can Assoc Radiol J 54:140-143 Blackmore CC (2003) Evidence-based imaging evaluation of the cervical spine in trauma. Neuroimaging Clin N Am 13:283-291 Blackmore CC (2004) Critically assessing the radiology literature. Acad Radiol 11:134-140 Brealey SD; DAMASK (Direct Access to Magnetic Resonance Imaging: Assessment for Suspect Knees) Trial Team (2007) Influence of magnetic resonance of the knee on GPs’ decisions: a randomised trial. Br J Gen Pract 57:622-629 Bui AA, Taira RK, Dionisio JD et al (2002) Evidence-based radiology: requirements for electronic access. Acad Radiol 9:662-669 Centre for Evidence-Based Medicine, Oxford University, England. http://cebm.net Cohen WA, Giauque AP, Hallam DK et al (2003) Evidence-based approach to use of MR imaging in acute spinal trauma. Eur J Radiol 48:49-60 Council of the European Union (1997) Council Directive 97/43/Euratom of 30 June 1997 on health protection of individuals against the dangers of ionizing radiation in relation with medical exposure, and repealing Directive 84/466/Euratom. J Eur Commun L 180:22-27 (http://europa.eu.int/eur-lex/en/dat/1997/en_397L0043.htlm) Dixon AK (1997) Evidence-based diagnostic radiology. Lancet 350:509-512 Dodd JD (2007) Evidence-based practice in radiology: steps 3 and 4 – Appraise and apply diagnostic radiology literature. Radiology 242:342-354 Dodd JD, MacEneaney PM, Malone DE (2004) Evidence-based radiology: how to quickly assess the validity and strength of publications in the diagnostic radiology literature. Eur Radiol 14:915-922 Erden A (2004) Evidence based radiology Tani Girisim Radyol 10:89-91 Evidence-Based Radiology Working Group (2001) Evidence-based radiology: a new approach to the practice of radiology. Radiology 220:566-575 FDA Radiological Health Program (2008). Available at: http://www.fda.gov/cdrh/rad- health/index.html Fineberg HV, Bauman R, Sosman M (1977) Computerized cranial tomography. Effect on diagnostic and therapeutic plans. JAMA 238:224-227 Fryback DG, Thornbury JR (1991) The efficacy of diagnostic imaging. Med Decis Making 11:88-94 Galli G (2002) Guida alla statistica nelle scienze radiologiche. Ecoedizioni internazion- ali, Rome, Italy Gilbert FJ, Grant AM, Gillan MGC (2004) Low back pain: influence of early MR imaging or CT on treatment and outcome – multicenter randomized trial. Radiology 231:343-351 16 Biostatistics for Radiologists

Giovagnoni A, Ottaviani L, Mensà A et al (2005) Evidence based medicine (EBM) and evidence based radiology (EBR) in the follow-up of the patients after surgery for lung and colon-rectal carcinoma. Radiol Med 109:345-357 Goergen SK, Fong C, Dalziel K, Fennessy G (2003) Development of an evidence-based guideline for imaging in cervical spine trauma. Australas Radiol 47:240-246 Greenhalgh T (2006) How to read a paper. The basics of evidence-based medicine. 3rd ed. Blackwell, Oxford, England: ix-xii (a); 1-3 (b) Guillerman RP, Brody AS, Kraus SJ (2002) Evidence-based guidelines for pediatric imaging: the example of the child with possible appendicitis. Pediatr Ann 31:629-640 Hillman BJ, Gatsonis CA (2008) When is the right time to conduct a clinical trial of a diagnostic imaging technology? Radiology 248:12-15 Hollingworth W, Jarvik JG (2007) Technology assessment in radiology: putting the evidence in evidence-based radiology. Radiology 244:31-38 Hunink MG, Glasziou PP, Siegel JE et al (2001) Decision making in health and medicine: integrating evidence and values. Cambridge University Press, Cambridge, UK, 2001 Hunink MG, Krestin GP (2002) Study design for concurrent development, assessment, and implementation of new diagnostic imaging technology. Radiology 222:604-614 Jarvik JG (2002) Study design for the new millennium: changing how we perform research and practice medicine. Radiology 222:593-594 Jordan HS, Bert RB, Chew P et al (2003) Magnetic resonance spectroscopy for brain tumors. Agency for Healthcare Research and Quality, Rockville, MD:109 Kainberger F, Czembirek H, Frühwald F et al (2002). Guidelines and algorithms: strategies for standardization of referral criteria in diagnostic radiology. Eur Radiol 12:673-679 Kuhl CK, Träber F, Schild HH (2008) Whole-body high-field-strength (3.0-T) MR imaging in clinical practice. Part I. Technical considerations and clinical applications. Radiology 246:675-696 Launois R (2003) Economic assessment, a field between clinical research and observational studies. Bull Cancer 90:97-104 Lee JM, Kopans DB, McMahon PM et al (2008). Breast cancer screening in BRCA1 mutation carriers: effectiveness of MR imaging – Markov Monte Carlo decision analysis. Radiology 246:763-771 Liedberg J, Panmekiate S, Petersson A, Rohlin M (1996) Evidence-based evaluation of three imaging methods for the temporomandibular disc. Dentomaxillofac Radiol 25:234-241 Mackenzie R, Dixon AK (1995) Measuring the effects of imaging: an evaluative framework. Clin Radiol 50:513-518 Malone DE (2007) Evidence-based practice in radiology: an introduction to the series. Radiology 242:12-14 Malone DE, Staunton M (2007) Evidence-based practice in radiology: step 5 (evaluate) – Caveats and common questions. Radiology 243:319-328 Matowe L, Gilbert FJ (2004) How to synthesize evidence for imaging guidelines. Clin Radiol 59:63-68 Medina LS, Aguirre E, Zurakowski D (2003) Introduction to evidence-based imaging. Neuroimaging Clin N Am 13:157-165 Medina LS, Blackmore CC (2006) Evidence-based imaging. 1st edn. Springer, New York, NY Medina LS, Blackmore CC (2007) Evidence-based radiology: review and dissemina- tion. Radiology 244:331-336 Möller-Hartmann W, Herminghaus S, Krings T et al (2002) Clinical application of pro- ton magnetic resonance spectroscopy in the diagnosis of intracranial mass lesions. Neuroradiology 44:371-381 Mukerjee A (1999) Towards evidence based emergency medicine: best BETs from the Manchester Royal Infirmary. Magnetic resonance imaging in acute knee haemarthro- sis. J Accid Emerg Med 16:216-217 Introduction 17

No authors listed (1997) Reports from the British Columbia Office of Health Technology Assessment (BCOHTA). Routine ultrasound imaging in pregnancy: how evidence- based are the guidelines? Int J Technol Assess Health Care 13:633-637 No authors listed (1997) Routine ultrasound imaging in pregnancy: how evidence-based are the guidelines? Int J Technol Assess Health Care 13:475-477 No authors listed (2004) Proceedings of the Second ALARA Conference. February 28, 2004. Houston, Texas, USA. Pediatr Radiol 34[Suppl 3]:S162-246 Oei EH, Nikken JJ, Ginai AZ et al; From the Program for the Assessment of Radiological Technology (ART Program) (2008) Costs and effectiveness of a brief MRI examination of patients with acute knee injury. Eur Radiol 2008 Sep 16. [Epub ahead of print] Ouwendijk R, de Vries M, Stijnen T et al; From the Program for the Assessment of Radiological Technology (2008) Multicenter randomized controlled trial of the costs and effects of noninvasive diagnostic imaging in patients with peripheral arterial disease: the DIPAD trial. AJR Am J Roentgenol 190:1349-1357 Plevritis SK (2005) Decision analysis and simulation modeling for evaluating diagnostic tests on the basis of patient outcomes. AJR Am J Roentgenol 185:581-590 Prasad KN, Cole WC, Haase GM (2004) Radiation protection in humans: extending the concept of as low as reasonably achievable (ALARA) from dose to biological dam- age. Br J Radiol 77:97-99 Raymond J, Trop I (2007) The practice of ethics in the era of evidence-based radiology. Radiology 244:643-649 RCR Working Party (1998) Making the best use of a department of clinical radiology: guidelines for doctors. 4th edn. The Royal College of Radiologists, London Sackett DL, Rosenberg WM, Gray JA et al (1996) Evidence based medicine: what it is and what it isn’t. BMJ 312:71-72 Semelka RC, Armao DM, Elias J Jr, Huda W (2007) Imaging strategies to reduce the risk of radiation in CT studies, including selective substitution with MRI. J Magn Reson Imaging 25:900-909 Soares HP, Kumar A, Daniels S et al (2005) Evaluation of new treatments in radiation oncology: are they better than standard treatments? JAMA 293:970-978 Sunshine JH, Applegate KE (2004) Technology assessment for radiologists. Radiology 230:309-314 Taïeb S, Vennin P (2001) Evidence-based medicine: towards evidence-based radiology. J Radiol 82:887-890 Thornbury JR (1994) Clinical efficacy of diagnostic imaging: love it or leave it. AJR Am J Roentgenol 162:1-8 Thornbury JR (1999) Intermediate outcomes: diagnostic and therapeutic impact. Acad Radiol 6[suppl 1]:S58-S65 Tonelli MR (1998) The philosophical limits of evidence-based medicine. Acad Med 73:1234-1240 Trinder L (2000) A critical appraisal of evidence-based practice. In: Trinder L, Reynolds S (eds) Evidence-based practice: a critical appraisal. Blackwell Science, Oxford, England:212-214 van Beek EJ, Malone DE (2007) Evidence-based practice in radiology education: why and how should we teach it? Radiology 243:633-640 White SJ, Ashby D, Brown PJ (2000) An introduction to statistical methods for health technology assessment. Health Technol Assess 4:i-iv, 1-59 Woolf SH, Grol R, Hutchinson A et al (1999) Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ 318:527-530

1 Diagnostic Performance

Don’t stop with an answer [...] An answer is always the stretch of road that’s behind you. Only a question can point the way forward.

JOSTEIN GAARDER

The performance of a diagnostic examination1 can be basically considered as its Diagnostic performance degree of accuracy, namely its ability to find the subjects affected with a given disease as positive and the subjects not affected with same disease as negative. The Measures of diagnostic indices which in different ways measure this performance are defined measures of performance diagnostic performance and the studies aimed at measuring the diagnostic performance of an examination or, more often, at comparing the diagnostic performance of two or more examinations, are defined studies of diagnostic performance. Firstly we will present the five most commonly used indices of diagnostic performance in radiologic papers: sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and overall accuracy. Thereafter, we will consider receiver operator characteristic (ROC) curves, which are also widely used in radiologic papers, and likelihood ratios, which are special indices which quantify the ability of a diagnostic examination to change the disease probability (i.e. the power of the diagnostic examination) and which to date have been little used in radiologic papers. In this setting we will show some features of probability theory and Bayes’ theorem. For the sake of clarity, the likelihood ratio will be explained before ROC analysis.

1 For the sake of clarity, we will avoid naming a radiologic examination as a test as much as possible. Although this term is entirely correct, we prefer to use the term exam or examination in order to avoid confusion with statistical tests. Exceptions will be pretest probability and post-test probability for a given disease which we will approach with Bayes’ theorem to represent the ability of each diagnostic test to increase or decrease the probability of a given disease in the subjects who underwent the examination with a positive or a negative result, respectively. Other rare exceptions will be evident from the context.

F. Sardanelli, G. Di Leo, Biostatistics for Radiologists, © Springer-Verlag Italia 2009. 20 Biostatistics for Radiologists

1.1. The Results of an Examination Compared to a Reference Standard

If we want to evaluate the performance of a diagnostic examination, we need Reference standard to compare its results to a reference standard, a term which today is prefer- able to gold standard, since the latter is considered exceedingly optimistic (as in other fields, in Biostatistics all that glistens is not golden). In oncologic diagnostics, the typical example is to verify each result of a diagnostic examination for a sample of n patients with the pathology report, both of which refer to a defined lesion. Suppose that both the radiologist and the pathologist are required to give a dichotomous judgment (yes/no) about the malignancy of a lesion. In this case, the pathology examination is the reference standard and states whether each result of the diagnostic examination is True positive, true negative, true or false. It will be true positive when the radiologist has correctly false positive, false negative defined a pathologically proven malignant lesion as positive, true negative when the radiologist has correctly defined a non-malignant finding as negative, false positive when the radiologist has incorrectly defined a non-malignant finding as positive, and false negative when the radiologist has incorrectly defined a malignant lesion as negative. The n cases which make up the sample of this comparison are distributed among these four possibilities according to the rule that each case is assigned to only one of the four cate- Two-by-two contingency table gories. Using these data, we can generate a two-by-two contingency table where the number of true positives, false positives, false negatives, and true negatives are reported (Table 1.1).

Note that this table can be completed with the total of the lines and with the total of the columns, namely with a series of marginal totals (all the positive cases at the diagnostic examination; all the negative cases at the diagnostic examination; all the positive cases at the reference standard; and all the negative cases at the reference standard), and with the grand total of the n patients or subjects under investigation, as shown in Table 1.2.

Cases, lesions, findings, The careful reader has probably realized that we have intermingled different patients, subjects terms: cases, lesions, findings, patients, and subjects. Pay attention to the meaning of these words. In a scientific paper, these terms cannot be inter- changed and one of them (cases) should be carefully avoided. We can proper- ly consider the study subjects as patients when they present with symptoms or signs for a disease. On the other hand, we name the asymptomatic persons enrolled in population screening program only as subjects. However, it is cor-

Table 1.1. Two-by-two contingency table for the comparison between the results of a radiologic examination and those of a reference standard Reference standard Positive Negative

Positive True positives (TP) False positives (FP) Radiologic examination Negative False negatives (FN) True negatives (TN) Chapter 1 Diagnostic Performance 21

Table 1.2. Two-by-two contingency table for the comparison between the results of a radiologic examination and those of a reference standard in a series of subjects, completed with marginal totals and grand totals Reference standard Affected Nonaffected Total

Positive True positives False positives All positives Radiologic (TP) (FP) (TP + FP) examination Negative False negatives True negatives All negatives (FN) (TN) (FN + TN) Total All affected All nonaffected Grand total (TP + FN) (FP + TN) (TP + FP + FN + TN)

rect to name a group of patients as subjects. Words mean things: the frequency of disease is certainly greater in patients than in symptomatic subjects, with relevant practical consequences which we will see. However, the distinction between patients and subjects is relatively trivial. More importantly, we should fully understand what changes when the statis- The statistical unit tical unit is no longer the patient (or the subject) but each lesion (or finding). to be measured Of course, if each patient has no lesions or only one lesion, we have no consequences in statistical calculations. But a patient can have more than one lesion, as typically we find in the study of liver metastases. The same reasoning can be applied to each of the two kidneys, breasts, lungs, or to a single lobe or segment of the brain, liver, lung, prostate, coronary tree, etc. We should always be extremely clear regarding the application of the indices of diagnostic performance. On what basis are they calculated? Patient by patient? Organ by organ? Segment by segment? Lesion by lesion? Note that the term case is ambiguous Avoid the term case in a because it can be used for both patients and lesions. It should therefore be scientific context strictly avoided in a scientific context. Refer your description to the real statistical units under investigation. We can at this point note the value of a general principle. The initial studies on Initial studies versus large the diagnostic performance of a new imaging modality or technique benefit great- clinical studies ly from the reporting of indices on a lesion-by-lesion basis: we can have a small number of patients and obtain a measure of what happens for each of the lesions. Afterwards, more conclusive information on the value of the clinical application of a new modality or technique can be obtained with a patient-by-patient analysis. In the latter situation, we sometimes have to solve relevant conceptual problems (implying the clinical relevance of radiologic findings) for the definition of true positive, false positive, true negative, and false negative patient when multiple lesions are present and lobes, segments or organs are affected by the disease.

1.2. Measures of Diagnostic Performance

Using the figures of the true positives, false positives, true negatives, and false negatives, we can calculate a series of indices which measure the diagnostic performance. Table 1.3 reports definitions and formulas of these indices, as well as their dependence or independence on disease prevalence. 22 Biostatistics for Radiologists

Table 1.3. Indices measuring diagnostic performance Index Definition Formula Dependence on disease prevalence

1. Sensitivity Ability to identify the TP/(TP+FN) No (or TP rate) presence of disease

2. Specificity Ability to identify the TN/(TN+FP) No (or TN rate) absence of disease

3. Positive predictive Reliability of the TP/(TP+FP) Yes value (PPV) positive result

4. Negative predictive Reliability of the TN/(TN+FN) Yes value (NPV) negative result

5. Overall accuracy Global reliability (TP+TN)/(TP+TN+FP+FN) Yes

6. FN rate Proportion between FN/(FN+TP) = (1 – Sensitivity) No FN and all affected

7. FP rate Proportion between FP/(FP+TN) = (1 – Specificity) No FP and all nonaffected

8. Positive Increase in disease Sensitivity/(1 – Specificity) No likelihood ratio probability when the result is positive

9. Negative Decrease in disease probability (1 – Sensitivity)/Specificity No likelihood ratio when the result is negative

Note that disease prevalence is equal to (TP+FN) / (TP+TN+FP+FN), being the ratio between the number of subjects affected by the disease and the grand total of sample of subjects under investigation.

All of these are simple proportions or ratios which differently combine the four quantities of the two-by-two contingency table. The first seven indices range between 0 and 1 and frequently are reported as percentages. The first five indices indicate an increasingly high diagnostic performance of the examination under investigation the closer they are to 1. The sixth and seventh indices indicate an increasingly high diagnostic performance of the examination under investigation the closer they are to 0. Moreover, they are frequently defined as 1 – sensitivity and 1 – specificity, respectively. The meaning of the last two indices, i.e. the likelihood ratios (LRs), is a bit more complex. They theoretically range between 0 and infinity but practically indicate an increasingly high diagnostic performance the further they move away from 1, with the positive LR moving towards values higher than 1 and the negative LR towards values lower than 1.

1.3. Sensitivity, Specificity, FN Rate and FP Rate

Sensitivity: the ability to The meaning of sensitivity is intuitive: it is the ability of an examination to identify the presence of a disease identify the presence of a given disease. It can also be considered as the proportion between the number of positive subjects with the disease and the total Chapter 1 Diagnostic Performance 23 number of subjects with the disease, namely the proportion of subjects with the disease who were correctly detected by the radiologist. Sensitivity is given by the ratio TP/(TP + FN), i.e. the proportion of positives among the subjects with the disease. If the number of true positives is unchanged, sensitivity is inversely related to the number of false negatives. In fact, the false negative rate, namely the False negative rate proportion of subjects falsely considered nonaffected by the disease, summed with the sensitivity gives a result equal to 1. In other words, the false negative rate is the complement to 1 of sensitivity.

Example 1.1. Sensitivity of mammography and dynamic contrast enhanced magnetic resonance (MR) imaging for the detection of malignant lesions in patients candidate for mastectomy. The authors investi- gate 99 breasts in 90 candidates for unilateral (n = 81) or bilateral (n = 9) mastectomy. The reference standard, i.e. the pathology examination of the whole excised breast, establishes the presence of 188 malignant lesions. Mammography has 124 true positives and 64 false negatives, MR imaging 152 true positives and 36 false negatives. As a consequence, sensitivity is 124/(124+64) = 0.660 for mammography and 152/(152+36) = 0.809 for MR imaging. The lesion-by-lesion sensitivity of mammography is 66.0%, that of MR imaging is 80.9%. The FN rate is 0.340 or 34.0% and 0.191 or 19.1%, respectively. Note that the statistical unit is the lesion and not the patient or the breast [SARDANELLI ET AL, 2004].

The meaning of specificity is evident less immediately. It refers to the abili- Specificity: the ability ty of the examination to identify the absence of a given disease, given by the to identify the absence ratio TN/(TN + FP), i.e. the proportion of the negatives among the subjects not of a disease affected with the disease. If the number of true negatives is unchanged, it is inversely related to the number of false positives. In fact, the false positive rate, False positive rate i.e. the proportion of subjects falsely considered to be affected by the disease, summed with specificity gives 1. In other words, the false positive rate is the complement to 1 of specificity. The less immediate understanding of the term specificity is due to its com- “Specificity” in common mon improper use, at least in spoken language, to indicate the ability of an language examination to make a certain diagnosis. This improper use often implies several logical mistakes. For instance, if we state that computed tomography (CT) is highly “specific” for the diagnosis of intracranial hemorrhage, we would mean that this imaging modality can reliably identify a hyperattenu- ation on nonenhanced scans as a hemorrhage. However, this statement has two different meanings: if really there is an intracranial hemorrhage, it is highly probable that CT can detect it; a CT diagnosis of intracranial hemorrhage is rarely a false positive. Using correct scientific terminology, these two sentences are the same as saying that CT has both high sensitivity and high positive predictive value for intracranial hemorrhage. As both specificity and positive predictive value are inversely related to false positives, if we have very few false positives, it will be true that the examination will also be highly specific (under the condition of having a suitable number of true negatives). At any rate, we cannot say that an examination is highly specific thinking that our audience also understands it to 24 Biostatistics for Radiologists

be highly sensitive. This is a conceptual error. If both sensitivity and specificity are high, the examination is highly accurate (not only highly specific), as CT actually is for intracranial hemorrhage. Care must be taken, since an examination could have few false positives and many false negatives, thus at the same time being highly specific but not very sensitive. As a consequence, it will be of little use as a diagnostic tool in symptomatic patients, despite being highly specific. Moreover, if we say CT is highly specific for the differentiation between acute intracranial hemorrhage and acute brain ischemia, we fall into a deeper complication. This sentence should imply a high sensitivity for both conditions, probably relatively higher for the former than for the latter due to the false negatives associated with tiny ischemias. Similarly, the specificity of CT will be different due to the relatively large number of hypoattenuations caused by previous infarcts in elderly patients or due to artifacts, etc, when compared with the hyperattenuations which can be falsely attributed to hemorrhage. The key point is that high CT specificity for acute intracranial hemorrhage does not imply high specificity for acute brain ischemia. The same reasoning can be applied to sensitivity. For the sake of clarity, we should always distinguish between CT sensitivity and CT specificity for each of the two conditions.

Example 1.2. Specificity. Low-dose CT screening for lung cancer. Of a total of 1611 asymptomatic subjects who undergo the first screening event, 186 are found to be positive and are further studied with high-resolution scanning; 21 of these undergo biopsy. Thirteen subjects are found to be affected by lung cancer. There are no interval cancers (cancers detected between the first and the second screening event). As a result there are 1425 true negatives (the total of 1611 minus 186 positives) and 173 false positives (186 positives minus 13 true positives). Specificity is 1425/(1425+173) = 1425/1598 = 0.892 = 89.2% [SOBUE ET AL, 2002]. In this series only one possible lesion is considered for each subject. Lesion and subject are coincident as a statistical unit.

Sensitivity and specificity: Sensitivity and specificity answer questions which can be raised before answers to pretest questions requesting or performing an examination. They therefore provide answers to aprioristic2 questions:

− If the patient is affected by the disease, what is the probability that the examination produces a positive result (sensitivity)? − If the patient is not affected by the disease, what is the probability that the examination produces a negative result (specificity)?

2 The differentiation between sensitivity and specificity as answers to pre-examination questions on the one hand and predictive values as answers to post-examination questions on the other hand underlines the different logic of these indices of diagnostic performance. Sensitivity and specificity should be used to refer to the intrinsic diagnostic performance of a given examination while predictive values enable us to evaluate the reliability of the results of the same examination once it is permormed. It should be borne in mind that these are not the same thing, as we will explain by demonstrating the influence of disease prevalence on predictive values. Notice that another pre- /post-examination differentiation is related to the concepts of pretest probability and post-test probability which we will introduce for the application of Bayes’ theorem (see Section 1.5). Chapter 1 Diagnostic Performance 25

Sensitivity and specificity (as well as the false negative rate and the false pos- Sensitivity and specificity itive rate) depend on the technical characteristics of the examination, on the do not depend on capability of the radiologist and her/his team (radiographers, nurses, etc.) to per- disease prevalence form the examination, and on the radiologist’s skill in interpreting the examination. Sensitivity and specificity are not influenced by the disease prevalence in the study population (they are instead influenced by the degree, the stage of the disease, as we will demonstrate in the next section). The term prevalence indi- Prevalence cates the proportion between the number of subjects affected by a disease and the total number of subjects of an entire population (or of a sample, frequently named study population) for a defined time interval, whereas the term incidence Incidence indicates the number of subjects newly diagnosed as affected by the disease during a defined time interval (see the Note to Table 1.3). As a matter of fact, the optimal situation in clinical practice is when a single diagnostic examination is available with levels of sensitivity or specificity high enough to produce conclusive decision-making. These two extreme conditions are defined as follows: an examination is SNOUT when its negative result excludes the SNOUT and SPIN possibility of the presence of the disease (when a test has a very high Sensitivity, a Negative result rules OUT the diagnosis); it is instead SPIN when its positive result definitely confirms the presence of the disease (when a test has a very high SPecificity, a positive result rules IN the diagnosis). In most situations, a certain degree of certainty can be reached with a single diagnostic examination but not a definitive conclusion. More than one examination is generally needed. They are ordered according to a flow-chart which takes into account sex, age, familial and personal history, clinical history, previous examinations, etc. In other words, sensitivity and specificity alone cannot translate the result of a radiologic examination into clinical practice.

1.4. Predictive Values, Diagnostic Accuracy and Disease Prevalence

A first possibility for the translation of the result of an examination in clinical Predictive values: practice is provided by predictive values. These indicate the reliability of the answers to post-test questions positive or negative result and answer questions posed after having performed the examination:

− If the result of the examination is positive, what is the probability that the patient really is affected by the disease (positive predictive value)? − If the result of the examination is negative, what is the probability that the patient is really not affected by the disease (negative predictive value)?

The predictive values depend not only on technical parameters and on the abil- Predictive values depend ity to perform the examination and interpret the results. In fact, if sensitivity and on disease prevalence specificity are kept unchanged, predictive values change in relation with disease prevalence: the positive predictive value is directly related to disease prevalence whereas the negative predictive value is inversely related to disease prevalence. Predictive values depend on disease prevalence. This is not intuitive and implies important practical consequences. Let us reflect upon this statement for a moment: when the disease prevalence is very low, a very high sensitivity is associated with a very low positive predictive value. 26 Biostatistics for Radiologists

A useful way of envisaging this situation is provided by the following example. If all the sample subjects have the disease, the positive predictive value is always 1.0 (i.e. 100%) even with very low sensitivity (but not 0) and the negative predictive value is always 0.0 (i.e. 0%) even with very high specificity (even equal to 1.0, i.e. 100%). Similarly, if all the sample subjects do not have the disease, the negative predictive value is always 1.0 (i.e. 100%) even with very low sensitivity (but not 0) and the positive predictive value is always 0.0 (i.e. 0%) even with very high specificity (even equal to 1.0, i.e. 100%). It therefore follows that an examination with the highest possible sensitivity cannot correctly diagnose a non-exis- tent disease, and an examination with the highest possible specificity cannot correctly diagnose the absence of disease in subjects who have the disease. The reliability of our We can obtain increasingly higher levels of sensitivity and specificity, but the reports also depends reliability of our reports (i.e. our predictive values) will depend on disease on patient selection by prevalence, namely on the epidemiologic context and, in clinical practice, on the referring physicians patient selection by the referring physician with a diagnostic query. Now, we should at this point introduce a new variable to the system. A disease can affect a patient with different levels of severity (or stage) and the probability of a positive result of an examination increases with the level of severity. The level of severity should be lower in subjects in whom the disease is diagnosed with peri- odic screening than that found in symptomatic subjects in whom the disease is diagnosed in clinical practice. In this way we observe a direct influence on sensitivity and specificity: they are higher in symptomatic subjects than in asymptomatic subjects in whom the disease is more likely in an early stage. This difference is lower at the first round of an oncologic screening program (when we detect the prevalent tumors, with numerous cases which could have been diagnosed even in an earlier stage) and is higher in the later rounds (when we detect the incident tumors, not present at the first round). Basically, subject selection, which deter- mines the level of severity of the disease, also influences sensitivity and specificity. We will return to this feature after introducing the concept of diagnostic threshold. Overall accuracy: Overall accuracy is the ability of an examination to correctly diagnose both the ability to correctly subjects affected with the disease and subjects not affected with the disease as a identify the presence and the fraction of the total number of examined subjects. It answers the question: what is absence of a disease the probability of a correct result? It is somewhat like a global index of diagnostic performance, but its linear distribution ranges between the sensitivity value and the specificity value. It approaches the higher of the two with increasing disease prevalence and approaches the lower of the two with decreasing disease prevalence. In practice, it is a kind of “mean” between sensitivity and specificity which is weighted for disease prevalence. Dependence on disease prevalence is the feature shared with the predictive values. The graphs in Figure 1.1. show the dependence of predictive values and overall accuracy on disease prevalence.

Example 1.3. Predictive values of clinical and screening mammography. Imagine 10,000 women with a palpable lump are studied (clinical mammography), with 95% sensitivity and 80% specificity. With a disease prevalence of 50%, we would have 4,750 true positives, 4,000 true negatives, 1,000 false positives, and 250 false negatives. The PPV would be 4,750/(4,750+1,000) = 0.826 = 82.6%; the NPV 4,000/(4,000+250) = 0.941 = 94.1%. For nearly every 5 women affected with cancer there would be a healthy woman who undergoes diagnostic work-up with possible needle Chapter 1 Diagnostic Performance 27

biopsy (4,750/1000 = 4.75). This woman with a benign palpable lump is unlikely to consider invasive examinations as useless or dangerous. However, if we were to study 10,000 asymptomatic women (screening mammography) with the same levels of sensitivity and specificity (95% and 80%, respectively) with a disease prevalence of 3%, we would have 285 true positives, 7,760 true negatives, 1,940 false positives, and 15 false negatives. The NPV would go up to 7,760/(7,760+15) = 0.998 = 99.8%, PPV would go down to 285/(285+1,940) = 0.128 = 12.8%. This means that nearly 7 healthy women would be sent for diagnostic work-up with a possible needle biopsy for every woman effectively diagnosed with cancer (1,940/285 = 6.8). The recall rate would be very high, equivalent to 22.25% (2,225/10,000). The overall effect would be a false alarm (if at every round we recall 20-25% of the women, after 4-5 rounds on average all the women would be recalled).