Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary How Good Is My Diagnostic Test? Mike Kokko Thayer School of Engineering Dartmouth College Hanover, NH Engineering in Medicine Seminar April 28, 2017 1/55 Sensitivity Specificity Accuracy Positive/Negative Predictive Value Positive/Negative Likelihood Ratio Diagnostic Odds Ratio AUC (ROC Curve) Which metrics are most appropriate? How do clinicians actually use this information? Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Is it a good test? 2/55 Specificity Accuracy Positive/Negative Predictive Value Positive/Negative Likelihood Ratio Diagnostic Odds Ratio AUC (ROC Curve) Which metrics are most appropriate? How do clinicians actually use this information? Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Is it a good test? Sensitivity 2/55 Accuracy Positive/Negative Predictive Value Positive/Negative Likelihood Ratio Diagnostic Odds Ratio AUC (ROC Curve) Which metrics are most appropriate? How do clinicians actually use this information? Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Is it a good test? Sensitivity Specificity 2/55 Positive/Negative Predictive Value Positive/Negative Likelihood Ratio Diagnostic Odds Ratio AUC (ROC Curve) Which metrics are most appropriate? How do clinicians actually use this information? Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Is it a good test? Sensitivity Specificity Accuracy 2/55 Positive/Negative Likelihood Ratio Diagnostic Odds Ratio AUC (ROC Curve) Which metrics are most appropriate? How do clinicians actually use this information? Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Is it a good test? Sensitivity Specificity Accuracy Positive/Negative Predictive Value 2/55 Diagnostic Odds Ratio AUC (ROC Curve) Which metrics are most appropriate? How do clinicians actually use this information? Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Is it a good test? Sensitivity Specificity Accuracy Positive/Negative Predictive Value Positive/Negative Likelihood Ratio 2/55 AUC (ROC Curve) Which metrics are most appropriate? How do clinicians actually use this information? Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Is it a good test? Sensitivity Specificity Accuracy Positive/Negative Predictive Value Positive/Negative Likelihood Ratio Diagnostic Odds Ratio 2/55 Which metrics are most appropriate? How do clinicians actually use this information? Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Is it a good test? Sensitivity Specificity Accuracy Positive/Negative Predictive Value Positive/Negative Likelihood Ratio Diagnostic Odds Ratio AUC (ROC Curve) 2/55 How do clinicians actually use this information? Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Is it a good test? Sensitivity Specificity Accuracy Positive/Negative Predictive Value Positive/Negative Likelihood Ratio Diagnostic Odds Ratio AUC (ROC Curve) Which metrics are most appropriate? 2/55 Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Is it a good test? Sensitivity Specificity Accuracy Positive/Negative Predictive Value Positive/Negative Likelihood Ratio Diagnostic Odds Ratio AUC (ROC Curve) Which metrics are most appropriate? How do clinicians actually use this information? 2/55 2 Probabilistic Foundation Visualizing study results Definition of metrics Implications for test development 3 Clinical Use Cases 4 Emerging Technologies Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Outline 1 Introduction What is a diagnostic test? Motivational example: Am I pregnant? 3/55 3 Clinical Use Cases 4 Emerging Technologies Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Outline 1 Introduction What is a diagnostic test? Motivational example: Am I pregnant? 2 Probabilistic Foundation Visualizing study results Definition of metrics Implications for test development 3/55 4 Emerging Technologies Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Outline 1 Introduction What is a diagnostic test? Motivational example: Am I pregnant? 2 Probabilistic Foundation Visualizing study results Definition of metrics Implications for test development 3 Clinical Use Cases 3/55 Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Outline 1 Introduction What is a diagnostic test? Motivational example: Am I pregnant? 2 Probabilistic Foundation Visualizing study results Definition of metrics Implications for test development 3 Clinical Use Cases 4 Emerging Technologies 3/55 Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Outline 1 Introduction What is a diagnostic test? Motivational example: Am I pregnant? 2 Probabilistic Foundation Visualizing study results Definition of metrics Implications for test development 3 Clinical Use Cases 4 Emerging Technologies 4/55 Measured quantity known to be strongly correlated with a (typically unobservable) condition of interest Often a continuous measure (e.g. concentration in µg/dL) that produces a binary/dichotomous result when subjected to a set threshold Examples: Prostate-Specific Antigen Test (serum level, prostate cancer) Mammogram (imaging, breast cancer) Microbial Culture (microorganism growth, infection) Electrocardiogram (electrical activity, cardiac conditions) Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Diagnostic Tests What is a Diagnostic Test? Gill, et. al. 2005 5/55 Often a continuous measure (e.g. concentration in µg/dL) that produces a binary/dichotomous result when subjected to a set threshold Examples: Prostate-Specific Antigen Test (serum level, prostate cancer) Mammogram (imaging, breast cancer) Microbial Culture (microorganism growth, infection) Electrocardiogram (electrical activity, cardiac conditions) Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Diagnostic Tests What is a Diagnostic Test? Measured quantity known to be strongly correlated with a (typically unobservable) condition of interest Gill, et. al. 2005 5/55 Examples: Prostate-Specific Antigen Test (serum level, prostate cancer) Mammogram (imaging, breast cancer) Microbial Culture (microorganism growth, infection) Electrocardiogram (electrical activity, cardiac conditions) Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Diagnostic Tests What is a Diagnostic Test? Measured quantity known to be strongly correlated with a (typically unobservable) condition of interest Often a continuous measure (e.g. concentration in µg/dL) that produces a binary/dichotomous result when subjected to a set threshold Gill, et. al. 2005 5/55 Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Diagnostic Tests What is a Diagnostic Test? Measured quantity known to be strongly correlated with a (typically unobservable) condition of interest Often a continuous measure (e.g. concentration in µg/dL) that produces a binary/dichotomous result when subjected to a set threshold Examples: Prostate-Specific Antigen Test (serum level, prostate cancer) Mammogram (imaging, breast cancer) Microbial Culture (microorganism growth, infection) Electrocardiogram (electrical activity, cardiac conditions) Gill, et. al. 2005 5/55 Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Diagnostic Tests 6/55 Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Diagnostic Tests 6/55 Reference standard: Item ! f0; 1g 1 Threshold: R ! f0; 1g Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Diagnostic Tests 1 Index test: Item ! R 7/55 1 Threshold: R ! f0; 1g Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Diagnostic Tests 1 Index test: Item ! R Reference standard: Item ! f0; 1g 7/55 Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Diagnostic Tests 1 Index test: Item ! R Reference standard: Item ! f0; 1g 1 Threshold: R ! f0; 1g 7/55 Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Diagnostic Tests Assumptions Condition and index test both truly dichotomous Existence of perfect reference standard for true diagnosis Independent application of reference standard and index test 8/55 \It should tell me if I'm pregnant" P(POSjPREG) ≈ 1 (Sensitivity) P(∼POS|∼PREG) ≈ 1 (Specificity) Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Am I Pregnant? What makes a good pregnancy test? http://www.firstresponse.com 9/55 P(POSjPREG) ≈ 1 (Sensitivity) P(∼POS|∼PREG) ≈ 1 (Specificity) Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Am I Pregnant? What makes a good pregnancy test? \It should tell me if I'm pregnant" http://www.firstresponse.com 9/55 (Sensitivity) P(∼POS|∼PREG) ≈ 1 (Specificity) Introduction Probabilistic Foundation Clinical Use Cases Emerging Technologies Summary Am I Pregnant? What makes a good pregnancy test? \It should tell me if I'm pregnant" P(POSjPREG) ≈ 1 http://www.firstresponse.com 9/55 P(∼POS|∼PREG) ≈ 1 (Specificity)