Development of Statistical Methods for the Surveillance and Monitoring of Adverse Events Which Adjust for Differing Patient and Surgical Risks
Total Page:16
File Type:pdf, Size:1020Kb
Development of Statistical Methods for the Surveillance and Monitoring of Adverse Events which adjust for Differing Patient and Surgical Risks Ronald Albert Webster Bachelor of Applied Science (Mathematics), QUT Bachelor of Applied Science (Honours), QUT A thesis submitted in partial fulfilment for the degree of Doctor of Philosophy February 2008 Principal Supervisor: Professor Anthony N Pettitt Associate Supervisor: Dr Helen L Johnson Industry Supervisor: Dr Anthony P Morton Queensland University of Technology School of Mathematical Sciences Faculty of Science Brisbane, Queensland, 4000, AUSTRALIA c Copyright by R. A. Webster, 2008 All Rights Reserved Keywords Adverse outcomes, ARL, control chart, Markov chain property, medical monitor- ing, minimum effect CUSUM, meta-factors, patient level uncertainty, parameter estimation uncertainty, risk adjustment, risk adjusted CUSUM, report cards, risk score, risk model performance. i Abstract The research in this thesis has been undertaken to develop statistical tools for mon- itoring adverse events in hospitals that adjust for varying patient risk. The studies involved a detailed literature review of risk adjustment scores for patient mortal- ity following cardiac surgery, comparison of institutional performance, the perfor- mance of risk adjusted CUSUM schemes for varying risk profiles of the populations being monitored, the effects of uncertainty in the estimates of expected probabili- ties of mortality on performance of risk adjusted CUSUM schemes, and the insta- bility of the estimated average run lengths of risk adjusted CUSUM schemes found using the Markov chain approach. The literature review of cardiac surgical risk found that the number of risk factors in a risk model and its discriminating ability were independent, the risk factors could be classified into their “dimensions of risk”, and a risk score could not be generalized to populations remote from its developmental database if accurate predictions of patients’ probabilities of mortality were required. The conclusions were that an institution could use an “off the shelf” risk score, provided it was recalibrated, or it could construct a customized risk score with risk factors that provide at least one measure for each dimension of risk. The use of report cards to publish adverse outcomes as a tool for quality im- provement has been criticized in the medical literature. An analysis of the report cards for cardiac surgery in New York State showed that the institutions’ outcome rates appeared overdispersed compared to the model used to construct confidence intervals, and the uncertainty associated with the estimation of institutions’ out- iii come rates could be mitigated with trend analysis. A second analysis of the mor- tality of patients admitted to coronary care units demonstrated the use of notched box plots, fixed and random effect models, and risk adjusted CUSUM schemes as tools to identify outlying hospitals. An important finding from the literature review was that the primary reason for publication of outcomes is to ensure that health care institutions are accountable for the services they provide. A detailed review of the risk adjusted CUSUM scheme was undertaken and the use of average run lengths (ARLs) to assess the scheme, as the risk profile of the population being monitored changes, was justified. The ARLs for in-control and out-of-control processes were found to increase markedly as the average outcome rate of the patient population decreased towards zero. A modification of the risk adjusted CUSUM scheme, where the step size for in-control to out-of-control out- come probabilities were constrained to no less than 0.05, was proposed. The ARLs of this “minimum effect” CUSUM scheme were found to be stable. The previous assessment of the risk adjusted CUSUM scheme assumed that the predicted probability of a patient’s mortality is known. A study of its performance, where the estimates of the expected probability of patient mortality were uncertain, showed that uncertainty at the patient level did not affect the performance of the CUSUM schemes, provided that the risk score was well calibrated. Uncertainty in the calibration of the risk model appeared to cause considerable variation in the ARL performance measures. The ARLs of the risk adjusted CUSUM schemes were approximated using sim- ulation because the approximation method using the Markov chain property of CUSUMs, as proposed by Steiner et al. (2000), gave unstable results. The cause of the instability was the method of computing the Markov chain transition prob- abilities, where probability is concentrated at the midpoint of its Markov state. If probability was assumed to be uniformly distributed over each Markov state, the ARLs were stabilized, provided that the scores for the patients’ risk of adverse outcomes were discrete and finite. iv Contents 1 Overview of the Research and Investigations 1 2 Literature Review 7 2.1 StandardControlCharts . 7 2.1.1 ForNumericalData. .. .. 8 2.1.2 ForCountData......................... 9 2.1.3 ForProportionData . 11 2.2 MonitoringAdverseEvents. 12 2.2.1 RareEvents........................... 12 2.2.2 RiskAdjustedCharts. 14 2.2.3 ComparingInstitutions. 17 2.2.4 PerformanceEvaluation . 19 2.3 RiskScoresforTraumaVictims . 20 2.3.1 TriageScores .......................... 24 2.3.2 QualityImprovement . 29 2.4 Conclusion................................ 34 3 Meta-Factors for Cardiac Surgical Risk Models 37 3.1 Introduction............................... 38 3.2 ThreeRiskModels ........................... 43 3.2.1 SimpleClassification . 43 3.2.2 ParsonnetScore. .. .. 44 v 3.2.3 EuroSCORE........................... 47 3.3 RiskFactorClasses ........................... 51 3.3.1 Meta-Factors .......................... 53 3.3.2 RiskScorePerformance . 54 3.4 PerformanceatMultipleLocations . 60 3.4.1 ParsonnetScore. .. .. 61 3.4.2 EuroSCORE........................... 63 3.5 Cardiac Surgical Scores for an Institution . .... 65 3.5.1 Recalibrated“OfftheShelf”Scores . 67 3.5.2 CustomizedScores . 67 3.6 Conclusion................................ 70 4 Multi-Institutional Monitoring Schemes for Adverse Events 73 4.1 Introduction............................... 73 4.2 DebateaboutReportCards . 76 4.3 Report Cards for Cardiac Surgery in New York State . .. 79 4.3.1 MonitoringScheme . 80 4.3.2 Issues .............................. 82 4.3.3 Overdispersion ......................... 87 4.3.4 An Alternative Monitoring Scheme . 90 4.4 Multi-HospitalDatainYorkshire . 92 4.4.1 RiskModels........................... 93 4.4.2 ExploratoryAnalysis . 94 4.4.3 FixedEffectsModel . .. .. 98 4.4.4 RandomEffectsModel . 100 4.4.5 R-ACUSUMSchemes . 101 4.5 Discussion ................................104 5 Performance of Risk Adjusted CUSUM Schemes: Expected Out- come Probability Assumed Known 107 5.1 ReviewofCUSUMSchemes . 109 5.1.1 IndustrialSchemes . 109 vi 5.1.2 SchemesforMortalityOutcomes . 112 5.2 Review of Assessment Methods . 116 5.3 Assessing Performance Using the ARL . 121 5.3.1 PopulationDistributions . 122 5.3.2 Definition of the Risk Adjusted CUSUM . 126 5.3.3 Performance of the Risk Adjusted CUSUM . 127 5.4 Properties of Risk Adjusted CUSUMs . 134 5.4.1 Mean&VarianceofWeights. 135 5.4.2 EffectSize............................139 5.5 Risk Adjusted & Minimum Effect CUSUMs . 142 5.6 Industrial & Risk Adjusted CUSUMs . 153 5.7 Conclusion&Recommendation . 156 6 Performance of Risk Adjusted CUSUM Schemes: Expected Out- come Probability Uncertain 159 6.1 Introduction...............................159 6.2 UncertaintyatPatientLevel . 161 6.2.1 Beta Distribution—p T Unbiased. 164 6.2.2 Gamma Distribution—Odds Unbiased . 165 6.2.3 Gamma Distribution—p T Unbiased . 166 6.2.4 Normal Distribution—Logit Unbiased . 167 6.2.5 Normal Distribution—p T Unbiased . 167 6.3 ARLs with Patient Level Uncertainty . 168 6.4 Effect of Patient Level Uncertainty . 193 6.5 Uncertainty in Model Parameter Estimates . 196 6.5.1 Estimated Probability of Outcome . 197 6.5.2 ARL0sofRiskAdjustedCUSUMs. 201 6.5.3 Effect on Risk Adjusted CUSUM Alarms . 204 6.6 Conclusion................................210 6.7 Addendum................................212 vii 7 Stability of ARL Approximations Using Markov Chains 215 7.1 Introduction...............................215 7.2 Methods.................................218 7.2.1 Conditional Distribution of Ct .................219 7.2.2 TransitionProbabilities. 220 7.2.3 SimulationMethod . 222 7.3 Results..................................222 7.4 Discussion ................................226 8 Summary and Conclusion 231 8.1 ResultsofInvestigations . 231 8.1.1 CardiacSurgicalScores. 231 8.1.2 Multi-Institutional Monitoring . 233 8.1.3 R-A CUSUM Performance: Outcome Rates Known . 234 8.1.4 R-A CUSUM Performance: Outcome Rates Uncertain . 234 8.1.5 ARLsUsingtheMarkovApproach . 236 8.2 PossibleFurtherResearch . 236 8.2.1 CardiacSurgicalRiskModels . 236 8.2.2 Other than Patient Characteristics’ Measures of Risk . 237 8.2.3 MonitoringSchemes . 240 8.3 ClosingComments ........................... 241 A Risk Factors and Associated Meta-Factors 243 B Risk Scores for Mortality Following Cardiac Surgery 251 C Evaluations of the Parsonnet Score or the EuroSCORE 259 Bibliography 265 viii List of Figures 3.1 Each of the AROC values given for the risk models in Table 3.4 plotted against the number of risk factors used to construct the risk score................................... 59 4.1 Trends of the RAMRs for CABG surgery and associated 95% con- fidence intervals for two hospitals in the New York