Development and Validation of the Pediatric Risk Estimate Score for Children Using Extracorporeal Respiratory Support (Ped-RESCUERS) Ryan P

Home , Permissive hypercapnia

Development and validation of the pediatric risk estimate score for children using extracorporeal respiratory support (Ped-RESCUERS) Ryan P. Barbaro, University of Michigan Philip S. Boonstra, University of Michigan Matthew Paden, Emory University Lloyd A. Roberts, Monash University Melbourne Gail M. Annich, University of Toronto Robert H. Bartlett, University of Michigan Frank W. Moler, University of Michigan Matthew M. Davis, University of Michigan

Journal Title: Intensive Care Medicine Volume: Volume 42, Number 5 Publisher: Springer (part of Springer Nature): Springer Open Choice Hybrid Journals - CC-BY-NC | 2016-05-01, Pages 879-888 Type of Work: Article | Post-print: After Peer Review Publisher DOI: 10.1007/s00134-016-4285-8 Permanent URL: https://pid.emory.edu/ark:/25593/tnns8

Final published version: http://dx.doi.org/10.1007/s00134-016-4285-8 Copyright information: © 2016, Springer-Verlag Berlin Heidelberg and ESICM. Accessed September 25, 2021 1:40 AM EDT HHS Public Access Author manuscript

Author ManuscriptAuthor Manuscript Author Intensive Manuscript Author Care Med. Author Manuscript Author manuscript; available in PMC 2019 February 18. Published in final edited form as: Intensive Care Med. 2016 May ; 42(5): 879–888. doi:10.1007/s00134-016-4285-8.

Development and Validation of the Pediatric Risk Estimate Score for Children Using Extracorporeal Respiratory Support (Ped- RESCUERS)

Ryan P. Barbaro, MD, MSc1,2, Philip S. Boonstra, PhD3, Matthew L. Paden, MD4, Lloyd A. Roberts, MBBS5, Gail M. Annich, MD, MS6, Robert H. Bartlett, MD7, Frank W. Moler, MD, MS2, and Matthew M. Davis, MD, MAPP1,2,8,9 1Department of Pediatrics, University of Michigan, Ann Arbor; 2Child Health Evaluation and Research (CHEAR) Unit, University of Michigan, Ann Arbor; 3School of Public Health Department of Biostatistics, University of Michigan, Ann Arbor 4Division of Pediatric Critical Care, Emory University, Atlanta, Georgia; 5Intensive Care Department, Alfred Hospital and School of Public Health and Preventative Medicine, Monash University Melbourne, Australia; 6Critical Care Medicine, University of Toronto, Toronto, Canada; 7Department of Surgery, University of Michigan, Ann Arbor; 8Department of Internal Medicine, University of Michigan, Ann Arbor, 9Gerald R. Ford School of Public Policy and Department of Health Management and Policy, School of Public Health, University of Michigan, Ann Arbor

Abstract Purpose: To develop and validate the Pediatric Risk Estimation Score for Children Using Extracorporeal Respiratory Support (Ped-RESCUERS). Ped-RESCUERS is designed to estimate the in-hospital mortality risk for children prior to receiving respiratory extracorporeal membrane oxygenation (ECMO) support.

Methods: This study used data from an international registry of patients aged 29 days to less than 18 years who received ECMO support from 2009 to 2014. We divided the registry into development and validation datasets by calendar date. Candidate variables were selected for model inclusion if the variable independently changed the mortality risk by at least 2 % in a Bayesian logistic regression model with in-hospital mortality as the outcome. We characterized the model’s ability to discriminate mortality with the area under curve (AUC) of the receiver operating characteristic.

Address correspondence to: Ryan Barbaro, University of Michigan, 1500 East Medical Center Drive, Mott F-6790/Box 5243, Ann Arbor, MI 48109, [email protected], (734) 764-5302. Reprints: Reprints will not be ordered. Conflict of Interest: Drs. Bartlett, Paden, and Annich acknowledge that they are on the Extracorporeal Life Support Organization steering committee. The other authors have no conflicts of interest relevant to this article to disclose. Barbaro et al. Page 2

Results: From 2009 to 2014, 2458 non-neonatal children received ECMO for respiratory Author ManuscriptAuthor Manuscript Author Manuscript Author Manuscript Author support, with a mortality rate of 39.8 %. The development dataset contained 1611 children receiving ECMO support from 2009 to 2012. The model included the following variables: pre- ECMO pH, pre-ECMO arterial partial pressure of carbon dioxide, hours of intubation prior to ECMO support, hours of admission at ECMO center prior to ECMO support, ventilator type, mean airway pressure, pre-ECMO use of milrinone, and a diagnosis of pertussis, asthma, bronchiolitis, or malignancy. The validation dataset included 438 children receiving ECMO support from 2013 to 2014. The Ped-RESCUERS model from the development dataset had an AUC of 0.690, and the validation dataset had an AUC of 0.634.

Conclusions: Ped-RESCUERS provides a novel measure of pre-ECMO mortality risk. Future studies should seek external validation and improved discrimination of this mortality prediction tool.

Keywords extracorporeal membrane oxygenation; risk assessment; risk adjustment; severity of illness index; mortality; pediatric

Introduction The case-mix adjusted mortality rate is considered essential for accurate evaluation of hospital-level outcomes [1–7]. Numerous clinical registries have incorporated risk-adjusted mortality measurements to enhance internal and external benchmarking and as drivers for quality improvement among participating institutions [8–12]. Researchers have applied risk- adjustment tools to facilitate observational research [2,1,13], and physicians can utilize risk- adjustment tools to anticipate the mortality risk for patients [14,15]. In pediatric extracorporeal membrane oxygenation (ECMO) for respiratory support such a risk- adjustment tool would be of great utility in clinical and analytic applications. owever, no such risk-adjustment tool exists [16].

In this study we use data from the Extracorporeal Life Support Organization (ELSO) registry, an international registry of 298 centers, to develop and internally validate the Pediatric Risk Estimate Score for Children Using Extracorporeal Respiratory Support (Ped- RESCUERS) tool. Ped-RESCUERS is designed to estimate the pre-ECMO risk of in- hospital death for children receiving respiratory ECMO support.

Materials and Methods This study was designed in accordance with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement [17]. As a retrospective analysis of deidentified data, formal consent was not required, and it was determined to be exempt from human subjects review by the Institutional Review Board of the University of Michigan Medical School.

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 3

Patient Selection Author ManuscriptAuthor Manuscript Author Manuscript Author Manuscript Author We queried the ELSO registry for all pediatric patients aged 29 days to <18 years who received ECMO support for respiratory failure from 2009–2014. A priori, we chose 2009 as the start date for data accrual because advances in ECMO technology from that year onward have been reported to enhance delivery of ECMO support [18,19]. If a patient was placed on ECMO more than once, we only considered the first ECMO run. The development dataset incorporated data among children who received pediatric respiratory ECMO between January 1, 2009, and December 31, 2012. A validation dataset was created from children who received respiratory ECMO support between January 1, 2013 and December 31, 2014. We limited our validation dataset to those with complete data for selected variables (Figure 1).

Multiple Imputations We derived Ped-RESCUERS from a Bayesian logistic regression model predicting mortality. Our development dataset included variables with missing data (Online Resource 1-Table e1). We addressed missing data through multiple imputation with iterative chained equations [20–22] because logistic regression models that limit analysis to patients with complete data can lead to biased results [17,23]. Multiple imputation uses the partial information available in the observation and data contained in other observations in the dataset to predict an observation’s missing data (additional details in Online Resource 1-Supplemental Methods) [20]. Importantly, the outcome variable of death prior to hospital discharge contained no missing observations.

Candidate Variables We adapted primary diagnostic fields from a previous publication of pediatric respiratory ECMO mortality risk factors [24] (Table 1) and defined diagnostic groups using International Classification of Disease-9-Clinical Modification (ICD-9-CM) diagnostic codes. Primary diagnoses were not considered as one variable with 15 categories, but rather as 15 present or absent dummy variables. This allowed the contribution of each diagnosis to the model to be considered individually.

We recoded all other categorical variables into dummy variables. This yielded 53 candidate variables, listed in Table 1 and Table 2. In addition to primary diagnoses, candidate variables included clinical data collected ≤ 6 hours prior to ECMO, such as physiologic (the worst pre-ECMO blood gas and lowest systolic blood pressure) and therapeutic (ventilator settings and number of days of mechanical ventilation prior to ECMO) data. Other variables included the presence of pre-ECMO cardiac arrest, pre-ECMO renal failure and any clinical comorbidities as defined by Feudtner et al. [25] (additional details in Online Resource 1- Supplemental Methods and Table e2). A priori we identified two types of interactions to consider: the interaction between ventilator settings and ventilator type and an interaction between blood pressure and age.

Model Fitting It can be difficult to compare the effect sizes (often presented as odds ratios or beta- coefficients) across dichotomous variables and continuous variables. We address this

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 4

challenge by transforming all variables to a common standard through location-scale Author ManuscriptAuthor Manuscript Author Manuscript Author Manuscript Author transforming before model fitting [26]. First, we center all variables at the observed mean and then divide by twice the standard deviation of the observed data. Next, we fit a multiple logistic regression model including the candidate variables. Each individual variable’s beta- coefficient (βk) is interpreted as the log odds ratio for mortality for each two-standard deviation change in measurement. The standardization allows for direct comparison of odds ratios between variables but does not necessarily translate into clinically relevant quantities. For a dichotomous variable with 50% prevalence (and therefore twice the standard deviation equal to 1), such as gender, βk is the difference in log odds between female and male. For dichotomous variables with prevalence not equal to 50%, βk is no longer the difference in log odds between the two values, since two standard deviations is less than 1 (additional details in Online Resource 1-Supplemental Methods).

Development of Ped-RESCUERS We fit a single Bayesian multivariate logistic regression with outcome of mortality and 53 candidate variables. A variable was selected if there was a 75% probability that a two- standard deviation variable increase changed a child’s likelihood of mortality by ≥2%, from a baseline of 40%. Statistically, this translates to having a βk >0.08 (mortality risk factor) or −1 −1 βk <−0.08 (protective factor) since logit [0.08]=0.42 and logit [−0.08]=0.38 (additional details in Online Resource 1-Supplemental Methods).

Discrimination, Calibration and Validation of Ped-RESCUERS For development and internal validation we assessed the model discrimination using the area under curve (AUC) of the receiver operating characteristic curves, and we characterized the calibration using the Brier score [27,28]. Validation was against those with complete data.

We conducted a sensitivity analysis in which we re-created the Ped-RESCUERS score after marking data for 64 patients with potentially incorrect data as missing and then imputing the data. Potentially incorrect data included patients with implausible ventilator settings such as a positive end expiratory pressure on a high frequency oscillatory ventilator (HFOV) or extremely abnormal blood gas values such as an arterial partial pressure of oxygen (PaO2) < 10 mm Hg (additional details in Online Resource 1-Supplemental Methods).

Results During 2009–2014, 2,458 children aged 29 days to <18 years received ECMO for respiratory support, with an overall mortality of 39.8%. The duration of ECMO was a median of 188 hours (7.8 days) with an interquartile range of 104–356 hours. Prior to ECMO, 49% of children were receiving support via a HFOV and 47% were receiving inhaled nitric oxide. The median oxygenation index (mean airway pressure × fraction of inspired oxygen × 100/PaO2) was 44, with an interquartile range of [30–60]. Two-thirds of subjects were neuromuscularly blocked and 61% received a vasoactive infusion.

The median duration of time between admission to an ECMO center and the subsequent start of ECMO support was 17 hours less in those who survived compared to those who died (Table 1). There was a similar distribution of primary diagnoses and comorbidities in the

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 5

development (2009–2012) and validation (2013–2014) time periods (Table 1, Online Author ManuscriptAuthor Manuscript Author Manuscript Author Manuscript Author Resource 1-Table e3). Blood gas values and ventilator measurements were not substantively different between the two time periods (Table 2, Online Resource 1-Table e4). The mortality rate from 2009–2012 was 40.8%, while the mortality rate was 38.0% from 2013–2014.

Ped-RESCUERS Development The development dataset included 1,611 children who received ECMO from 2009–2012. Eleven variables satisfied selection criteria and were included in the model fit to estimate the Ped-RESCUERS score (Table 3). To calculate Ped-RESCUERS for an individual child, use the β-coefficients in combination with the patient’s variable values as described in Table 4 or visit www.ped-rescuers.com.

The largest estimated association with mortality was pH, having an odds ratio of 0.46 (95% Credible Interval (CI)=0.29–0.70). Three primary diagnoses were independently associated with mortality compared to our reference category of pneumonia. Asthma and bronchiolitis were associated with lower mortality, whereas pertussis was associated with higher mortality (Table 3). The presence of a malignancy was the only comorbidity meeting our selection criteria. Malignancy was associated with a 10% increase in a child’s mortality risk relative to a child without malignancy (Table 3).

Among pre-ECMO support, a higher mean airway pressure, longer pre-ECMO intubation time, and pre-ECMO milirinone use were associated with an increased mortality risk. Compared to a child intubated for 2 days pre-ECMO, a person intubated for 10 days would have an associated 5% increase in their mortality risk.

Unexpectedly, the model demonstrated that a higher arterial partial pressure of arterial carbon dioxide (PaCO2) within 6 hours prior to ECMO was associated with a decreased mortality risk. In univariate analysis (Wilcoxon rank-sum test), there was no statistically significant difference between the PaCO2 of survivors and children who died (p-value=0.41). The association between higher PaCO2 and lower mortality was only apparent after multivariable adjustment in the logistic regression model. In an attempt to understand what might be underlying this association, we compared the mortality rate and median PaCO2 of children with obstructive lung disease (asthma and bronchiolitis) to all other diagnoses. As expected, mortality rate was lower for children with asthma and bronchiolitis (22%) compared to all other diagnoses (44%). Those with asthma and bronchiolitis also had a higher median PaCO2 72 [53–95] mm Hg versus 63 [48–84] mm Hg.

Discrimination, Calibration, and Validation of Ped-RESCUERS Ped-RESCUERS had an AUC=0.690 in the developmental dataset (Figure 2). From 2013– 2014 there were 847 children who received respiratory ECMO support. Over the 11 selected variables in Ped-RESCUERS, 438 children had complete data (Figure 1). These 438 children made up the internal validation dataset. In the validation dataset, the AUC decreased to 0.634 (95% CI 0.595, 0.649) (Figure 2).

Figure e1 (Online Resource 1) illustrates the model’s calibration in the validation dataset by comparing the difference in expected and observed mortality. In a well-calibrated model, the

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 6

plotted line should not deviate far from the line y=0. Overall calibration, as measured by the Author ManuscriptAuthor Manuscript Author Manuscript Author Manuscript Author Brier score, is 0.229 (95% CI 0.227, 0.238). Thus, there remains variability in mortality prediction unexplained by the Ped-RESCUERS model.

Sensitivity Analysis We investigated the sensitivity of our findings to the 64 patients with unusual recorded measurements of ventilator settings and the 5 patients with a very low PaO2 (< 10 mm Hg) by assuming these specific measurements were unobserved and re-running our entire imputation and analysis strategy. Confirming our findings, the identical set of predictor variables was selected in the sensitivity analysis, with very similar estimated associations. Because 17 patients in our validation dataset had unusual measurements, the size of the validation dataset decreased by 17 patients, from 438 to 421. The AUC, 0.635 (95% CI 0.594, 0.651), and Brier score, 0.229 (95% CI 0.227, 0.239), were equivalent to our primary findings.

Discussion To our knowledge, Ped-RESCUERS is the first risk-adjustment tool created for children receiving respiratory ECMO support. This tool performs similarly in development and validation datasets. However, the discrimination of Ped-RESCUERS as currently formulated provides opportunities for improvement.

The tool is composed of three types of data including physiologic (i.e. blood gases), administrative (i.e. diagnostic), and therapeutic support (i.e. mechanical ventilator). Selected variables have face validity that resonates with routine clinical experience in ECMO support. Patients with cancer and pertussis have relatively poor outcomes, while those with asthma and bronchiolitis do well [24]. Additionally, children requiring ECMO later in their course and children with low pH and high ventilator settings also typically do more poorly [15,24]. Previous studies have demonstrated an association between ECMO mortality and milrinone [29]. Milrinone, is used for patients with cardiac dysfunction and its association with ECMO mortality may suggest that children with cardiac dysfunction and respiratory failure are less likely to survive.

Age, renal function, and measures of hypoxemia are often incorporated into severity of illness scores [30–33,15], but were not selected in Ped-RESCUERS. The exclusion of acute renal dysfunction and measures of hypoxemia is consistent with the findings in the adult Respiratory ECMO Survival Prediction (RESP) Score [15]. In Ped-RESCUERS, the ratio of the arterial partial pressure of oxygen to the fraction of inspired oxygen (PF ratio) was excluded because there is little difference between the PF ratio of those who survived versus died (Table 2). Since ECMO is able to oxygenate the blood regardless of the severity of lung disease, it may be that the most prognostic variables are those that predict lung recovery. We were unable to evaluate the predictive capacity of the ratio of the peripheral saturation of oxygen (SpO2) to FiO2 (the SF ratio) because ELSO does not record SpO2 [34].

Significantly, Ped-RESCUERS found that a higher PaCO2 was associated with a lower mortality risk, while the RESP score found that a PaCO2 >75 mm Hg was associated with a

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 7

higher risk of mortality. We suspect two factors are underlying this difference. First, only Author ManuscriptAuthor Manuscript Author Manuscript Author Manuscript Author after adjusting for pH (and the other variables) was there a statistically significant association between a higher PaCO2 and survival. The Ped-RESCUERS model suggests that among people with a similar pH, those with a higher PaCO2 tend to survive more frequently. The RESP score does not include pH, which makes the PaCO2 effects less comparable; this phenomenon is known as Simpson’s paradox [35]. Second, in the RESP score, only 1% of the cohort had an obstructive respiratory failure (asthma). In Ped-RESCUERS, 15% of children had asthma or bronchiolitis and these children survived twice as often and had a 15% higher PaCO2. Consequently, the difference also may be because some of the survival benefit of asthma and bronchiolitis is being attributed to the associated high PaCO2.

Implications We believe Ped-RESCUERS has three potential applications. First, it can be used to provide risk-adjusted internal and external benchmarking of ECMO performance quality to centers that participate in ELSO [36]. Benchmarking is needed to characterize the substantial, clinically meaningful variation in hospital-level mortality rates [37]. Second, risk-adjustment tools such as Ped-RESCUERS can enhance conduct of observational research [1]. For example, if a researcher wanted to perform an observational study comparing outcomes for children who received ECMO care under conditions of an awake state versus deep sedation, then Ped-RESCUERS measure could be used to match patients or more efficiently adjust for differences in the intervention and control arms. Third, Ped-RESCUERS could help physicians anticipate the risk of mortality for similar patient groups prior to ECMO.

Study Limitations Ped-RESCUERS cannot predict if an individual child will benefit from ECMO support, because this tool is derived from a sample of patients who all received ECMO. Ped- RESCUERS is designed to estimate the pre-ECMO mortality risk among patients who clinically are deemed to require ECMO. Additionally, Ped-RESCUERS was developed and internally validated using ELSO data. Consequently, it may not generalize to patients cared for at non-ELSO centers. Future studies should seek to test its discrimination and calibration at non-ELSO centers.

Ped-RESCUERS also has less discriminatory power than risk-adjustment tools for other clinical populations [30,38,31], but we believe there is an opportunity to improve the model discrimination with additional clinical data. This model does not include some physiologic measures included in other severity of illness measures [30,38,31] such as pupillary response [30,38,31], and other laboratory measures such as such as creatinine [30,31], lactatemia [39,30,31], white blood cell count [30,31], platelets [30,31], and prothrombin level [30]. Models with this information are able to better predict mortality with an AUC >0.85.

The discriminatory power of Ped-RESCUERS must be considered when applying the tool. For the first time, Ped-RESCUERS will enable inter-institutional risk-adjusted mortality rate comparisons, but the adjusted outcomes must be considered with the understanding that this model leaves significant variance in mortality unexplained. The unexplained variance in mortality will also impact Ped-RESCUERS applicability in research. It means the tool

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 8

cannot completely adjust for differences between two groups in an observational trial. Author ManuscriptAuthor Manuscript Author Manuscript Author Manuscript Author Nonetheless, Ped-RESCUERS provides a more efficient adjustment using 1 variable instead of 11, and this efficiency allows for a powered study with fewer patients.

A challenge of the database is that not all cases have complete data. Although most candidate variables are missing <10% of observations, some are missing more frequently (Online Resource 1; Table e1). One variable, mean airway pressure, is missing in 29% of cases. Importantly, the model performs similarly in the validation sample when only observations without missing data are tested, and we have attempted to minimize the effect of missing with multiple imputation. If we decided to exclude mean airway pressure based on its degree of missing, then we would have lost the predictive value of this variable, and excluded a clinically important factor.

Over time, pre-ECMO care has evolved. The median number of days between intubation and ECMO cannulation has decreased from 3.5 days [24] to 2 days. Additionally, in the 1990s, the median PaCO2 was 50 mm Hg by 2005 it was 60 mm Hg [24] and now it is 65 mm Hg. This shift in the median pre-ECMO PaCO2 demonstrates increasing permissive hypercapnia, which may be motivated by the 2000 publication demonstrating improved survival with low tidal volume ventilation in acute respiratory distress syndrome [40]. Because of these changes in pre-ECMO care and advances in ECMO care and technology, Ped-RESCUERS and any ECMO risk-adjustment tool will need re-calibration with time [33].

Conclusions Ped-RESCUERS is a promising first step in creating a pre-ECMO mortality risk estimation tool. Ped-RESCUERS does not explain all the difference in mortality risk, but it is an important incremental step that advances the risk-adjustment for mortality for purposes of benchmarking and research.

Supplementary Material

Refer to Web version on PubMed Central for supplementary material.

Acknowledgements:

The authors would like to thank the Extracorporeal Life Support organization for the opportunity to conduct this research. They also thank Folafoluwa O. Odetola, MD, MPH for his valuable assistance in editing and revising this manuscript.

Source of Funding: Dr. Barbaro received a research award from the Extracorporeal Life Support Organization to support this study. Dr. Barbaro was supported by a T32 (HD007534) grant funded by the Eunice Kennedy Shriver National Institute for Child Health and Human Development, for which Dr. Davis was the principal investigator.

References 1. Pollack MM, Ruttimann UE, Getson PR, (1987) Accurate prediction of the outcome of pediatric intensive care. A new quantitative method. N Engl J Med 316: 134–139. [PubMed: 3796684] 2. Pollack MM, Holubkov R, Funai T, Berger JT, Clark AE, Meert K, Berg RA, Carcillo J, Wessel DL, Moler F, Dalton H, Newth CJ, Shanley T, Harrison RE, Doctor A, Jenkins TL, Tamburro R, Dean JM, (2015) Simultaneous Prediction of New Morbidity, Mortality, and Survival Without New

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 9

Morbidity From Pediatric Intensive Care: A New Paradigm for Outcomes Assessment. Crit Care Author ManuscriptAuthor Manuscript Author Med Manuscript Author 43: 1699–1709. Manuscript Author [PubMed: 25985385] 3. Fonarow GC, Pan W, Saver JL, Smith EE, Reeves MJ, Broderick JP, Kleindorfer DO, Sacco RL, Olson DM, Hernandez AF, Peterson ED, Schwamm LH, (2012) Comparison of 30-day mortality models for profiling hospital performance in acute ischemic stroke with vs without adjustment for stroke severity. JAMA 308: 257–264. [PubMed: 22797643] 4. Krumholz HM, Wang Y, Mattera JA, Wang Y, Han LF, Ingber MJ, Roman S, Normand SL, (2006) An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with heart failure. Circulation 113: 1693–1701. [PubMed: 16549636] 5. Krumholz HM, Brindis RG, Brush JE, Cohen DJ, Epstein AJ, Furie K, Howard G, Peterson ED, Rathore SS, Smith SC, Jr., Spertus JA, Wang Y, Normand SL, (2006) Standards for statistical models used for public reporting of health outcomes: an American Heart Association Scientific Statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group: cosponsored by the Council on Epidemiology and Prevention and the Stroke Council. Endorsed by the American College of Cardiology Foundation. Circulation 113: 456–462. [PubMed: 16365198] 6. Lindenauer PK, Grosso LM, Wang C, Wang Y, Krishnan JA, Lee TA, Au DH, Mularski RA, Bernheim SM, Drye EE, (2013) Development, validation, and results of a risk-standardized measure of hospital 30-day mortality for patients with exacerbation of chronic obstructive pulmonary disease. J Hosp Med 8: 428–435. [PubMed: 23913593] 7. Horbar JD, Carpenter JH, Badger GJ, Kenny MJ, Soll RF, Morrow KA, Buzas JS, (2012) Mortality and neonatal morbidity among infants 501 to 1500 grams from 2000 to 2009. Pediatrics 129: 1019– 1026. [PubMed: 22614775] 8. Panzer RJ, Gitomer RS, Greene WH, Webster PR, Landry KR, Riccobono CA, (2013) Increasing demands for quality measurement. JAMA 310: 1971–1980. [PubMed: 24219953] 9. Jacobs JP, O’Brien SM, Pasquali SK, Gaynor JW, Mayer JE, Jr., Karamlou T, Welke KF, Filardo G, Han JM, Kim S, Quintessenza JA, Pizarro C, Tchervenkov CI, Lacour-Gayet F, Mavroudis C, Backer CL, Austin EH, 3rd, Fraser CD, Tweddell JS, Jonas RA, Edwards FH, Grover FL, Prager RL, Shahian DM, Jacobs ML, (2015) The Society of Thoracic Surgeons Congenital Heart Surgery Database Mortality Risk Model: Part 2-Clinical Application. Annals Thorac Surg 100 (3): 1063– 1070. 10. Thabut G, Christie JD, Kremers WK, Fournier M, Halpern SD, (2010) Survival differences following lung transplantation among US transplant centers. JAMA 304: 53–60. [PubMed: 20606149] 11. Cohen ME, Ko CY, Bilimoria KY, Zhou L, Huffman K, Wang X, Liu Y, Kraemer K, Meng X, Merkow R, Chow W, Matel B, Richards K, Hart AJ, Dimick JB, Hall BL, (2013) Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus. J Am Coll Surg 217: 336–346.e331. [PubMed: 23628227] 12. Richardson DK, Gray JE, McCormick MC, Workman K, Goldmann DA, (1993) Score for Neonatal Acute Physiology: a physiologic severity index for neonatal intensive care. Pediatrics 91: 617–623. [PubMed: 8441569] 13. Kahn JM, Goss CH, Heagerty PJ, Kramer AA, O’Brien CR, Rubenfeld GD (2006) Hospital volume and the outcomes of mechanical ventilation. N Engl J Med 355 (1):41–50. [PubMed: 16822995] 14. Tyson JE, Parikh NA, Langer J, Green C, Higgins RD (2008) Intensive care for extreme prematurity--moving beyond gestational age. N Engl J Med 358 (16):1672–1681. [PubMed: 18420500] 15. Schmidt M, Bailey M, Sheldrake J, Hodgson C, Aubron C, Rycus PT, Scheinkestel C, Cooper DJ, Brodie D, Pellegrino V, Combes A, Pilcher D, (2014) Predicting survival after extracorporeal membrane oxygenation for severe acute respiratory failure. The Respiratory Extracorporeal Membrane Oxygenation Survival Prediction (RESP) score. Am J Respir Crit Care Med 189: 1374–1382. [PubMed: 24693864] 16. Paden ML, Rycus PT, Thiagarajan RR (2014) Update and outcomes in extracorporeal life support. Semin Perinatol 38 (2):65–70. [PubMed: 24580761]

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 10

17. Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, Vickers AJ, Author ManuscriptAuthor Manuscript Author ManuscriptRansohoff Author DF, Collins Manuscript Author GS, (2015) Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration. Ann Intern Med 162: W1–W73. [PubMed: 25560730] 18. Gadepalli SK, Hirschl RB, (2015) Extracorporeal life support: updates and controversies. Semin Pediatr Surg 24: 8–11. [PubMed: 25639803] 19. MacLaren G, Combes A, Bartlett RH (2012) Contemporary extracorporeal membrane oxygenation for adult respiratory failure: life support in the new era. Intensive Care Med 38 (2):210–220. [PubMed: 22147116] 20. Rubin DB (2004) Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons. 21. Van Buuren S, Brand JPL, Groothuis-Oudshoorn CGM, Rubin DB (2006) Fully conditional specication in multivariate imputation. J Stat Comput Simul 76 (12):1049–1064. 22. White IR, Royston P, Wood AM (2011) Multiple imputation using chained equations: Issues and guidance for practice. Stat Med 30 (4):377–399. [PubMed: 21225900] 23. Janssen KJ, Donders AR, Harrell FE, Jr., Vergouwe Y, Chen Q, Grobbee DE, Moons KG (2010) Missing covariate data in medical research: to impute is better than to ignore. J Clin Epidemiol 63 (7):721–727. [PubMed: 20338724] 24. Zabrocki LA, Brogan TV, Statler KD, Poss WB, Rollins MD, Bratton SL (2011) Extracorporeal membrane oxygenation for pediatric respiratory failure: Survival and predictors of mortality. Crit Care Med 39 (2):364–370. [PubMed: 20959787] 25. Feudtner C, Berry JG, Parry G, Hain P, Morse RB, Slonim AD, Shah SS, Hall M (2011) Statistical uncertainty of mortality rates and rankings for children’s hospitals. Pediatrics 128 (4):e966–972. [PubMed: 21890830] 26. Gelman A (2008) Scaling regression inputs by dividing by two standard deviations. Stat Med 27 (15):2865–2873. [PubMed: 17960576] 27. Brier GW (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Review 78:1–3 28. Labarere J, Renaud B, Fine MJ (2014) How to derive and validate clinical prediction models for use in intensive care medicine. Intensive Care Med 40 (4):513–527. [PubMed: 24570265] 29. Cashen K, Thiagarajan RR, Collins JW, Jr., Rycus PT, Backer CL, Reynolds M, Costello JM (2015) Extracorporeal Membrane Oxygenation in Pediatric Trisomy 21: 30 Years of Experience from the Extracorporeal Life Support Organization Registry. J Pediatr 167 (2):403–408. [PubMed: 25982140] 30. Pollack MM, Holubkov R, Funai T, Dean JM, Berger JT, Wessel DL, Meert K, Berg RA, Newth CJ, Harrison RE, Carcillo J, Dalton H, Shanley T, Jenkins TL, Tamburro R (2016) The Pediatric Risk of Mortality Score: Update 2015. Pediatr Crit Care Med 17 (1):2–9. [PubMed: 26492059] 31. Leteurtre S, Duhamel A, Salleron J, Grandbastien B, Lacroix J, Leclerc F (2013) PELOD-2: an update of the PEdiatric logistic organ dysfunction score. Crit Care Med 41 (7):1761–1773. [PubMed: 23685639] 32. Zimmerman JE, Kramer AA, McNair DS, Malila FM (2006) Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med 34 (5):1297–1310. [PubMed: 16540951] 33. Slater A, Shann F, Pearson G (2003) PIM2: a revised version of the Paediatric Index of Mortality. Intensive care medicine 29 (2):278–285. doi:10.1007/s00134-002-1601-2 [PubMed: 12541154] 34. Khemani RG, Rubin S, Belani S, Leung D, Erickson S, Smith LS, Zimmerman JJ, Newth CJ (2015) Pulse oximetry vs. PaO2 metrics in mechanically ventilated children: Berlin definition of ARDS and mortality risk. Intensive Care Med 41 (1):94–102. [PubMed: 25231293] 35. Blyth CR (1972) On Simpson’s Paradox and the Sure-Thing Principle. J Am Stat Assoc 67 (338): 364–366. 36. Antonelli M, Bonten M, Cecconi M, Chastre J, Citerio G, Conti G, Curtis JR, Hedenstierna G, Joannidis M, Macrae D, Maggiore SM, Mancebo J, Mebazaa A, Preiser JC, Rocco P, Timsit JF, Wernerman J, Zhang H (2013) Year in review in Intensive Care Medicine 2012: III. Noninvasive ventilation, monitoring and patient-ventilator interactions, acute respiratory distress syndrome, sedation, paediatrics and miscellanea. Intensive Care Med 39 (4):543–557. [PubMed: 23338570]

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 11

37. Barbaro RP, Odetola FO, Kidwell KM, Paden ML, Bartlett RH, Davis MM, Annich GM (2015) Author ManuscriptAuthor Manuscript Author ManuscriptAssociation Author of Hospital-Level Manuscript Author Volume of Extracorporeal Membrane Oxygenation Cases and Mortality - Analysis of the Extracorporeal Life Support Organization Registry. Am J Respir Crit Care Med 191 (8):894–901. [PubMed: 25695688] 38. Straney L, Clements A, Parslow RC, Pearson G, Shann F, Alexander J, Slater A (2013) Paediatric index of mortality 3: an updated model for predicting mortality in pediatric intensive care*. Pediatr Crit Care Med 14 (7):673–681. [PubMed: 23863821] 39. Morris KP, McShane P, Stickley J, Parslow RC (2012) The relationship between blood lactate concentration, the Paediatric Index of Mortality 2 (PIM2) and mortality in paediatric intensive care. Intensive Care Med 38 (12):2042–2046. [PubMed: 23100008] 40. Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome. The Acute Respiratory Distress Syndrome Network (2000). N Engl J Med 342 (18):1301–1308. [PubMed: 10793162]

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 12 Author ManuscriptAuthor Manuscript Author Manuscript Author Manuscript Author

Figure 1. Flow Diagram for Development and Internal Validation Datasets ECMO=Extracorporeal Membrane Oxygenation. In the developmental dataset, all children were analyzed because missing data was imputed.

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 13 Author ManuscriptAuthor Manuscript Author Manuscript Author Manuscript Author

Figure 2. Receiver Operating Characteristic Curves Receiver operating characteristics (ROC) curve dashed line depicts the area under the curve (AUC) for the development sample and solid line ROC curve gives the AUC for the validation data.

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 14

Table 1.

Author ManuscriptAuthor Pre-ECMO Manuscript Author characteristics Manuscript Author of patients in Manuscript Author the development (2009–2012) dataset

Variable All (N=1,611) Survived (N=954) Died (N=657) Median (Interquartile Range) a Mean Arterial Pressure, mm Hg 56 (45–68) 57 (46–70) 55 (45–67) Time between admission and ECMO, hours 48 (9–174) 44 (8–143) 61 (10–251) Weight, kilograms 12 (6–35) 12 (6–32) 12 (6–40) Age, months 26 (5–122) 25 (6–114) 28 (5–132) Number (%) Female 756 (46.9) 452 (47.4) 304 (46.2) Primary Diagnosis Asthma 39 (2.4) 33 (3.4) 6 (0.9) Aspiration Pneumonia 26 (1.2) 19 (2.0) 7 (1.1) Bronchiolitis 198 (12.3) 153 (16.0) 45 (6.9) Pertussis 49 (3.0) 15 (1.6) 34 (5.2) Viral or Bacterial Pneumonia 361 (22.4) 224 (23.5) 137 (20.9) Pulmonary Hemorrhage 18 (1.1) 11 (1.2) 7 (1.1) Chronic Respiratory Failure 37 (2.3) 22 (2.3) 15 (2.3) Congenital Airway Anomaly 20 (1.2) 15 (1.6) 5 (0.8) Other Respiratory Disease 273 (16.9) 144 (15.1) 129 (19.6) Drowning, Inhalation, or Foreign Body 31 (1.9) 24 (2.5) 7 (1.1) Non-pulmonary Infection 174 (10.8) 86 (9.0) 88 (13.4) Trauma or Postoperative 52 (3.2) 29 (3.0) 23 (3.5) Pulmonary Hypertension 41 (2.6) 26 (2.7) 15 (2.3) Cardiac Disease 136 (8.4) 73 (7.7) 63 (9.6) Other 156 (9.7) 80 (8.4) 76 (11.6) b Comorbidities Neuromuscular 6 (0.4) 3 (0.3) 3 (0.3) Cardiovascular 174 (10.8) 99 (10.4) 75 (11.4) Respiratory 81 (5.0) 52 (5.5) 29 (4.4) Renal 20 (1.2) 8 (0.8) 12 (1.8) Gastrointestinal 15 (0.9) 9 (0.9) 6 (0.9) Hematology 20 (1.2) 13 (1.4) 7 (1.1) Immunodeficiency 28 (1.7) 12 (1.3) 16 (2.4) Metabolic 16 (1.0) 4 (0.4) 12 (1.8) Other Congenital or Genetic Defects 84 (5.2) 43 (4.5) 41 (6.2) Malignancy 83 (5.2) 36 (3.8) 47 (7.2) Complications of Acute Illness c Acute Renal Failure 32 (2.0) 14 (1.5) 18 (2.7) Cardiac Arrestc 74 (4.6) 43 (4.5) 31 (4.7)

a Most abnormal value recorded within 6 hours of receipt of ECMO support,

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 15

b Comorbidities are defined by Feudtner et al [25] c Pre-ECMO renal failure and pre-ECMO cardiac arrest are defined by ICD-9-CM codes plus the respective absence of renal failure or cardiac arrest as an ECMO complication. Author ManuscriptAuthor Manuscript Author Manuscript Author Manuscript Author

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 16

Table 2:

Author ManuscriptAuthor Blood Manuscript Author gas and supportive Manuscript Author therapies prior Manuscript Author to extracorporeal membrane oxygenation (ECMO) support in the development (2009–2012) dataset

Variable All (N=1,611) Survived (N=954) Died (N=657) Median (IQR) a pH 7.21 (7.09–7.33) 7.23 (7.11–7.34) 7.19 (7.07–7.30) a PaCO2, mm Hg 65 (49–87) 64 (48–87) 65 (51–86) a Bicarbonate, mmol/L 26 (20–32) 26 (21–32) 25 (20–31) a PF Ratio 56 (43–74) 56 (43–76) 55 (43–72) Pre-ECMO duration of mechanical ventilation, hours 48 (15–149) 46 (14–135) 53 (16–172) Conventional Ventilator Settings a Rate, breaths per minute 28 (20–35) 26 (20–35) 28 (22–35) a Peak Inspiratory Pressure, cm H2O 34 (29–40) 34 (28–39) 35 (30–40) a Positive End Expiratory Pressure, cm H2O 10 (7–14) 10 (6–12) 10 (8–14) a Mean Airway Pressure, cm H2O 19 (15–24) 19 (14–24) 20 (16–25) High Frequency Oscillatory Ventilator Settings a Frequency, Hertz 7 (5–8) 7 (5–8) 7 (5–8) a Amplitude, cm H2O 55 (45–66) 55 (45–65) 55 (45–68) a Mean Airway Pressure, cm H2O 29 (24–33) 28 (22–32) 30 (25–35) b Other Ventilator Settings a Mean Airway Pressure, cm H2O 24 (21–28) 23 (22–26) 25 (18–31) Number (%) Ventilator Type Conventional Ventilator 655 (40.7) 398 (41.7) 257 (39.1) High Frequency Oscillatory Ventilator 692 (43.0) 401 (42.0) 291 (44.3) b Other Ventilator 31 (1.9) 16 (1.7) 15 (2.2) Missing 233 (14.5) 139 (14.6) 94 (14.3) Other Pre-ECMO Supportive Therapies Hand Ventilation 162 (10.1) 101 (10.6) 61 (9.3) Inhaled Nitric Oxide 798 (49.5) 460 (48.2) 338 (51.4) Neuromuscular Blockade 773 (48.0) 614 (64.3) 401 (61.0) Vasoactive Infusions 939 (46.7) 434 (45.5) 339 (51.6) Milrinone Infusion 185 (11.5) 93 (9.8) 92 (14.0) Continuous Renal Replacement Therapy 26 (1.6) 16 (1.7) 10 (1.5)

PaCO2= the arterial partial pressure of carbon dioxide, PF ratio=the ratio of arterial partial pressure of oxygen to fraction of inspired oxygen a Most abnormal value recorded within 6 hours prior to the receipt of ECMO support

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 17

b Other Ventilator in the Extracorporeal Life Support Organization Registry is defined as other high frequency ventilator and usually corresponds to the jet ventilator. Author ManuscriptAuthor Manuscript Author Manuscript Author Manuscript Author

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 18

Table 3.

Author ManuscriptAuthor Multivariable Manuscript Author analysis Manuscript Author of factors associated Manuscript Author with death prior to hospital discharge

Change from median Standardized Odds of resulting in 5% increase in Variable Mortality (95% CI) Observed Median mortality Blood Gas a pH 0.46 (0.29–0.70) 7.20 −0.09 a PaCO2, mm Hg 0.68 (0.44–1.01) 65 −34.7 Ventilator Settings Conventional Ventilator a Mean Airway Pressure, cm H2O 1.41 (0.98–2.33) 19.0 +8.7 High Frequency Oscillatory Ventilator a Mean Airway Pressure, cm H2O 1.62 (1.02–2.53) 29.0 +6.6 Duration of Pre-ECMO Care Log-transformed time between admission and ECMO, 3.89 +1.99 1.47 (1.06–1.97) log-hours (48 hours) (359 hours = 15 days) Log-transformed pre-ECMO duration of mechanical 3.89 +1.36 1.57 (1.09–2.19) ventilation, log-hours (48 hours) (189 hours = 8 days) Comorbidity Change in mortality when variable present Malignancy 1.20 (0.99–1.51) 10.3% Pre-ECMO Support Milrinone 1.24 (0.99–1.61) 7.7% Primary Diagnosis Asthma 0.82 (0.59–1.02) −14.8% Bronchiolitis 0.60 (0.45–0.79) −17.2% Pertussis 1.39 (1.06–1.80) 23.4%

With the exception of row three (Conventional Ventilator), columns 3 and 4 are based on a reference patient with 42% risk of mortality on a high frequency oscillatory ventilator with median mean airway pressure and median values of all other continuous variables, no malignancy, and primary diagnosis of pneumonia. The third row corresponds to a similar reference patient (with 41% risk of mortality) on a conventional ventilator with median mean airway pressure and identical values of all other covariates. a Most abnormal value recorded within 6 hours prior to the receipt of ECMO support.

Intensive Care Med. Author manuscript; available in PMC 2019 February 18. Barbaro et al. Page 19

Table 4.

Author ManuscriptAuthor Example Manuscript Author Calculation of Manuscript Author Ped-RESCUERS Manuscript Author for a child with pneumonia

Patient Characteristics How to Calculate Ped-RESCUERS Example Calculation pH = 7.20 − 2.171 × pH −2.171 × 7.20 = −15.631

PaCO2 = 65 mm Hg − 0.006 × PCO2 −0. 006 × 65 = −0.390

+ 0.102 × loge(hours from admission to ECMO center until initiation of admitted 48 hours prior to ECMO ECMO+1) 0.102 × loge(48+1) = 0.397

intubated 48 hours prior to ECMO + 0.150 × loge(hours from intubation to initiation of ECMO+1) 0.150 × loge(48+1) = 0.584 not on CMV − 0.463 (if patient on CMV) −0.463 × 0 = 0 not on CMV + 0.023 × MAP measurement (if patient on CMV) 0.023 × 0 = 0 yes, on HFOV − 0.890 (if patient on HFOV) −0.890 × 1 = –0.890

MAP = 29 cm H2O + 0.031 × MAP measurement (if patient on HFOV) 0.031 × 29 = 0.899 no malignancy + 0.415 (if patient has a comorbidity of malignancy) 0.415 × 0 = 0 no pertussis + 0.959 (if patient’s primary diagnosis is pertussis) 0.959 × 0 = 0 no asthma − 0.665 (if patient’s primary diagnosis is asthma) −0.665 × 0 = 0 no bronchiolitis − 0.788 (if patient’s primary diagnosis is bronchiolitis) −0.788 × 0 = 0 no milrinone + 0.313 (if patient received milrinone prior to ECMO support) 0.313 × 0 = 0 + 14.70, the intercept + 14.70 Total = Ped-RESCUERS Total = –0.331

Probability of Mortality = ePed-RESCUERS/1+ePed-RESCUERS e−0.331 /(1+ e−0.331) = 0.418

PaCO2 = the arterial partial pressure of carbon dioxide, CMV= conventional mechanical ventilation, HFOV = high frequency oscillatory ventilator, MAP= mean airway pressure, and malignancy is defined by Feudtner et al. [25] Ped-RESCUERS does not need to be manually calculated if you visit www.Ped-RESCUERS.com and enter the prompted patient characteristics. The probability of mortality with 95% prediction interval will be calculated. In this case, this child’s pre-ECMO mortality risk is estimated as 42% with a 95% prediction interval of 38% to 45%.

Intensive Care Med. Author manuscript; available in PMC 2019 February 18.