A Time-Updated, Parsimonious Model to Predict AKI in Hospitalized Children

CLINICAL RESEARCH www.jasn.org

Ibrahim Sandokji ,1,2 Yu Yamamoto ,2 Aditya Biswas,2 Tanima Arora ,2 Ugochukwu Ugwuowo,2 Michael Simonov,2 Ishan Saran,2 Melissa Martin ,2 Jeffrey M. Testani,3 Sherry Mansour,2 Dennis G. Moledina ,2 Jason H. Greenberg ,1,2 and F. Perry Wilson 2

1Department of Pediatrics, Section of Nephrology, Yale University School of Medicine, New Haven, Connecticut 2Clinical and Translational Research Accelerator, Department of Medicine, Yale University School of Medicine, New Haven, Connecticut 3Section of Cardiovascular Medicine, Department of Internal Medicine, Yale University School of Medicine, New Haven, Connecticut

ABSTRACT Background Timely prediction of AKI in children can allow for targeted interventions, but the wealth of data in the electronic health record poses unique modeling challenges. Methods We retrospectively reviewed the electronic medical records of all children younger than 18 years old who had at least two creatinine values measured during a hospital admission from January 2014 through January 2018. We divided the study population into derivation, and internal and external validation cohorts, and used five feature selection techniques to select 10 of 720 potentially predictive variables from the electronic health records. Model performance was assessed by the area under the receiver operating characteristic curve in the validation cohorts. The primary outcome was development of AKI (per the Kidney Disease Improving Global Outcomes creatinine definition) within a moving 48-hour window. Secondary outcomes included severe AKI (stage 2 or 3), inpatient mortality, and length of stay. Results Among 8473 encounters studied, AKI occurred in 516 (10.2%), 207 (9%), and 27 (2.5%) encounters in the derivation, and internal and external validation cohorts, respectively. The highest-performing model used a machine learning-based genetic algorithm, with an overall receiver operating characteristic curve in the internal validation cohort of 0.76 [95% confidence interval (CI), 0.72 to 0.79] for AKI, 0.79 (95% CI, 0.74 to 0.83) for severe AKI, and 0.81 (95% CI, 0.77 to 0.86) for neonatal AKI. To translate this prediction model into a clinical risk-stratification tool, we identified high- and low-risk threshold points. Conclusions Using various machine learning algorithms, we identified and validated a time-updated prediction model of ten readily available electronic health record variables to accurately predict imminent AKI in hospitalized children.

JASN 31: 1348–1357, 2020. doi: https://doi.org/10.1681/ASN.2019070745

AKI develops in approximately 30% of hospitalized children in intensive care units (ICUs) and 5% in Received July 26, 2019. Accepted March 13, 2020. 1–5 non-ICU settings. Hospital-acquired AKI is as- Published online ahead of print. Publication date available at sociated with a longer hospital stay, higher mortal- www.jasn.org. ity, and an increased economic burden.1–4,6 AKI Correspondence: Francis Perry Wilson, Program of Applied may also increase the risk of long-term complica- Translational Research, Temple Medical Center, 60 Temple tions such as CKD, proteinuria, and hypertension.3 Street Suite 6C, New Haven, CT 06510. Email: francis.p.wilson@ AKI management strategies are largely supportive and yale.edu include avoiding nephrotoxin exposure, optimizing Copyright © 2020 by the American Society of Nephrology

1348 ISSN : 1046-6673/3106-1348 JASN 31: 1348–1357, 2020 www.jasn.org CLINICAL RESEARCH volume status, and addressing electrolyte imbalances.4 Prior Significance Statement studies have suggested that the early identification of patients at risk of developing AKI and subsequent nephrotoxin avoidance Because AKI in hospitalized children is associated with poor can decrease the AKI rate by 64%.7 outcomes, a tool allowing early identification of children at risk of The AKI prediction workgroup of the Acute Dialysis developing AKI may facilitate timely interventions. The authors describe various machine learning techniques used to build a par- Quality Initiative recommended creating electronic health simonious model predictive of pediatric AKI. From an initial pool of record (EHR)-integrated, real-time AKI prediction models 720 potential variables, they evaluated multiple feature selection that combine risk factors from prototype prediction models techniques to create a ten-feature logistic regression model that with novel risk factors using machine learning methods.8 could predict, in time-updated fashion, the risk of AKI in the next fl Current AKI prediction models have helped build our un- 48 hours. A machine learning-based genetic algorithm (re ecting fi the process of natural selection) was the best variable selection derstanding of the eld of AKI prediction in children, but method, using ten factors extracted from electronic health records they have limitations as they mainly rely on baseline admis- to use for AKI prediction. Risk-stratifying hospitalized children sion data, were developed with a limited set of AKI predictors, might allow clinicians to implement targeted and timely interven- and do not include neonates.9,10 Although neonates have a tions prior to AKI development. lower GFR as compared to older children, they share many 4,11 similar AKI risk factors. Additionally, none of the current level ,4 mg/dl or an eGFR of .15 ml/min per 1.73 m2 were prediction models were created in an unbiased clinically ag- excluded. We calculated eGFR using the modified Schwartz nostic approach leveraging the diversity of variables available equation, or if height was not available, using the Full Age 9,10,12 in the EHR. Spectrum equation.14 The Yale Human Investigation Com- The EHR contains a wealth of variables that may predict mittee reviewed and approved the study protocol under a AKI, but brute-force methods to evaluate parsimonious mod- waiver of informed consent. els are computationally infeasible. For example, there are 21 9.69*10 models containing 10 variables that could be created Dependent (Outcome) Variables from a set of 720 variables (as we had in this study). Even The primary outcome was the development of AKI within the assuming one could evaluate 1,000,000 such models per sec- next 48 hours, updated throughout the hospital stay. We used ond using logistic regression (infeasible with even modern the Kidney Disease Improving Global Outcome creatinine supercomputers), it would take more than 300 million years definition using a rolling window that compared the current to find the best possible combination of 10 variables in a space creatinine to the lowest in the past 48 hours or 7 days. Stage 1 with 7001 potential variables. Therefore, researchers are often AKI was defined as a 0.3 mg/dl increase in creatinine within forced to create parsimonious models based on prior research 48 hours or an increase of 1.5 times the lowest measured cre- or clinical intuition, which is biased against novel predictors. atinine within the prior 7 days.4 Secondary outcomes included Several algorithms, including forward selection, Least Absolute severe AKI (stage 2 or 3) on the same time scale, inpatient Shrinkage and Selection Operator (LASSO) regression, and oth- mortality, and length of stay. Stage 2 and 3 AKI are defined ers, have been proposed to more efficiently find parsimonious as an increase in creatinine level by 2–2.9 times and an models that may allow for novel features to be included.13 This increase $3 times baseline, respectively.4 Urine output was study presents several such methods of narrowing the difficulty not used to define AKI due to missingness and expected diffi- of the problem. We aimed to create a novel AKI prediction culty in application in real time. To qualify for our AKI defini- model that is time-updated, is parsimonious, and could be tion, an absolute value of serum creatinine .0.5 mg/dl was also used in all hospitalized neonates and children to predict AKI required.11,15 This avoids clinically dubious changes in creati- within the next 48 hours. nine (such as from 0.2 mg/dl to 0.3 mg/dl) being considered diagnostic of AKI. To limit the number of imputed values and to make predictions similar to what would be done in prospec- METHODS tive implementation, we excluded timepoints prior to the first creatinine measurement. Study Design and Population We retrospectively reviewed the medical records of all children Independent (Predictor) Variables Selection younger than 18 years old, who had at least two creatinine We included a total of 720 candidate variables extracted from values measured at any time during a hospital admission the EHR including patient demographics, vital signs, patient from January 2014 to January 2018 to two hospitals in the locations, diagnoses, laboratory results, and medication expo- Yale-New Haven Health System, a large, academic, tertiary sures (Figure 1, Supplemental Table 1). We excluded from care center. The two hospitals are Yale-New Haven Children’s analysis highly collinear variables with a correlation coeffi- Hospital and Bridgeport Hospital, the latter being a hospital cient r of 0.95 or greater. For continuous variables with data affiliated with the same Health System, but staffed by other missing for ,25% of patient encounters, measured values physicians and caring for a significantly less acutely ill pediat- were carried forward until remeasurement occurred. For ric population. Patients with an initial serum creatinine timepoints with no previous measurement, we imputed

JASN 31: 1348–1357, 2020 EHR-Based Pediatric AKI Prediction 1349 CLINICAL RESEARCH www.jasn.org

Demographics Vital signs Locations Laboratory variables Medications Procedures

7 variables 6 variables 24 27 222 315 34 85 variables Continuous Categorical Individual Medication variables variables variables medications Groups

Figure 1. Candidate variable groups. A total of 720 candidate variables available in the EHR were included to create a parsimonious predictive model. derivation-set medians. We transformed continuous variables is chosen and applied on regression coefficients. The least with .25% missing data into binary markers indicating if the significantvariableswillbeforcedtohavecoefficients of variable had ever been measured up to the current timestamp. zero and therefore will be eliminated. We chose a penalty For example, chloride level, which was missing in 1.6% of level that allowed for selection of only ten variables to create encounters, was modeled as the measured value of the deri- this model. vation cohort median (102 mEq/L), while lactate (missing in 66% of encounters) was modeled simply as having been mea- Clinical Model sured at the current point in the hospitalization or not yet We manually selected ten variables highly associated with the having been measured (regardless of the lactate value). This development of AKI in children as determined by the research simple imputation strategy was chosen to facilitate adoption team’s clinical expertise and prior research.2,4 Previous re- of the model into the EHR, as multiple imputation techniques search has identified that mechanical ventilation, acidosis, el- are not applicable in real-time, prospective analyses. We trans- evated BUN, and vasoactive support are associated with AKI in formed medication exposure (and procedure) variables into children.1,2,18 Studies have also shown that children admitted binary markers to indicate exposure to the medication at the to an ICU are at an increased risk of AKI as compared with current timestamp and once exposed, this status was contin- those not admitted to an ICU.1,4,11,12,19 Hyper- and/or hypo- ued until the end of encounter. calcemia were previously reported to be associated with AKI We used multiple feature selection strategies to select a development.20 Lastly, recent EHR-based AKI prediction parsimonious set of ten features from the entire feature set. models and AKI severity scores have included acidosis, We selected ten features as each additional feature introduces BUN, changes in weight, fluid overload, chemotherapy medi- greater complexity to EHR-integration (as over time, data cations, ICU admission, and ventilatory and inotropic support points may be changed or require different cleaning strat- in their prediction models.9,10,12,13 egies). We also performed logistic regression analyses of incremental numbers of variables and observed an increas- Genetic Algorithm ing model performance as additional variables were added, Genetic algorithms are a machine learning technique in the but there were diminishing returns after the first ten vari- domain of evolutionary optimization; to prevent confusion, ables (Supplemental Figure 1). Selection strategies were as we stress that this study’s usage of genetic algorithms does not follows: use any biologic genetic data. Our genetic algorithm models parsimonious groups of ten features as “individuals” or geno- Univariable Area under the Receiver Operating Characteristic types in a population. Each genotype corresponds to 10 fea- Curve Ranking tures chosen, initially at random, from the 720-feature pool. Each of the 720 features were ranked according to the Uni- Each genotype is assigned a fitness score; in this case, this variable Area under the Receiver Operating Characteristic corresponds to the AUC with logistic regression using only Curve (AUC) after a univariable logistic regression. The top the feature set given by the genotype. The fittest individuals ten features were then chosen. are preferentially allowed to exchange genetic information (mate) to create offspring, creating a next generation with Forward Selection the same population size. Algorithmically, this translates to We built a logistic regression model that sequentially added each child having ten features randomly selected from the the most statistically significant variables from the remaining combined features present in the parents (genetic recombina- pool, one at a time, until a model of ten variables was tion). Akin to genetic mutation, single features are also substituted obtained.16 at random with low probability from the 720-feature pool. This process terminates when the median fitness score in the pop- LASSO Logistic Regression ulation has not improved for 20 generations. Finally, we esti- This is a form of penalized logistic regression that promotes mate a mean fitness score using fivefold crossvalidation for sparsity over the model coefficients.13,17 In LASSO regression, each genotype in the final population, and return the feature the process of feature selection occurs when a penalty (l) level set that produces the highest score (Supplemental Figures 2 and

1350 JASN JASN 31: 1348–1357, 2020 www.jasn.org CLINICAL RESEARCH

3 and Supplemental Methods).21 Genetic algorithm code can pediatric ICU (PICU) and 13.5% in the neonatal ICU] be found at https://github.com/Yale-PATR/PopFS. (Table 1). The baseline median serum creatinine was 0.51 mg/dl (IQR, 0.32–0.77) and baseline eGFR was 96.7 ml/min Statistical Analysis per 1.73 m2 (IQR, 44–129). AKI occurred in 516 (10.2%), We divided the study population at the Yale-New Haven 207 (9%), and 27 (2.5%) encounters in derivation, and internal Children’s Hospital into derivation (70%) and internal vali- and external validation cohorts, respectively. AKI was associ- dation (30%) cohorts, randomized at the patient level to en- ated with higher inpatient mortality (11% compared with 0.9% sure the same patient would never appear in both data sets. We without AKI, P,0.001) and increased length of stay (average of used the cohort of patients admitted to Bridgeport Hospital as 17.1 days compared with 4.2 days without AKI, P,0.001) an external validation cohort. We summarized the data as me- (Supplemental Table 2). dian [interquartile range (IQR)] for continuous variables and Table 2 lists the results of the feature selection strategies we count (percentage) for categorical variables. Univariable com- employed. Notably, several variables were selected by multiple parisons of categorical data were analyzed with the Pearson’s feature selection methods: BUN, creatinine, glucose, sodium chi-squared or Fisher’s exact tests. Continuous data were an- bicarbonateuse,andthetimesinceadmission(Table3). alyzed using Wilcoxon rank-sum tests. We set the threshold of Supplemental Table 3 shows the prediction model coefficients statistical significance at P,0.05. and odds ratios. The genetic algorithm model outperformed We performed logistic regression analyses on the five other logistic regression-based models with an AUC of 10-feature models. All models were developed entirely in 0.76 [95% confidence interval (95% CI), 0.72 to 0.79]. This the derivation set, and evaluated in the internal and external was significantly better performance than the other feature validation sets. Model AUC was calculated by applying the selection algorithms (P,0.05 for comparing the genetic algo- c-statistic transformation to the rank-based Somers D sta- rithm model with clinical, LASSO, and forward models in the tistic, accounting for within-patient clustering as multiple internal validation cohort). The genetic algorithm model was predictions are made over time in each patient.22 We compared well calibrated in both the derivation and internal validation the AUC across models by computing the linear combination cohorts (Supplemental Figure 4). The area under the of the difference in AUC from the above process. We also eval- precision-recall curve of the genetic algorithm model was uated the area under the precision-recall curve and calibration 0.11 in the internal validation cohort (Supplemental Figure 5). of the prediction models. We performed subgroup analyses of Figure 3 demonstrates an example of the predictive prob- age groups, patient locations, medical diagnoses, and stages ability of the genetic algorithm model along with the rise in of AKI. serum creatinine and development of AKI in an individual A sensitivity analysis was performed on high- and low-risk patient. cutoff threshold levels of the genetic algorithm model predic- In evaluating secondary outcomes, the genetic algorithm tive probability of 0.24 and 0.08, respectively. These analyses model AUC increased to 0.79 (95% CI, 0.74 to 0.83) in pre- included sensitivity, specificity, positive and negative predic- dicting severe AKI (stage 2 or 3) and to 0.80 (95% CI, 0.72 to tive values, positive and negative likelihood ratios, odds ratios, 0.88) for only stage 3 AKI (Supplemental Figure 6). In a sub- and median time from crossing cutoff to AKI development group analysis, the genetic algorithm model performed the with IQRs. Additionally, we performed a sensitivity analysis best among newborns (AUC 0.81; 95% CI, 0.77 to 0.86) and on different pediatric age groups using the high- and low-risk worse in the adolescent age group (AUC 0.67; 95% CI, 0.60 to cutoff threshold levels. 0.73), better in children with CKD (AUC 0.91; 95% CI, 0.78 to Statistical analyses were performed using SAS version 9.4 0.99) than children with malignancy (AUC 0.61; 95% CI, (SAS Institute Inc., Cary, NC), Stata version 15 (StatCorp, 0.52 to 0.69), and better in ICU (AUC 0.77; 95% CI, 0.75 College Station, TX), R version 3.3.3 (R Foundation for Sta- to 0.82) than non-ICU populations (AUC 0.68; 95% CI, tistical Computing, Vienna, Austria), and the scikit-learn in 0.62 to 0.74). Python version 3.6. To apply the predictive model in clinical practice, we took into consideration the burden of alerting physicians and the development of alert fatigue versus the risk of missing too RESULTS many high-risk patients; therefore, we chose low- and high- risk cutoff threshold points using the highest-performing From January 1, 2014 to January 30, 2018, a total of model: the genetic algorithm model (Supplemental Tables 4 6328 children (8608 encounters) were hospitalized and had and 5). The low-risk cutoff threshold point of 0.08 predictive at least two serum creatinine values measured (Figure 2). probability would identify the vast majority of patients at risk One hundred and thirty-five encounters were excluded due of AKI with a negative predictive value of 97.9% in the internal to an initial serum creatinine level of .4 mg/dl and/or an validation cohort, but would include many patients who do eGFR of ,15 ml/min per 1.73 m2. Median age was 4.4 years not develop AKI and is thus well suited for low-level interven- (IQR, 0.04–13.1), 4614 (54.4%) were male, and 2608 (30.8%) tions (such as further monitoring of serum creatinine). The encounters had at least some time in an ICU [13% in the high-risk cutoff threshold point of 0.24 predictive probability

JASN 31: 1348–1357, 2020 EHR-Based Pediatric AKI Prediction 1351 CLINICAL RESEARCH www.jasn.org

8608 hospital admissions (6328 unique patients) From 2014 to 2018

135 excluded 97 eGFR<15 1 initial creatinine>4 37 met both

8473 hospital admissions

Yale-New Haven Bridgeport Hospital Children’s Hospital

Derivation cohort Internal validation cohort External validation cohort n=5072 n=2299 n=1102

AKI No AKI AKI No AKI AKI No AKI n=516 n=4556 n=207 n=2092 n=27 n=1075 (10.2%) (89.8%) (9%) (91%) (2.5%) (97.5%)

Figure 2. Patient population and cohort allocation. To build the prediction model, study population was divided into derivation, and internal and external validation cohorts. would identify a smaller percentage of patients who are at a clinical decision making and augment prediction of impor- substantially higher risk of developing AKI with a 99.6% spec- tant outcomes. Leveraging the EHR to predict AKI in chil- ificity, 12-fold increase in baseline positive predictive value to dren may be a low-cost approach to prevent incident AKI 26.6% (positive likelihood ratio of 12.5), would provide a me- or limit the severity of AKI. We used multiple feature selec- dian time of 23.7 hours (IQR 5.3–343.7) prior to AKI devel- tion techniques to develop and validate a parsimonious opment, and would be well suited toward more intensive or risk prediction model for AKI in hospitalized neonates costly interventions (such as empirical volume restoration or and children. We determined that a genetic feature selec- pharmacist consultation) (Figure 4). The suggested low-risk tion algorithm was the highest-performing model. We also threshold point of 0.08 showed nearly similar performance in identified potential AKI predictors such as sodium bicar- the external validation cohort as compared with the internal bonate use and glucose. All of the information required by validation cohort, with a specificity of 97.3% and negative pre- this genetic model is reported in the EHR and could be used dictive value of 99.5%. On the other hand, the suggested high- in real time to identify children at the highest risk of im- risk cutoff threshold point of 0.24 did not identify any patients pending AKI, allowing clinicians to intervene prior to AKI at risk of AKI in the external validation cohort (no true posi- development. tives) (Supplemental Table 5). Supplemental Tables 6 and 7 A few studies have attempted to develop EHR-based AKI show the sensitivity analysis of these cutoff threshold points prediction models. Sanchez-Pinto et al.9 created a pediatric at different age groups. Our web-based AKI risk prediction EHR-based AKI risk prediction model, which had good dis- calculator uses the features from the genetic algorithm model crimination for AKI, but was limited to children admitted to and can be accessed at https://yalepatr.shinyapps.io/TRACK. the PICU and did not include neonates. Additionally, their prediction model only uses data available in the first 12 hours of a PICU admission to predict AKI within the first 72 hours of DISCUSSION a PICU stay. Another EHR-based prediction model was created by Wang et al.,10 which had modest discrimination for The utility of the EHR has expanded from its inception as AKI with AUCs of 0.74 in the ICU and 0.69 for non-ICU primarily a data storage medium to its potential today to guide patients. However, this study only included predictor variables

1352 JASN JASN 31: 1348–1357, 2020 www.jasn.org CLINICAL RESEARCH

Table 1. Baseline characteristics of patient groups Variables Derivation (N55072) Internal Validation (N52299) External Validation (N51102) Demographic Age, yr (range) 5 (0.2–13.6) 5.2 (0.2–12.8) 0.2 (0–10.1) Ethnicity (Hispanic), n (%) 1347 (26.6) 577 (25.1) 404 (36.7) Race (black), n (%) 1048 (20.7) 466 (20.3) 296 (26.9) Sex (male), n (%) 2756 (54.3) 1252 (54.5) 606 (55) Medical history, n (%) Congenital heart disease 674 (13.3) 248 (10.8) 89 (8.1) CKD 77 (1.5) 23 (1) 2 (0.2) Malignancy 841 (16.6) 405 (17.6) 24 (2.2) Inpatient location, n (%) Pediatric ICU 758 (14.9) 362 (15.7) 0 (0) Neonatal ICU 818 (16.1) 348 (15.1) 292 (26.5) General pediatrics 1288 (25.4) 602 (26.2) 546 (49.5) Pediatric surgery 518 (10.2) 252 (11) 0 (0) Hematology/Oncology 693 (13.7) 315 (13.7) 0 (0) Nursery 249 (4.9) 106 (4.6) 218 (19.8) Laboratory (ﬁrst values) Serum creatinine, mg/dl 0.5 (0.3–0.8) 0.5 (0.3–0.7) 0.7 (0.5–0.8) eGFR, ml/min per 1.73 m2 99.4 (55.3–132.1) 101.9 (58.7–135.6) 61.2 (25.6–105.3) BUN, mg/dl 11 (8–16) 11 (8–16) 11 (8–15) Glucose, mg/dl 100 (83–128) 101 (83–126) 82 (53–102) Bicarbonate, mmol/L 21 (18.7–23.5) 21 (18.5–23.4) 23 (20–25) Calcium, mg/dl 9.3 (8.7–9.8) 9.3 (8.7–9.8) 9.3 (8.4–9.8) Platelet count, 31000/ml253(183–341) 259 (185–351) 251 (206–314) Lymphocyte percent (%) 21.6 (9–37) 24 (10–42) 4 (2.2–7.1) INR measured (%) 1656 (32.6) 772 (33.6) 103 (9.3) Fibrinogen measured (%) 497 (9.8) 219 (9.5) 8 (0.7) Lactate measured (%) 1947 (38.4) 883 (38.4) 89 (8.1) Medication,a n (%) Loop diuretic 922 (18.2) 348 (15.1) 117 (10.6) Vasopressor 529 (10.4) 218 (9.5) 23 (2.1) Chemotherapy 527 (10.4) 253 (11) 0 (0) Sodium bicarbonate 353 (7) 133 (5.8) 4 (0.4) Calcium gluconate 216 (4.3) 86 (3.7) 3 (0.3) Alprostadil 43 (0.8) 13 (0.6) 0 (0) Carboplatin 13 (0.3) 13 (0.6) 0 (0) Foscarnet 5 (0.1) 2 (0.1) 0 (0) Paclitaxel 2 (0) 0 (0) 0 (0) Nimodipine 1 (0) 0 (0) 0 (0) Procedure, n (%) Mechanical ventilation 475 (9.4) 193 (8.4) 48 (4.4) RBC transfusion 1150 (22.7) 457 (19.9) 136 (12.3) Data are presented as median (IQR) or proportion. All comparisons across the groups have P,0.01. INR, international normalized ratio; RBC, red blood cell. aIf a medication was ever given during hospitalization for non-AKI encounters or prior to AKI for AKI encounters.

3 days before AKI and patients younger than 28 days old were may improve overall predictive ability. To build a model excluded. that can be widely used in EHR systems, we decided to in- In contrast to the previously published AKI prediction clude all children younger than 18 years and admitted in all models, our prediction approach uses time-varying data, units of the hospital. allowing the model to incorporate clinical events that occur Neonates represent a unique pediatric population in which during hospitalization and making predictions that are con- their glomerular and tubular function is still maturing, and tinuously updated in real time. We also used a clinically their creatinine levels initially reﬂect maternal creatinine. As a agnostic approach in selecting predictor variables, includ- result, neonates are often excluded from pediatric AKI re- ing a wide array of variables available in the EHR. This ap- search studies. However, neonates are subjected to many of proach allows for the identiﬁcation of novel predictors of the same risk factors as other children such as kidney ischemia, AKI and, insofar as some of those predictors are strong, surgery, sepsis, and nephrotoxin exposure.4,11 Furthermore,

JASN 31: 1348–1357, 2020 EHR-Based Pediatric AKI Prediction 1353 CLINICAL RESEARCH www.jasn.org

Table 2. Feature selection methods with comparison based on AUCs Model AUC (95% CI) Model Name Feature Selection Method Derivation Internal Validation External Validation Clinical Clinical experience and literature review 0.73 (0.71 to 0.76) 0.72 (0.69 to 0.76) 0.76 (0.66 to 0.87) Lasso LASSO 0.77 (0.75 to 0.79) 0.73 (0.69 to 0.77) 0.79 (0.70 to 0.88) Top AUC Highest AUCs in univariable analysis 0.76 (0.73 to 0.78) 0.74 (0.70 to 0.77) 0.83 (0.76 to 0.91) Forward Forward selection 0.78 (0.75 to 0.80) 0.73 (0.69 to 0.77) 0.84 (0.78 to 0.91) Genetic Genetic algorithm 0.79 (0.77 to 0.81) 0.76 (0.72 to 0.79) 0.87 (0.81 to 0.92) The genetic algorithm signiﬁcantly improves prediction of AKI as compared with other feature selection methods (P,0.05 for comparing the genetic algorithm model with clinical, LASSO, and forward models in internal validation cohort).

AKI in neonates occurs at similar rates as PICU patients, and is In this study, we noted that prediction modeling using ma- strongly associated with length of stay and mortality. chine learning feature selection methods had higher predictive Whereas including more variables in the model can im- ability, for example an AUC of 0.76 (95% CI, 0.72 to 0.79) in prove performance and increase the AUC, we aimed to create the genetic algorithm model, compared with a clinical a prediction model that can be practically applied in the EHR. experience-based model with an AUC of 0.72 (95% CI, This is an effort to recognize that for prediction methods to be 0.69 to 0.76), emphasizing the increased efﬁcacy of clinically useful, they must be prospectively implemented into the EHR agnostic machine learning-based prediction models over that and every variable is a potential break point. New or modiﬁed of current models based on clinical intuition and experience. laboratory tests, medication names, and hospital locations Despite the generally good discriminatory ability of the may change over time and result in inaccurate predictions. models, we observed a less robust precision recall AUC. This

Table 3. Variables selected by different feature selection models Variable Clinical Lasso Top AUC Forward Genetic BUN 11 1 1 1 Time since admission —— — — Sodium bicarbonate use 111 Glucose 11 1 Creatinine 11 1 1 Calcium —— — RBC transfusion 11 Loop diuretic use 11 Alprostadil use 11 D creatinine 48 h 11 Chemotherapy use 1 1 General pediatrics admission — Carboplatin use 1 Foscarnet use 1 Nimodipine use 1 Paclitaxel use 1 Fibrinogena 1 Oxygen saturation — Lymphocyte percent — Platelet count — Respiratory rate — INRa 1 Calcium gluconate use 1 Bicarbonate — Lactatea 1 ICU admission 1 Mechanical ventilation 1 Pressor use 1 Weight 1 Variables at the top were selected by several feature selection methods. RBC, red blood cell; D creatinine 48 h, change in serum creatinine within the last 48 h; INR, international normalized ratio. aTransformed into categorical variables (measured versus not measured) due to missingness .25%; 1, positive correlation with AKI; –, negative correlation with AKI.

1354 JASN JASN 31: 1348–1357, 2020 www.jasn.org CLINICAL RESEARCH

Creatinine-based AKI definition 0.6 → 1.1

Creatinine = 0.7 0.20 Glucose = 120 1.1

Real-time Alert (Genetic model probability >8%) 0.16 1.0 /dL) g 0.12 0.9 Glucose = 134

0.08 Loop diuretic given 0.8

Predicted probability Glucose = 112 Sodium bicarb given Serum creatinine (m

0.04 Glucose = 132 0.7 RBC transfusion

0.00 0.6

36 24 12 0 Hours prior to AKI

Predicted probability Serum creatinine (mg/dL)

Figure 3. An example in one patient of the predictive probability of the genetic algorithm model along with the rise in serum creatinine until the development of AKI. In this example, our prediction model identiﬁed the patient to be at risk of AKI 10 hours before creatinine criteria were met. suggests that a signiﬁcant proportion of AKI in our population Using a clinically agnostic approach of feature selection by occurred without warning, at least insofar as warning signals including 720 predictor variables from the EHR allowed us to can be derived from the EHR. This has important implications identify clinical parameters not typically thought to be asso- for targeting interventions to at-risk children, as the models ciated with or causative of AKI. Sodium bicarbonate use and may identify those with an AKI prodrome and miss those with red blood cell transfusions are two examples of variables se- more sudden AKI. lected by machine learning algorithms that are not necessarily

Low risk High risk Usual Care Mild interventions recommended Intensive interventions recommended

Low-risk threshold (0.08) High-risk threshold (0.24) Sensitivity 28.8% Sensitivity 5.3% Specificity 94.1% Specificity 99.6% PPV 12.4% PPV 26.6% NPV 97.9% NPV 97.3% TP:FP 1:7 TP:FP 1:2.8 LR+ 4.9 LR+ 12.5 LR- 0.8 LR- 1 OR 6.5 OR 13.1 Median (IQR) time to AKI 142.7 hours Median (IQR) time to AKI 23.7 hours (32.7-300.3) (5.3-343.7)

Figure 4. Diagnostic test characteristics of the genetic model at two cutoff threshold points. A low-risk cutoff threshold point can be used to identify patients suitable for low-risk interventions (such as monitoring serum creatinine). A high-risk cutoff threshold point can be used to apply costly, risky, or intensive interventions (such as medication and ﬂuid changes). LR1, positive likelihood ratio; LR-, negative likelihood ratio; NPV, negative predictive value; OR, odds ratio; PPV, positive predictive value; TP:FP, true positive to false positive ratio.

JASN 31: 1348–1357, 2020 EHR-Based Pediatric AKI Prediction 1355 CLINICAL RESEARCH www.jasn.org causative, but can reflect the severity of a patient’s illness and ACKNOWLEDGMENTS predict the development of AKI. A clinically agnostic feature selection approach may thus generate hypotheses that would The authors acknowledge Boian Etropolski for his assistance in de- not have been investigated under a purely hypothesis-driven veloping the online risk prediction calculator. paradigm. This paradigm has no ability to assess causality. The I. Sandokji, J.H. Greenberg, and F.P. Wilson designed the study; model may be predictive of the outcome, but there’snocur- I. Sandokji, Y. Yamamoto, A. Biswas, and I. Saran reviewed medical rent indication to change clinical management to affect the charts; I. Sandokji, Y. Yamamoto, A. Biswas, and F.P. Wilson analyzed features that appear in the model. This model is to be used the data; I. Sandokji, Y. Yamamoto, and M. Simonov made the figures; purely for prognostic purposes and not for treatment deci- I. Sandokji, Y. Yamamoto, A. Biswas, T. Arora, U. Ugwuowo, I. Saran, sions at the variable level. M. Martin, J.M. Testani, S. Mansour, D.G. Moledina, J.H. Greenberg, Our study has several strengths. We created a continuously and F.P. Wilson drafted and revised the paper; and all authors ap- updating AKI prediction model using EHR data. This model proved the final version of the manuscript. requires only ten variables that are commonly available in the EHRs. We used a clinically agnostic approach in selecting pre- DISCLOSURES dictor variables from a large number of candidate variables utilizing several machine learning algorithms, which allowed us to identify several novel predictors of AKI. Dr. Testani reports grants and personal fees from Sequana Medical, BMS, 3ive labs, Boehringer Ingelheim, Sanofi, and FIRE1; personal fees from We also acknowledge several limitations of our study. First, AstraZeneca, Novartis, Cardionomic, Bayer, MagentaMed, Renalguard, including all variables in the EHR forced us to impute a large and W.L. Gore; and grants from Otsuka and Abbott outside the submitted number of laboratory variables due to missingness of categor- work. Dr. Mansour reports grants and other from the American Heart ical variables. However, predictive potential would only be Association and the Patterson Trust Fund during the conduct of the study. reduced by imputation, implying that more robust measure- Dr. Moledina reports grants from the National Institutes of Health/National Institute of Diabetes and Digestive Kidney Diseases during the conduct of the ments of risk factors would further improve predictive per- study. In addition, Dr. Moledina has a patent system and methods for diagnos- formance. Additionally, medication variables were treated as ing acute interstitial nephritis pending. Dr. Wilson reports grants from the exposures regardless of their doses. We chose to not include National Institute of Diabetes and Digestive Kidney Diseases during medication doses to avoid increasing the complexity of the the conduct of the study. Dr. Wilson is the founder of Efference, LLC, a model in prospective implementation, because medication medical communications company. doses vary based on age, weight, treatment indication, and kidney function in children. We acknowledge the limited cal- FUNDING ibration performance in the external validation cohort (Bridgeport Hospital), however the model maintained the dis- Dr. Wilson received support from National Institute of Diabetes and Digestive crimination measures. As noted, patients in this cohort are of and Kidney Diseases (NIDDK) grants R01 DK113191 and P30 DK079310. lower acuity and there were only 27 AKI events, which high- Dr. Greenberg is funded by career development grant K08DK110536. This research is funded by a Charles H. Hood Foundation, Inc. grant (to Dr. Greenberg). lights the need to recalibrate such prediction models when Dr. Moledina is funded by NIDDK grant K23DK117065. used at different institutions. Furthermore, most patients in this cohort did not cross the high-risk threshold point of 0.24, and therefore using a high threshold point can have limitations SUPPLEMENTAL MATERIAL in a low-acuity patient population. We recommend examining different threshold points on the population of interest before This article contains the following supplemental material online at implementing the model. Lastly, the nature of retrospective http://jasn.asnjournals.org/lookup/suppl/doi:10.1681/ASN.2019070745/-/ studies limits our ability to make conclusions on the causal DCSupplemental. relationship between predictor variables and outcome. Supplemental Table 1. Predictor variables used to create parsi- In conclusion, we present different feature selection para- monious prediction models. digms to create a time-updated, parsimonious, clinically ag- Supplemental Table 2. Comparing demographic and clinical nostic prediction modeling approach for pediatric AKI that characteristics according to development of AKI in the derivation can be applied in real-time to all pediatric patients. After pro- cohort. spectively assessing the prognostic performance of the model, Supplemental Table 3. Regression-based prediction models coef- the best test of clinical utility is to determine if interventions, ficients and odds ratios. targeted to those crossing a given risk threshold, can mean- Supplemental Table 4. Genetic algorithm model operating points ingfully change clinical outcomes. Prediction models of AKI applied to the internal validation cohort. could also be further studied to identify children at high risk of Supplemental Table 5. Genetic algorithm model operating points AKI for enrollment in clinical trials. These high-risk individ- applied to the external validation cohort. uals would be the most likely to develop AKI and may benefit Supplemental Table 6. Low-risk cut-off threshold (0.08) analysis from real-time interventions, such as alerts for nephrotoxin using the genetic algorithm model in age subgroups (on the internal avoidance and best-practice advisories. validation cohort).

1356 JASN JASN 31: 1348–1357, 2020 www.jasn.org CLINICAL RESEARCH

Supplemental Table 7. High-risk cut-off threshold (0.24) analysis to predict acute kidney injury risk and outcomes: Workgroup state- using the genetic algorithm model in age subgroups (on the internal ments from the 15(th) ADQI Consensus Conference. Can J Kidney validation cohort). Health Dis 3: 11, 2016 9. Sanchez-Pinto LN, Khemani RG: Development of a prediction model of Supplemental Figure 1. Logistic regression analyses of incremental early acute kidney injury in critically ill children using electronic health numbers of variables of the stepwise forward selection model. record data. Pediatr Crit Care Med 17: 508–515, 2016 Supplemental Figure 2. Genetic algorithm feature selection 10. Wang L, McGregor TL, Jones DP, Bridges BC, Fleming GM, Shirey-Rice process. J, et al: Electronic health record-based predictive models for acute – Supplemental Figure 3. Convergence in evaluation set (of deri- kidney injury screening in pediatric inpatients. Pediatr Res 82: 465 473, 2017 vation cohort) area under the receiver operating characteristic curve 11. Selewski DT, Charlton JR, Jetton JG, Guillet R, Mhanna MJ, Askenazi (AUC) of the genetic algorithm’s population. DJ, et al: Neonatal acute kidney injury. Pediatrics 136: e463–e473, Supplemental Figure 4. Genetic algorithm model calibration 2015 curves. 12. Basu RK, Zappitelli M, Brunner L, Wang Y, Wong HR, Chawla LS, et al: Supplemental Figure 5. Genetic algorithm model performance Derivation and validation of the renal angina index to improve the prediction of acute kidney injury in critically ill children. Kidney Int 85: illustrated by a precision-recall curve. 659–667, 2014 Supplemental Figure 6. Genetic model for the prediction of severe 13. Sanchez-Pinto LN, Venable LR, Fahrenbach J, Churpek MM: Compar- AKI and subgroup analyses in the internal validation cohort. ison of variable selection methods for clinical predictive modeling. Int Supplemental Methods. Genetic Algorithm Methodology. JMedInform116: 10–17, 2018 14. Pottel H, Dubourg L, Goffin K, Delanaye P: Alternatives for the bedside Schwartz equation to estimate glomerular filtration rate in children. Adv Chronic Kidney Dis 25: 57–66, 2018 REFERENCES 15. Selewski DT, Cornell TT, Heung M, Troost JP, Ehrmann BJ, Lombel RM, et al: Validation of the KDIGO acute kidney injury criteria in a pediatric 1. Kaddourah A, Basu RK, Bagshaw SM, Goldstein SL; AWARE Investi- critical care population. Intensive Care Med 40: 1481–1488, 2014 gators: Epidemiology of acute kidney injury in critically ill children and 16. Steyerberg EW: Clinical Prediction Models: A Practical Approach to young adults. NEnglJMed376: 11–20, 2017 Development, Validation, and Updating, Berlin, Germany, Springer 2. Sutherland SM, Ji J, Sheikhi FH, Widen E, Tian L, Alexander SR, et al: Science & Business Media, 2008 AKI in hospitalized children: Epidemiology and clinical associations in a 17. Tibshirani R: Regression shrinkage and selection via the Lasso. JRStat national cohort. Clin J Am Soc Nephrol 8: 1661–1669, 2013 Soc B 58: 267–288, 1996 3. Greenberg JH, Coca S, Parikh CR: Long-term risk of chronic kidney 18. Choi SJ, Ha EJ, Jhang WK, Park SJ: Factors associated with mortality in disease and mortality in children after acute kidney injury: A systematic continuous renal replacement therapy for pediatric patients with acute review. BMC Nephrol 15: 184, 2014 kidney injury. Pediatr Crit Care Med 18: e56–e61, 2017 4. Kwiatkowski DM, Sutherland SM: Acute kidney injury in pediatric pa- 19. Selewski DT, Symons JM: Acute kidney injury. Pediatr Rev 35: 30–41, tients. Best Pract Res Clin Anaesthesiol 31: 427–439, 2017 2014 5. Jetton JG, Boohaker LJ, Sethi SK, Wazir S, Rohatgi S, Soranno DE, et al; 20. Thongprayoon C, Cheungpasitporn W, Mao MA, Sakhuja A, Erickson Neonatal Kidney Collaborative (NKC): Incidence and outcomes of SB: Admission calcium levels and risk of acute kidney injury in hospi- neonatal acute kidney injury (AWAKEN): A multicentre, multinational, talised patients. Int J Clin Pract 72: e13057, 2018 observational cohort study. Lancet Child Adolesc Health 1: 184–194, 21. Vafaie H, Jong KD: Genetic algorithms as a tool for feature selection in 2017 machine learning. Presented at the Fourth International Conference on 6. Silver SA, Chertow GM: The economic consequences of acute kidney Tools with Artificial Intelligence 1992, Arlington, VA, November 10–13, injury. Nephron 137: 297–301, 2017 1992 7. Goldstein SL, Mottes T, Simpson K, Barclay C, Muething S, Haslam DB, 22. Newson R: Parameters behind “nonparametric” statistics: Kendall’s et al: A sustained quality improvement program reduces nephrotoxic tau, Somers’ D and median differences. Stata J 2: 45–64, 2002 medication-associated acute kidney injury. Kidney Int 90: 212–221, 2016 8. Sutherland SM, Chawla LS, Kane-Gill SL, Hsu RK, Kramer AA, Goldstein SL, et al; 15 ADQI Consensus Group: Utilizing electronic health records J.H.G. and F.P.W. contributed equally to this work.

JASN 31: 1348–1357, 2020 EHR-Based Pediatric AKI Prediction 1357

Supplemental Material

TABLE OF CONTENTS

Supplemental Table 1. Predictor variables used to create parsimonious prediction models. Supplemental Table 2. Comparing demographic and clinical characteristics according to development of AKI in the derivation cohort. Supplemental Table 3. Regression-based prediction models coefficients and odds ratios. Supplemental Table 4: Genetic algorithm model operating points applied to the internal validation cohort.

Supplemental Table 5: Genetic algorithm model operating points applied to the external validation cohort. Supplemental Table 6: Low-risk cut-off threshold (0.08) analysis using the genetic algorithm model in age subgroups (on the internal validation cohort). Supplemental Table 7: High-risk cut-off threshold (0.24) analysis using the genetic algorithm model in age subgroups (on the internal validation cohort). Supplemental Figure 1: Logistic regression analyses of incremental numbers of variables of the stepwise forward selection model. Supplemental Figure 2. Genetic algorithm feature selection process.

Supplemental Figure 3: Convergence in evaluation set (of derivation cohort) area under the receiver operating characteristic curve (AUC) of the genetic algorithm’s population. Supplemental Figure 4: Genetic algorithm model calibration curves. Supplemental Figure 5: Genetic algorithm model performance illustrated by a precision-recall curve. Supplemental Figure 6. Genetic model for the prediction of severe AKI and subgroup analyses in the internal validation cohort. Supplemental Methods: Genetic Algorithm Methodology.

Supplemental Table 1. Predictor variables used to create parsimonious prediction models.

Demographic variables (n=7) • Age • Height • Sex • Weight • Ethnicity • Race • Time since admissiona Vital signs (n=6) • Arterial line diastolic blood • Diastolic blood pressure • Oxygen saturation • Systolic blood pressure pressure • Heart rate • Respiratory rate Locations variables (n=24) • Cardiology • Hematology/oncology • NICU • SICU • Catheter lab • Interventional radiology • Nursery • Surgery • CTICU • Maternity • Perioperative • Surgical oncology • Emergency room • MICU • PICU • Transplant • Endoscopy suite • Neurological ICU • Psychiatry • Transport • General pediatrics • Neurology Procedures variables (n=85) • Abdomen CT • Critical care transport • Limited code order • Renal diet • Abdomen MR • Head CT • Liver diet • Renal US • Abdomen US • Head MR • Low salt diet • Restraints • Abdomen XR • Educational US • Mechanical ventilation • SCD order • Arterial line placement • EEG • Neonatal abstinence score • Sepsis trigger order • Autopsy • ETT adjustment • NIPPV • Soft tissue US • Biopsy • Extremities CT • Nitric oxide • Speech swallow test • BiPAP • Extremities MR • NPO order • Spine MR • Blood culture • Extubation • Nuclear scan • Spine US • Bone XR • Fall risk order • Oxygen • TEE • Car seat test • Feeding tube • Paracentesis • Telemetry • Cardiac catheter • Fluoroscopy study • Phototherapy • TTE • Central line • Foley catheter • Physical therapy • Turn patient order • Chest CT • Foreign body XR • PICC placement • Umbilical line placement • Chest MR • Full code status • PICU consult • Urine culture • Chest PT • Gadolinium study • Plasma transfusion • Venous US • Chest XR • Head US • Plasmapheresis • Ventilation order • Comfort code order • HFOV • Platelet transfusion • Wound care • Consult DART • High flow oxygen • Posttransfusion reaction • Wound VAC • Contact precaution order • Hip US • RBC transfusion • 1:1 Order (Sitter) • Contrast study • Contrast IR procedure • Rectal tube • CPAP • Isolette placement Continuous laboratory variables (n=27) • Absolute lymphocyte count • Chloride • Lymphocytes • Potassium • Anion gap • Creatinine • MCHC • RBC count • Bicarbonate • eGFR • MCV • RDW • BUN • Eosinophils • Monocytes • Sodium • BUN/creatinine ratio • Glucose • MPV • WBC count • Calcium • Hematocrit • Neutrophils • Δ Creat 48 • Carbon dioxide • Hemoglobin • Platelet count Medication groups variables (n=34) • ACEi/ARB • Anti-thymocyte globulin • CMV immunoglobin • Potassium-sparing • Vasopressors • ACEi • Breast milk • Diuretics diuretics • Sedatives • Aminoglycosides • ß-blockers • Fluoroquinolones • Loop diuretics • Statins • Antibiotics • ß-lactam antibiotics • HAART • Narcotics • Steroids • Antidepressants • Calcium channel • Immune globulin • NSAIDs • Thiazide diuretics • Antifungals blockers • Immunomodulators • Pancreatic enzymes • Total parental • ARB • Chemotherapy • Insulins • Paralytic agents nutrition • Iodine contrasts • PPIs

Continued Supplemental Table 1. Predictor variables used to create parsimonious prediction models.

Categorical laboratory variables (n=222) • 1,25 vitamin D • Blood smear • EBV PCR • LDH • Total complement • 25, vitamin-D2 • Carbamazepine • EKG heart rate • Lipase • Total creatine kinase • ABO group • Brain natriuretic • EKG QRS • Lithium • Total protein • Acetaminophen peptide • EKG QRS axis • Lyme Ab • Triglycerides • Acetylcholine Ab • Bone marrow • EKG R axis • Random urine • Troponin • Activated Coagulation aspiration • Erythrocyte creatinine • TSH • Brucella Ab Time sedimentation rate • Magnesium • Type/screen • Adrenocorticotropic • C-peptide • Ethanol • MCH • UA appearance hormone • C3 complement • Ferritin • UA bacteria • Methemoglobin • ADAMTS13 • C4 complement • Fibrinogen • UA glucose • Mixed venous CO2 • Adenovirus DFA • Caffeine • FiO2 • UA leukocyte esterase • Adenovirus PCR • Carboxyhemoglobin • Flow cytometry • Mixed venous pH • Urine • Adenovirus quantitative • Carnitine • Gentamicin peak • MRSA albumin/creatinine • AFP • Catheter culture • Gentamicin trough • Nucleated RBC • Urea/nitrogen in urine • Albumin • Complete blood count • Gamma glutamyl • Osmolality • Urinalysis • Aldolase • CD-19 lymphocytes transferase • Oxygen saturation • Urine bacteria • Alkaline phosphatase • CD-20 lymphocytes • GMB Ab • pCO2 • Urine barbiturate • Alpha 1 antitrypsin • CD-21 lymphocytes • Haptoglobin • P-ANCA • Urine ß-2 microglobulin • ALT • CD-22 lymphocytes • HbA1C • Pertussis Ab • Urine BK • Amikacin • Celiac disease screen • HDL • pH • Urine calcium • Ammonia • Chlamydia DNA • Heparin induced Ab • Phosphorous • Urine calcium oxalate • amylase • Cholesterol • Hep-B E Ag • Plasma amino acid • Urine calcium • ANA • Chromosomal analysis • Hep-B quantitative • Plasma hemoglobin pyrophosphate • ANA pattern • C. difficile Ag • Hep-B surface Ab • Plasma metanephrines • Urine chloride • ANCA screen • C. difficile PCR • Hep-B surface Ag • pO2 • Urine creatinine • Anticardiolipin • C. difficile toxin • Hep-C Ab • Prealbumin • Urine culture • Anti-glial nuclear Ab • CK-MB • Hep-C genotype • Procalcitonin • Urine eosinophils • Arterial HCO3 • CMV Ab • Hep-C quantitative • PTH • Urine fine casts • Arterial pCO • CMV quantitative • High sensitivity CRP • Qualitative CMV • Urine glucose • Arterial pCO2 • CMV PCR • HIV Ab • Random urine calcium • Urine gram stain • Arterial pO2 • Copper • HIV-1 Ab • Rapamycin • Urine granular casts • ASO titer • Cortisol • HIV-1 RNA • RBC morphology • Urine hyaline casts • AST/ALT ratio • Coxackie-B Ab • HIV-2 Ab • RDW • Urine norepinephrine • Relative eosinophils • Atypical lymphocytes • C-reactive protein • Immature granulocytes • Urine organic acids • Relative lymphocytes • Atypical P-ANCA • Creatine kinase • Influenza Ag • Urine osmolality • Autopsy • CSF albumin • Relative neutrophils • Urine phosphate • Influenza A Ab • Babesia Ab • CSF culture • Renin • Urine potassium • Influenza A PCR • Babesia PCR • Cyclosporine • Reticulocytes • Urine RBC • Bands • D-dimer • Influenza B Ab • Rheumatoid factors • Urine sodium • Bartonella Ab • Direct bilirubin • Influenza PCR • Salisylate • Urine uric acid • Base excess • Direct fluorescent Ab • INR • Serine protease 3 • Urine waxy casts • Basophilic stippling • DNAse • Intact PTH • Stool PCR • Urine WBC cast • BK PCR • Double stranded DNA • Ionized calcium • Tacrolimus • Valproate • BK quantitative • E.Coli shiga toxin • Iron • Tuberculosis test • Vancomycin trough • Blasts assay • Lactate • Thrombin time • Venous O2 saturation • Blood culture • Hep-B core Ab • Lead • Thyroxine • Venous pO2 • Blood ketones • EBV Ag • Leukemia panel • Total bilirubin

Continued Supplemental Table 1. Predictor variables used to create parsimonious prediction models.

Individual medications variables (n=315) • Acetaminophen • Calcium gluconate • Diphenhydramine • Ibuprofen • Milrinone • Acetazolamide • Captopril • Dobutamine • Ifosfamide • Mirtazapine • Acetylcysteine • Carbamazepine • Docusate • Indomethacin • Misoprostol • Acyclovir • Carboplatin • Dolutegravir • Iodixanol • Mitoxantrone • Adalimumab • Carvedilol • Dopamine • Infliximab • Morphine • Adenosine • Cefazolin • Dornase alpha • Interferon alpha • Moxifloxacin • Albumin • Cefdinir • Doxazosin • Iohexol • Mycophenolate • Albuterol • Cefotaxime • Doxorubicin • Iopamidol • Nadolol • Aldesleukin • Ceftazidime • Doxycycline • Ipratropium • Nafcillin • Alemtuzumab • Ceftriaxone • Dronabinol • Irinotecan • Naloxone • Allopurinol • Cefuroxime • Duloxetine • Isoniazid • Naproxen • Alprazolam • Celecoxib • Eculizumab • Isradipine • Neomycin • Alprostadil • Cephalexin • Enalapril • Levetiracetam • Nicardipine • Alteplase • Certolizumab • Enoxaparin • Ketamine • Nifedipine • Amantadine • Charcoal • Entecavir • Ketorolac • Nimodipine • Ambisome • Chlorpromazine • Epinephrine • Labetalol • Nitrofurantoin • Amikacin • Ciprofloxacin • Epoetin alpha • Lactulose • Nitroprusside • Amiodarone • Cisatracurium • Ertapenem • Lamotrigine • Norepinephrine • Amitriptyline • Cisplatin • Erythromycin • Lansoprazole • Nortriptyline • Amlodipine • Citalopram • Esmolol • Leucovorin • Nystatin • Amoxicillin • Cladribin • Ethosuximide • Levocarnitine • Octreotide • Amoxicillin / clavulanate • Clarithromycin • Etoposide • Levofloxacin • Olanzapine • Amphotericin • Clindamycin • Ezetimibe • Levothyroxine • Omeprazole • Ampicillin • Clobazam • Famotidine • Lidocaine • Ondansetron • Ampicillin / sulbactam • Clofarabin • Fentanyl • Lisinopril • Oseltamivir • Anakinra • Clonazepam • Ferrous sulfate • Lithium • Oxacillin • Anidulafungin • Clonidine • Ferrous sucrose • Loperamide • Oxcarbazepine • Aripiprazole • Clopidogrel • Filgrastim • Lorazepam • Oxybutynin • Asparaginase • Codeine • Fluconazole • Losartan • Oxycodone • Aspirin • Corticotropin • Fludarabin • Magic mouthwash • Paclitaxel • Atenolol • Cosyntropin • Fludrocortisone • Magnesium • Palivizumab • Atorvastatin • Creatine • Fluorouracil • Mannitol • Pantoprazole • Atovaquone • Cyclophosphamide • Fluoxetine • Meclizine • Paroxetine • Atropine • Cyclosporine • Foscarnet • Melatonin • Pegaspargase • Azathioprine • Cyproheptadine • Furosemide • Meperidine • Penicillin • Azithromycin • Cytarabine • Gabapentin • Mercaptopurine • Pentamidine • Aztreonam • Dactinomycin • Ganciclovir • Meropenem • Pentobarbital • Baclofen • Daptomycin • Gemcitabine • Mesalamine • Perphenazine • Barium • Daunorubicin • Gentamicin • Mesna • Phenazopyridine • Basiliximab • Decitabin • Glucagon • Metformin • Phenobarbital • Bleomycin • Deferasirox • Granisetron • Methadone • Phenylephrine • Budesonide • Desmopressin • Haloperidol • Methimazole • Bumetanide • Dexamethasone • Heparin • Methotrexate • Phenytoin • Bupropion • Dexmedetomidine • Hydrochlorothiazide • Methylprednisolone • Piperacillin / tazobactam • Busulfan • Dextrose drip • Hydralazine • Metoclopramide • Posaconazole • Caffeine • Diatrizoate • Hydrocodone • Metolazone • Potassium • Calcitriol • Diazepam • Hydrocortisone • Metoprolol • Pravastatin • Calcium carbonate • Diazoxide • Hydromorphone • Metronidazole • Prazosin • Calcium chloride • Digoxin • Hydroxychloroquine • Midazolam • Prednisone • Calcium citrate • Diltiazem • Hydroxyurea • Midodrine • Pregabalin • Calcium glubionate • Dinutuximab • Hypertonic saline

Continued Supplemental Table 1. Predictor variables used to create parsimonious prediction models.

Individual medications variables (n=315) • Prochlorperazine • Rituximab • Tamsulosin • Vancomycin • Propofol • Saline bolus • Technetium • Vasopressin • Propranolol • Saline drip • Temozolomide • Venlafaxine • Prostaglandin • Sertraline • Tenofovir • Verapamil • Protamine • Sevelamer • Terazosin • Vinblastine • Pseudoephedrine • Sildenafil • Terbutaline • Vincristine • Pyridoxine • Simethicone • Theophylline • Vitamin C • Quetiapine • Sirolimus • Thymoglobulin • Vitamin D • Raltegravir • Sodium acetate • Tobramycin • Vitamin K • Ranitidine • Sodium bicarbonate • Topiramate • Voriconazole • Rasburicase • Sodium citrate • Topotecan • Warfarin • Rifampin • Sotalol • Torsemide • Ziprasidone • Rifaximin • Spironolactone • Tramadol • Zoledronic acid • Ringer lactate bolus • Sulfamethoxazole / • Trazodone • Zolpidem • Ringer lactate drip trimethoprim • Valganciclovir • Zonisamide • Risperidone • Sulfasalazine • Valproic • Tacrolimus Variables excluded from analysis due to high collinearity • 25, vitamin-D3 • Catecholamines • Prothrombin time • Urine color • A/G ratio • CD-23 lymphocytes • PTT • Urine epinephrine • Absolute neutrophil count • EKG P wave • RH type • Urine ketones • Absolute nucleated RBC • EKG PR interval • TIBC • Urine pH • Aldosterone • EKG PR interval • UA blood • Urine phencyclidine • ANA titer • EKG qrs interval • UA nitrite • Urine protein/creatinine • Arterial bicarbonate • Globulin • UA pH ratio • Arterial pH • HIV-2 RNA • UA protein • Urine uric acid • AST • Influenza B PCR • UA specific gravity • Urine WBC • Basophils • Influenza DFA • Urine albumin • Vancomycin random • Basophils percent • Iron saturation • Urine benzodiazepine • Venous bicarbonate • C. difficile enzyme • LDL • Urine bilirubin • Venous pCO immunoassay • Mixed venous O2 • Urine clarity • Venous pCO2 • C-ANCA • Oxygen saturation • Urine cocaine • Venous pH a log transformed as time since admission (log (time since addmission+1)). Ab, antibody; ACEi, angiotensin-converting enzyme inhibitor; Ag, antigen; A/G, albumin/globulin; ALT, alanine aminotransferase; ANA, antinuclear antibody; ANCA, antineutrophil cytoplasmic antibodies, ARB, angiotensin receptor blocker; ASO, antistreptolysin O; AST, aspartate aminotransferase; BiPAP, bilevel positive airway pressure; BUN, Blood urea nitrogen; CK-MB, creatine kinase-muscle/brain; CMV, cytomegalovirus; CPAP, continuous positive airway pressure; CSF, Cerebrospinal fluid; CT, computerized tomography; CTICU, Cardiothoracic Intensive Care Unit; DART, detection assessment research & treatment; HDL, high-density lipoproteins; DFA, direct fluorescent antibody; EBV, Epstein-Barr virus; EEG, electroencephalogram; eGFR, estimated glomerular filtration rate; EKG, electrocardiogram; ETT, endotracheal tube; FiO2, fraction of inspired oxygen; GBM, glomerular basement membrane; HAART, highly active antiretroviral therapy; HFOV, high frequency oscillatory ventilation; INR, international normalized ratio; IR, interventional radiology; LDH, lactate dehydrogenase; LDL, low-density lipoproteins; MCH, mean corpuscular hemoglobin; MCHC, mean corpuscular hemoglobin; MCV, mean corpuscular volume; MPV, mean platelet volume; MR, magnetic resonance imaging; MRSA, Methicillin-resistant Staphylococcus aureus; NICU, neonatal intensive care unit; NIPPV, noninvasive positive-pressure ventilation; NPO, nil per os; NSAID, nonsteroidal anti-inflammatory drugs; PACU, post-anesthesia care unit; pCO, partial pressure of carbon monoxide; pCO2, partial pressure of carbon dioxide; PCR, polymerase chain reaction; PICU, pediatric intensive care unit; pO2, partial pressure of oxygen; PPI, proton-pump inhibitors; PT, physical therapy; PTH, parathyroid hormone; PTT, partial thromboplastin time; SICU, surgical intensive care unit; RBC, red blood cells; RDW, red cell distribution width; SCD, sequential compression device; TEE, transesophageal echocardiography; TIBC, total iron binding capacity; TSH, thyroid stimulating hormone; TTE, transthoracic echocardiogram; UA, urinalysis; US, ultrasound; VAC, negative- pressure wound therapy; XR, x-ray; Δ Creat 48, Change in serum creatinine within the last 48 hours.

Supplemental Table 2. Comparing demographic and clinical characteristics according to development of AKI in the derivation cohort. AKI Non-AKI Variables p-value (N=516) (N=4556) Demographic Age, yr (range) 2.9 (0,13.3) 5.4 (0.2,13.6) <0.01 Ethnicity (Hispanic), n (%) 135 (26.2%) 1212 (26.6%) 0.85 Race (black), n (%) 126 (24.4%) 922 (20.2%) 0.05 Sex (Male), n (%) 289 (56%) 2467 (54.1%) 0.47 Medical History Congenital heart disease 149 (28.9%) 525 (11.5%) <0.01 CKD 24 (4.7%) 53 (1.2%) <0.01 Malignancy 105 (20.3%) 736 (16.2%) 0.05 Inpatient location, n (%) Pediatric ICU 90 (17.4%) 668 (14.7%) 0.09 Neonatal ICU 169 (32.8%) 649 (14.2%) <0.01 General pediatrics 62 (12%) 1226 (26.9%) <0.01 Pediatric Surgery 39 (7.6%) 479 (10.5%) 0.04 Hematology/Oncology 78 (15.1%) 615 (13.5%) 0.37 Nursery 21 (4.1%) 228 (5%) 0.35 Laboratory (first values) Serum creatinine, mg/dl 0.6 (0.4,0.8) 0.5 (0.3,0.7) <0.01 eGFR, ml/min per 1.73 m2 73.6 (28.4,114.4) 101.8 (62.9,133.2) <0.01 BUN, mg/dl 14 (10,21) 11 (8,16) <0.01 Glucose, mg/dl 100 (76,129) 100 (83,128) 0.39 Bicarbonate, mmol/L 20.9 (18,23.1) 21 (18.8,23.6) <0.01 Calcium, mg/dl 8.9 (7.9,9.5) 9.3 (8.7,9.8) <0.01 Platelet count, x1000/µL 227 (156,313) 257 (186,344) <0.01 Lymphocyte percent, % 21 (8,36) 21.8 (9,37.4) 0.89 INR measured (%) 311 (60.3%) 1345 (29.5%) <0.01 Fibrinogen measured (%) 191 (37%) 306 (6.7%) <0.01 Lactate measured (%) 302 (58.5%) 1645 (36.1%) <0.01 Medication,a n (%) Loop diuretic 289 (56%) 633 (13.9%) <0.01 Vasopressor 198 (38.4%) 331 (7.3%) <0.01 Chemotherapy 72 (14%) 455 (10%) 0.01 Sodium bicarbonate 131 (25.4%) 222 (4.9%) <0.01 Calcium gluconate 106 (20.5%) 110 (2.4%) <0.01 Alprostadil 16 (3.1%) 27 (0.6%) <0.01 Carboplatin 5 (1%) 8 (0.2%) <0.01 Foscarnet 4 (0.8%) 1 (0%) <0.01 Paclitaxel 1 (0.2%) 1 (0%) <0.01 Nimodipine 1 (0.2%) 0 (0%) <0.01 Procedures, n (%) Mechanical ventilation 150 (29.1%) 325 (7.1%) <0.01 RBC Transfusion 305 (59.1%) 845 (18.5%) <0.01 Secondary outcomes Inpatient mortality, n (%) 57 (11%) 41 (0.9%) <0.01 Length of stay, days (range) 17.1 (6.4,52.3) 4.2 (2.1,9.3) <0.01 Data are presented as median (IQR) or proportion. a If a medication was ever given during hospitalization for non-AKI encounters or prior to AKI for AKI encounters. INR, international normalized ratio; RBC, red blood cells.

Supplemental Table 3. Regression-based prediction models’ coefficients and odds ratios.

Standard Odds Standard Z Coefficient 95% Confidence Interval 95% Confidence Interval P>|z| Error Ratio Error score Clinical model Chemotherapy use 1.255421 0.172388 0.917546 1.593296 3.509316 0.604966 2.50314 4.91994 7.28 <0.001 BUN 0.033048 0.005163 0.022929 0.043168 1.0336 0.005337 1.023194 1.044113 6.40 <0.001 Calcium -0.32967 0.059793 -0.44686 -0.21248 0.71916 0.043001 0.639632 0.808578 -5.51 <0.001 Lactatea 0.60651 0.115204 0.380715 0.832305 1.83402 0.211286 1.463331 2.298612 5.26 <0.001 ICU admission 0.595966 0.12527 0.350442 0.84149 1.814783 0.227337 1.419695 2.319821 4.76 <0.001 Creatinine 0.829801 0.197916 0.441892 1.217709 2.292861 0.453794 1.555648 3.379436 4.19 <0.001 Pressor use 0.516302 0.152033 0.218324 0.814281 1.67582 0.254779 1.24399 2.257551 3.40 0.001 Weight 0.008 0.002386 0.003323 0.012677 1.008032 0.002405 1.003329 1.012757 3.35 0.001 Bicarbonate -0.00935 0.013795 -0.03638 0.01769 0.990697 0.013666 0.964271 1.017848 -0.68 0.498 Mechanical ventilation 0.073266 0.157288 -0.23501 0.381545 1.076017 0.169245 0.790561 1.464546 0.47 0.641 Intercept -2.20092 0.693849 -3.56083 -0.841 0.110702 0.07681 0.028415 0.431281 -3.17 0.002 Lasso model Glucose 0.003391 0.000558 0.002298 0.004484 1.003397 0.00056 1.002301 1.004494 6.08 <0.001 Time since admission -0.35862 0.059802 -0.47583 -0.24141 0.698642 0.04178 0.621371 0.785522 -6.00 <0.001 BUN 0.030068 0.005139 0.019996 0.04014 1.030525 0.005296 1.020198 1.040957 5.85 <0.001 INRa 0.59734 0.108152 0.385366 0.809313 1.817278 0.196542 1.470153 2.246364 5.52 <0.001 Δ creatinine 48 hours 3.133266 0.588987 1.978872 4.287659 22.9488 13.51655 7.234581 72.79585 5.32 <0.001 Sodium bicarbonate use 0.763901 0.181498 0.408171 1.11963 2.146633 0.38961 1.504064 3.063722 4.21 <0.001 Alprostadil use 1.374857 0.39072 0.60906 2.140653 3.954509 1.545104 1.838702 8.504989 3.52 <0.001 Calcium -0.17351 0.059226 -0.2896 -0.05743 0.840705 0.049792 0.748566 0.944185 -2.93 0.003 Creatinine 0.462358 0.18594 0.097923 0.826793 1.587814 0.295238 1.102878 2.285977 2.49 0.013 Calcium gluconate use 0.43878 0.243384 -0.03824 0.915804 1.550814 0.377443 0.962478 2.498783 1.80 0.071 Intercept -2.63584 0.555438 -3.72448 -1.5472 0.071659 0.039802 0.024126 0.212843 -4.75 <0.001 Top AUC model BUN 0.031084 0.004733 0.021808 0.04036 1.031572 0.004882 1.022047 1.041186 6.57 <0.001 Glucose 0.003321 0.000555 0.002234 0.004408 1.003327 0.000557 1.002236 1.004418 5.99 <0.001 Oxygen saturation -0.0391 0.007343 -0.05349 -0.02471 0.961653 0.007061 0.947912 0.975592 -5.33 <0.001 Δ creatinine 48 hours 3.180922 0.599963 2.005016 4.356827 24.06893 14.44046 7.426213 78.00924 5.30 <0.001 Time since admission -0.24528 0.06027 -0.3634 -0.12715 0.782489 0.047161 0.695307 0.880603 -4.07 <0.001 Respiratory rate -0.00995 0.003029 -0.01588 -0.00401 0.990104 0.002999 0.984244 0.995999 -3.28 0.001 Calcium -0.1994 0.062739 -0.32236 -0.07643 0.819224 0.051397 0.724434 0.926416 -3.18 0.001 Platelet count -0.00125 0.000437 -0.00211 -0.0004 0.998747 0.000436 0.997893 0.999602 -2.87 0.004 Lymphocyte percent -0.00592 0.003172 -0.01213 0.000301 0.994101 0.003154 0.987939 1.000301 -1.87 0.062 Creatinine 0.287709 0.189704 -0.0841 0.659522 1.333369 0.252946 0.919335 1.933868 1.52 0.129 Intercept 2.497134 0.965801 0.6042 4.390068 12.14763 11.73219 1.829787 80.64593 2.59 0.010 Forward model Nimodipine use 3.837853 0.121538 3.599644 4.076063 46.42571 5.642484 36.58519 58.91309 31.58 <0.001 Time since admission -0.7377 0.05763 -0.85065 -0.62475 0.478213 0.02756 0.427136 0.535397 -12.80 <0.001 Foscarnet use 3.239226 0.355951 2.541576 3.936876 25.51396 9.081707 12.69966 51.25822 9.10 <0.001 BUN 0.036629 0.004052 0.028687 0.04457 1.037308 0.004203 1.029102 1.045579 9.04 <0.001 Carboplatin use 2.734362 0.369992 2.009191 3.459533 15.39992 5.697848 7.457285 31.80213 7.39 <0.001 RBC Transfusion 0.640064 0.121263 0.402393 0.877735 1.896602 0.229987 1.495399 2.405444 5.28 <0.001 Loop diuretic use 0.674762 0.137812 0.404656 0.944869 1.963566 0.270603 1.498786 2.572477 4.90 <0.001 Sodium bicarbonate use 0.857524 0.180742 0.503277 1.211772 2.357317 0.426066 1.654132 3.359432 4.74 <0.001 Fibrinogena 0.614002 0.14651 0.326848 0.901157 1.847812 0.270723 1.386591 2.46245 4.19 <0.001 Paclitaxel use 6.129291 1.647324 2.900596 9.357986 459.1106 756.3037 18.18498 11591.02 3.72 <0.001 Intercept -3.35249 0.119794 -3.58728 -3.1177 0.034997 0.004192 0.027674 0.044259 -27.99 <0.001 Genetic model Time since admission -0.64517 0.060229 -0.76322 -0.52713 0.524572 0.031594 0.466163 0.590299 -10.71 <0.001 General Pediatrics admission -1.29355 0.198857 -1.68331 -0.9038 0.274294 0.054545 0.185759 0.405027 -6.50 <0.001 Loop diuretic use 0.810097 0.140097 0.535511 1.084683 2.248126 0.314957 1.708321 2.958501 5.78 <0.001 BUN 0.029493 0.005114 0.01947 0.039516 1.029932 0.005267 1.019661 1.040307 5.77 <0.001 RBC Transfusion 0.680983 0.118269 0.449181 0.912785 1.975819 0.233677 1.567028 2.491251 5.76 <0.001 Glucose 0.003262 0.000579 0.002128 0.004396 1.003267 0.000581 1.00213 1.004406 5.64 <0.001 Sodium bicarbonate use 0.857802 0.172809 0.519103 1.196502 2.357973 0.407479 1.68052 3.308522 4.96 <0.001 Chemotherapy use 0.821921 0.175061 0.478808 1.165035 2.274867 0.39824 1.614149 3.206034 4.70 <0.001 Creatinine 0.752208 0.19168 0.376522 1.127894 2.121679 0.406683 1.457207 3.089142 3.92 <0.001 Alprostadil use 1.113824 0.414417 0.301581 1.926067 3.045983 1.262308 1.351994 6.862465 2.69 0.007 Intercept -4.05392 0.182843 -4.41229 -3.69556 0.017354 0.003173 0.012127 0.024834 -22.17 <0.001 a Transformed into categorical variables (measured vs not measured) due to missingness > 25%. BUN, blood urea nitrogen; ICU, intensive care unit; INR, international normalized ratio; RBC, red blood cells; Spo2, oxygen saturation; Δ creatinine 48 hours, change in serum creatinine within the last 48 hours.

Supplemental Table 4: Genetic algorithm model operating points applied to the internal validation cohort. The choice of threshold points should be decided based on the risk of the potential intervention and individual health systems may have to make their own choices. A suggested low-risk cut-off threshold point of 0.08 shows a high negative predictive value of 97.9%, includes a large number of patients but a small proportion were falsely negative and therefore it can aid in low-risk, screening interventions. A suggested high-risk cut-off threshold point of 0.24 is a highly specific operating point (specificity of 99.6%) at which the ratio of true positive to false positive alerts is 1:2.8, therefore it can be used to identify patients at imminent AKI to implement more intensive interventions. Given the low baseline rate of AKI episodes (2.8%), it would be hard to make a test with a high PPV, but a PPV of 26.6% represents a 12-fold increase over baseline (LR+ of 12.5).

Median (IQR) time Cut True False True False TP:FPa Sensitivity Specificity PPV NPV LR+ LR- OR from alert to AKI, point positivea positivea negativea negativea hours

0b 14,099 486,003 1:34.5 0 0 100.0% 0.0% 2.8% - 1.0 - - 159.1 (52.3-384.1)

0.04 7,593 99,111 1:13.1 386,892 6,506 53.9% 79.6% 7.1% 98.3% 2.6 0.6 4.6 136.6 (42.7-299.7)

0.08 4,055 28,526 1:7.0 457,477 10,044 28.8% 94.1% 12.4% 97.9% 4.9 0.8 6.5 142.7 (32.7-300.3)

0.12 2,396 11,993 1:5.0 474,010 11,703 17.0% 97.5% 16.7% 97.6% 6.9 0.9 8.1 124.9 (24.7-284.5)

0.16 1,420 5,620 1:4.0 480,383 12,679 10.1% 98.8% 20.2% 97.4% 8.7 0.9 9.6 113.7 (13.3-252.9)

0.20 953 3,274 1:3.4 482,729 13,146 6.8% 99.3% 22.5% 97.3% 10.0 0.9 10.7 116.2 (8.8-259.7)

0.24 749 2,071 1:2.8 483,932 13,350 5.3% 99.6% 26.6% 97.3% 12.5 1.0 13.1 23.7 (5.3-343.7)

0.28 642 1,394 1:2.2 484,609 13,457 4.6% 99.7% 31.5% 97.3% 15.9 1.0 16.6 10 (4.2-328.9)

0.32 440 781 1:1.8 485,222 13,659 3.1% 99.8% 36.0% 97.3% 19.4 1.0 20.0 7.6 (3.3-24.5)

0.36 355 558 1:1.6 485,445 13,744 2.5% 99.9% 38.9% 97.2% 21.9 1.0 22.5 7.1 (2.5-19.5) a The sensitivity analysis was performed at the timepoint level. For example, there were 749 timepoints in all the 2299 encounters in the internal validation cohort that crossed the 0.24 threshold point (alert) and lead to an AKI in the next 48 hours compared to 2,071 timepoints that did not lead to an AKI in the next 48 hours. B The cut point of 0 represents a baseline rate of all timepoints of the 2299 encounters in the validation cohort. At baseline, 14,099/486,003 (2.8%) timepoints were associated with AKI events afterwards. LR+, positive likelihood ratio; LR-, negative likelihood ratio; NPV, negative predictive value; OR, odds ratio; PPV, positive predictive value; TP:FP, true positive to false positive ratio.

Supplemental Table 5: Genetic algorithm model operating points applied to the external validation cohort. The suggested low-risk threshold point of 0.08 showed a comparable performance to the internal validation cohort, but the high-risk threshold point had no true positives due to the small number of patients that crossed this threshold in this low-acuity cohort.

Median (IQR) Cut True False True False TP:FPa Sensitivity Specificity PPV NPV LR+ LR- OR time from alert point positivea positivea negativea negativea to AKI, hours

0b 1,981 278,045 1:140.4 0 0 100.0% 0.0% 0.7% - 1.0 - - 153.3(57.8-336.2)

0.04 1,109 36,049 1:32.5 241,996 872 56.0% 87.0% 3.0% 99.6% 4.3 0.5 8.5 148.6(59.6-338.9)

0.08 526 7,416 1:14.1 270,629 1,455 26.6% 97.3% 6.6% 99.5% 10.0 0.8 13.2 139.3(65.9-393.6)

0.12 337 2,240 1:6.6 275,805 1,644 17.0% 99.2% 13.1% 99.4% 21.1 0.8 25.2 87.6(26.5-171.1)

0.16 104 764 1:7.3 277,281 1,877 5.2% 99.7% 12.0% 99.3% 19.1 1.0 20.1 23.6(5.1-58.2)

0.20 4 500 1:125.0 277,545 1,977 0.2% 99.8% 0.8% 99.3% 1.1 1.0 1.1 421.1(14.6-422.8)

0.24 0 341 - 277,704 1,981 0.0% 99.9% 0.0% 99.3% 0.0 1.0 0.0 422.2(420.7-423) a The sensitivity analysis was performed at the timepoint level. For example, there were 526 timepoints in all the 1102 encounters in external validation cohort that crossed the 0.08 threshold point (alert) and lead to an AKI in the next 48 hours compared to 7,416 timepoints that did not lead to an AKI in the next 48 hours. B The cut point of 0 represents a baseline rate of all timepoints of the 1102 encounters in the external validation cohort. At baseline, 1,981/278,045 (0.7%) timepoints were associated with AKI events afterwards. LR+, positive likelihood ratio; LR-, negative likelihood ratio; NPV, negative predictive value; OR, odds ratio; PPV, positive predictive value; TP:FP, true positive to false positive ratio.

Supplemental Table 6: Low-risk cut-off threshold (0.08) analysis using the genetic algorithm model in age subgroups (on the internal validation cohort).

Cohort Sensitivity Specificity PPV NPV

Entire validation cohort 28.8% 94.1% 12.4% 97.9% n=2299 (AKI=207, 9%) Newborns (birth–27 days) 32.8% 76.4% 24.2% 83.2% n=492 (AKI=70, 14.2%) Infants (28 days–12 months) 27.0% 90.5% 49.5% 78.3% n=256 (AKI=16, 6.2%) Children (1-12 years) 30.7% 89.8% 58.7% 73.2% n=913 (AKI=68, 7.4%) Adolescents (12-18 years) 21.5% 67.1% 25.3% 62.2% n=632 (AKI=53, 8.3%) NPV, negative predictive value; PPV, positive predictive value

Supplemental Table 7: High-risk cut-off threshold (0.24) analysis using the genetic algorithm model in age subgroups (on the internal validation cohort).

Cohort Sensitivity Specificity PPV NPV

Entire validation cohort 5.3% 99.6% 26.6% 97.3% n=2299 (AKI=207, 9%) Newborns (birth–27 days) 3.8% 99.1% 48.3% 81.7% n=492 (AKI=70, 14.2%) Infants (28 days–12 months) 0.2% 99.2% 6.7% 74.3% n=256 (AKI=16, 6.2%) Children (1-12 years) 9.8% 100.0% 100.0% 70.0% n=913 (AKI=68, 7.4%) Adolescents (12-18 years) 5.9% 95.4% 40.1% 66.1% n=632 (AKI=53, 8.3%) NPV, negative predictive value; PPV, positive predictive value

Number of variables

Supplemental Figure 1: Logistic regression analyses of incremental numbers of variables of the stepwise forward selection model. Model performance increases as additional variables are added, but there are diminishing returns after the first 10 variables.

Supplemental Figure 2. Genetic algorithm feature selection process.

Generation

Supplemental Figure 3: Convergence in evaluation set (of derivation cohort) area under the receiver operating characteristic curve (AUC) of the genetic algorithm’s population.

Derivation cohort Internal validation cohort External validation cohort

Supplemental Figure 4: Genetic algorithm model calibration curves. The prediction model is well calibrated in both derivation and internal validation cohorts. For prospective and external implementation, it would be appropriate to recalibrate the model on the local cohort.

Derivation cohort Internal validation cohort External validation cohort

PR-AUC 0.12 PR-AUC 0.11 PR-AUC 0.06

Supplemental Figure 5: Genetic algorithm model performance illustrated by a precision-recall curve. PR-AUC, the area under the precision–recall curve.

Supplemental Figure 6. Genetic model for the prediction of severe AKI and subgroup analyses in the internal validation cohort. The AUC is 0.79 (95% CI 0.74-0.83) for predicting severe AKI (stage 2 or 3) and 0.80 (95% 0.72-0.88) for stage 3 AKI. In newborns, the AUC is 0.81 (95% CI 0.77-0.86).

Supplemental Methods. Genetic Algorithm Methodology: Genetic algorithms model a population of genotypes and its temporal evolution, discretized as a sequence of generations. Several key components must be defined: a genotype representation, a fitness function and mating rules, and genetic recombination and mutation rules. These key components are used to produce the next generation in the sequence from the current one. A general implementation of genetic algorithms for feature selection is described below. • Genotype Representation: A genotype is defined as a subset of 푘 features chosen from the full feature set. • Fitness Function: The derivation set is randomly split into training and evaluation sets with proportions 훼 and 1 − 훼, respectively. A model 푀 is fit to the training set restricted to only the features in the genotype representation. The fitness function is defined as a performance metric 푃 of the model when applied to the evaluation set. This procedure occurs for every genotype. • Mating Rules: The fitness values over the population are used to generate a distribution 퐹 over the population describing their probability of being selected for mating. Mating pairs are then generated by randomly sampling from this distribution until the number of pairs equals the current population size 푛. • Genetic Recombination: For each mating pair, overlapping features of the parent genotypes are first placed into the child genotype. The remaining non-overlapping features are uniformly sampled without replacement and given to the child until the child has 푘 features. • Genetic Mutation: After recombination, each child has their 푘 features iterated through and randomly replaced by a new feature from the full feature set according to a probability set by the mutation rate 푟. For our particular implementation, the hyperparameters chosen were 훼 = 2/3, 푛 = 1000, 푘 = 10, 푟 = 0.05, 푀 is logistic regression, 푃 is the area under the receiving operator characteristic curve (AUC), and 퐹 is a linearly decreasing probability mass function (PMF) defined over the indices of the population reordered by fitness such that the worst performing genotype has probability 0. Furthermore, we incorporated elitism, a technique commonly used with genetic algorithms, which improves the rate of convergence. Elitism chooses the 푡 best genotypes for a generation and produces a clone of them in the next generation; we set 푡 = 3. We initialize the algorithm with uniformly randomly selected genotypes. The algorithm runs until a stopping criteria is met: a lack of improvement in the median evaluation AUC in the population over a 20 generation window. Once the stopping criteria is met, 5-fold cross validation is used to estimate a mean AUC for every genotype in the population. The algorithm returns the genotype that maximizes this statistic.