05 June 2020 No.09

An overview of predictive scoring systems used in ICU

UV Jaganath

Moderator: S Moodley

School of Clinical Discipline of Anaesthesiology and Critical Care

Page 1 of 20

CONTENTS Introduction ...... 3 Developing a predictive scoring system ...... 4 Choosing a predictive scoring system ...... 6 Acute Physiologic and Chronic Health Evaluation (APACHE) ...... 7 Sequential (-related) Organ Failure Score (SOFA) ...... 10 Simplified Acute Physiologic Score (SAPS) ...... 11 Mortality Predictive Model (MPM) ...... 12 Comparison of performance ...... 14 Use in the low to middle-income setting ...... 15 Other uses of predictive scoring systems ...... 16 Limitations ...... 17 Conclusion and recommendations ...... 18 References ...... 19

Page 2 of 20

Introduction With the rising healthcare costs and shortage of (ICU) beds, clinicians need to appropriately triage patient admissions into ICU to avoid wasteful expenditure and unnecessary bed utilisation. While the Critical Care Society of Southern Africa does recognise the use of predictive scoring systems in ICU to aid in triage, they have also acknowledged some if its pitfalls.1 Predictive scoring systems are tools that have been developed to describe the severity of a disease process and subsequently predict outcomes in patients. These tools typically utilise a combination of patient data, including clinical health information, physiological and laboratory data to determine a numerical severity of disease score, which in turn is used to determine outcomes, such as; length of hospital stay and mortality rates.2-4 Predictive scoring systems can be divided into two broad categories5: 1. Single organ or disease specific scoring systems such as: • European Systems for Cardiac Operative Risk Evaluation (EuroSCORE) that is used to predict mortality after cardiac .6 • The Model for End-Stage Liver Disease (MELD) score used in patients with end-stage liver disease to predict mortality risk.7 • The Glasgow Score (GCS) to assess and predict mortality after head injury.8 2. Generic scoring systems for the use in all ICU patients, such as: • Acute Physiologic and Chronic Health Evaluation (APACHE). • Sequential (Sepsis-related) Organ Failure Score (SOFA). • Simplified Acute Physiologic Score (SAPS). • Mortality Predictive Model (MPM). • Multiple Score (MODS). The ideal predictive scoring system should use objective and easily measured risk factors or predictors upon which the outcome prediction is made. The characteristics of the ideal predictive scoring system includes:5,9 1. Uses easily measurable variables (see below). 2. Has a high level of discrimination. 3. Well calibrated. 4. Validated for use in all patients in the ICU and in different countries or patient cohorts. 5. Can predict length of hospital stay, quality of life after ICU and/or mortality.

Page 3 of 20

The idea variables for describing organ dysfunction should be:5,10 1. Simple, cheap and routinely available. 2. Reliable, reproducible and objective to measure. 3. Specific to the acute dysfunction of the organ being evaluated but not chronic dysfunction. 4. It should not be affected by reversable, transient abnormalities that are associated with practical or therapeutic interventions. 5. Abnormalities should only be in one direction. 6. Variable should be continuous rather than dichotomous. It should be noted that the simultaneous use of more than one predictive scoring system on the same patient should be seen as complementary, as opposed to competitive or mutually exclusive, as their combined use may possibly offer a more accurate indication of the true severity of the disease process and hence overall prognosis.5

Developing a predictive scoring system To appreciate the basis of predictive scoring systems, it is important to have an understanding of the steps that are required to develop them. An overview of these steps are as follows2,4,10,11: 1. Selecting the outcome variable(s) • Mortality is most commonly used as it is an easy to define endpoint. Being binary, it easily lends itself to statistical analysis. • Long term prediction of quality of life after ICU and cost-effective medical treatment may also be used, however, unlike mortality, they cannot be represented as a binary endpoint, thus making them more difficult to measure.

2. Selecting the patient population • Population bias may be inherent. This is an important factor to consider when one is developing or choosing to use a given scoring system. This may be important in the South African setting whereby our population demographics and disease profiles may not mirror that of the population sample that the scoring system was originally developed for and validated against. • Ideally, the clinician should only use the predictive scoring system for decision making if the said scoring system was developed for and validated in a population group similar to that of the patient of interest. • Should the accuracy of the scoring system be found to be inaccurate in the population group being examined, it should then be updated or modified for that population group. An example of such an undertaking occurs with the APACHE scoring system.

Page 4 of 20

3. Selecting the variables or risk factors • Selecting variables to be studied are usually done in one of two ways. • In the first method, clinicians select variables that are believed to be related to the chosen outcome (e.g. mortality). Such a technique was employed to develop the APACHE II and SOFA scoring systems. • In the second method, statistical methods (e.g. linear discriminant function analysis) are used to narrow down the initial list of variables to only those that are individually related to the chosen outcome. Such a technique was employed to develop the APACHE III, SAPS and MPM II scoring system. • It is important that the variables identified are objective and easy to obtain.

4. Data collection (predictors) and analysis (of the outcome) • Prospective data collection is preferred as it allows for ongoing analysis of the data being collected. This helps reduce the risk of bias or errors which may reduce the accuracy of the predictive scoring system. • Another advantage of prospective data collection is that it allows for evaluation of the timing of the measured variable to the outcome. While the SAPS and MPM scoring systems use data collected within 1 hour of admission, the APACHE and SOFA scoring systems used the worst physiological values within 24 hours of admission. • Missing data from retrospective investigations may become challenging and result in inappropriate associations of mortality with a smaller number of organ system derangements, which may be reduced with prospective data collection methods. • Once the predictive factors are identified, they are analysed and presented either as a cumulative score or as a single score which is calculated from the sum of weights allocated to each individual predictive factor (as used in APACHE and SAPS).

5. Developing the model • This is done with the use of various statistical techniques including multivariant linear regression models (used to assess relationships between independent variables, i.e. the risk factors and the dependent outcome i.e. mortality) and logistic transformation (used when the predicting factors are not linearly related to the outcome). • Logistic regression methods may be used to help identify additional predictive factors that may impact the final outcome.

Page 5 of 20

6. Validation of the predictive scoring system • This can be done by directly comparing the predictive scoring system to a “gold standard” or to each other. • Another technique involves developing the receiver operation characteristic curve to aid with a more precise cut off point selection.

7. Evaluation of the predictive scoring system’s utility and impact • Depending on the chosen scoring system, they may be used to triage patients for ICU and assist in decision making. They may also be used during clinical trials to prove that the two patient population groups were similar.

8. Updating the system • All predictive scoring systems should be updated as its calibration over time may deteriorate because of case-mix and changes in both disease profiles and therapeutic interventions. This tends to result in an overestimation of mortality and subsequently poor discrimination between outcomes (e.g. survivors vs non-survivors). • This is important since past studies have shown that in certain circumstances older scoring systems performed no better, or perhaps worse than predictions made by clinicians.

Choosing a predictive scoring system While there are many different predictive scoring systems to choose for, each with their own advantages and limitations, some of the commonly used models include the APACHE, SOFA, SAPS and MPM scoring systems. When choosing to use a predictive model, is it important to select one that was most recently developed for and validated in the population group being evaluated. Other factors that should be taken into account include its user-friendliness, accessibility, feasibility, cost implications and the outcome that is being evaluated (e.g. length of hospital stay vs predicted mortality rate).2 An example of such limitations is that while the APACHE IV model is able to more accurately predict mortality over the SAPS model, this benefit is off set by the fact that it is more challenging and costly to use as it requires more variables and depends on propriety software. Additional, APACHE IV is better calibrated for predicting length of hospital stay when used in ICUs within the United States. In contrast, while the SAPS model is cheaper and easier to use and better suited for use internationally, it is more prone to case-mix effects and is unable to reliably predict length of stay when compared to the APACHE IV model.2 A summary of each of these predictive scoring systems will be discussed separately.

Page 6 of 20

Acute Physiologic and Chronic Health Evaluation (APACHE) The APACHE scoring system has been found to be efficient in predicting mortality and estimating length of stay in ICU. There are four versions that is widely used, APACHE I to IV. The original APACHE model was developed in 1981 and consisted of two parts. The former is the Acute Physiological Score (APS) which represents the degree of the acute illness. A total of 34 physiological variables are measured and allocated a score between 0 and 4, depending on the degree of severity. The worst value of each variable measured within the first 32 hours of ICU admission are used. The latter part of the score, the Chronic Health Evaluation (CHE) that is classified from A to D, represents the physiological status of the patient before the illness. The classification of A representing excellent health while a classification of D representing severely failing health. The initial study demonstrated a direct relationship between the APS score and the likelihood of mortality. With regards to the CHE, only class D was shown to be an independent risk factor for mortality.12 Developed in 1985 as a modification of the original model, the APACHE II scoring system uses a point score based on 12 physiological parameters that is routinely investigated.13 These physiological measurements are done during the first 24 hours after admission, in addition to this, the patients age, medical history and surgical requirements are considered. If a variable has not been measured a point of zero is allocated. A score between 0 and 71 is then calculated based on these measurements. Generally, higher scores are more predictable of severe disease and subsequently higher rates of mortality. From the APACHE II score, the estimated risk of in hospital mortality is then calculated using a logistic regression equation, utilizing specific beta co-efficient made for this purpose (Tables 1 and 2).4,14,15 The APACHE III was developed in 1991 using 26 variables. It comprises of two components, namely the APACHE III score, ranging from 0 – 299, and the APACH III predictive equation that uses the APACHE III score to predict in hospital mortality rates.16 With regards to predicting ICU mortality in trauma patients, the APACHE III has been validated against and found be as accurate as the Trauma Injury Severity Score (TRISS).17 Developed in 2006, the APACHE IV is more complex and entails the input of 142 variables and 115 various disease groups, however web based calculations can be done.18 Despite the APACHE IV being the most recent version, the APACHE II score is still amongst the most commonly model in current clinical use. When comparing the APACHE II to the APACHE IV model, both demonstrated good discrimination capabilities, with an area under the receiver operating characteristics curve of 0.805 and 0.832 respectively. With regards to predicting mortality, the cut-off points with the best Youden index were 17 and 72 of the APACHE II and APACHE IV respectively.13

Page 7 of 20

Table 1. Acute physiologic and chronic health evaluation (APACHE II)4,14

Page 8 of 20

Table 2. Acute physiologic and chronic health evaluation II-diagnostic category weight4,14

Advantages: The measurements required to calculate the APACHE II score are routine parameters that are monitored in the ICU thus no additional investigations are needed. This predictive scoring system can predict sepsis with a single assessment at 24 hours. Additionally, it has been validated in several countries and has been proven to be highly reproducible.4,14,15 Disadvantages: The major setback of the APACHE scoring system is that patients may have several co-morbid illnesses and therefore selecting only one main diagnostic category may be challenging. The physiological parameters used in the calculation of the APACHE score are dynamic and may be easily affected by factors such as on-going treatment and .4,14 Regarding the APACHE IV scoring system, even though more updated, it is more complex and requires physiological setting software resulting in added costs.18

Page 9 of 20

Sequential (Sepsis-related) Organ Failure Score (SOFA) The SOFA score was developed in 1994 by the European Society of , then revised in 1996. It was originally used to understand the natural progression of individual organ failure and the interaction between failure of other organs and to describe the sequence of complications (in terms of said organ dysfunction or failure) in critically ill patients with sepsis.4,19 Intuitively, while any calculation of morbidity or organ dysfunction should then relate to mortality to some degree, the SOFA score was originally not intended to predict this outcome. However, it has since been validated for use in critically ill patients with non-sepsis related organ dysfunction and as a tool for predicting mortality rate.4,10,19 It measures 6 organ systems with scores ranging between 0 – 4 for each (Table 5). Interpretation: Irrespective of the initial SOFA score, an increase in the score within 48 hours of ICU admission is associated with a mortality rate of ≥ 50%, an unchanged score was associated with a mortality rate of between 27 – 35% (if the initial score was < 8) and 60% (if the initial score was ≥ 8), whereas a decreasing score is associated with a mortality rates of < 6% and 27% if the initial score was < 8 or ≥ 8 respectively.20

In the setting of sepsis, a score of ≥ 2 is associated with a predicted mortality rate of ≥ 10%, however, in the setting of septic with a score of ≥ 2 the predicted mortality rate is around 40%.2,21 Further information regarding the interpretation of the of SOFA score may be found in table 5. Advantages: Unlike the APACHE and SAPS scores, which are models of physiological assessment, the SOFA score may be considered to be a multi-organ scoring system, and as such it can be used to track the response of organ dysfunction to therapy over time.9 It is also easier and cheaper to use than models such as the APACHE IV. Disadvantages: In contrast to the above-mentioned predictive scoring systems, the SOFA score does not predict individual outcomes, instead it helps identify high risk patients at risk of sepsis related death as a group.2 It does not take chronic health status into account. Table 5. Sequential organ failure assessment score (SOFA)4

Page 10 of 20

Simplified Acute Physiologic Score (SAPS) The SAPS II scoring system uses 17 variables i.e. 12 physiological variables which are measured within the first 24 hours after admission, age of the patient, the type of admission and three disease-related variables.22 Several of these variables are assigned a score depending whether they are present or not, whilst the 12 physiological variables are scored according to a range of values. The SAPS II score may vary between 0 –163 points. The probability of mortality is then calculated using a logistic regression analysis.4,14 Advantages: The SAPS III which is the latest version has a greater potential for universal use as it has been designed and validated in ICU settings amongst thirty- five countries. Disadvantages: The SAPS II was designed from a database of patients in North America and Europe. Unfortunately this sample is not representative of the population and ICUs in other countries where variability in structures and resources are related to the outcome.23 Unlike APACHE IV, the SAP scoring system cannot be used to predict length of stay, however it may be used to compare the use of resources amongst ICUs.24

Table 3. Simplified acute physiology score (SAPS II)4,14

Page 11 of 20

Mortality Predictive Model (MPM) The Mortality Probability Model was developed on an international sample of patients between 1989 – 1990. It assesses patients probability of mortality at hospital discharge, based on measurements attained within the first hour of ICU admission.25 There are three models. The first version of the model was developed to predict mortality based on data from admission and after the first 24 hours in the ICU. Later, models were developed to include data from 48 – 72 hours after ICU admission. This model uses the patients’ chronic illnesses, acute diagnosis, some physiological variables and other variables such as .4,14 Advantages: The MPM scoring system uses less physiological data in comparison to other scoring systems thus this scoring system may be preferred in settings where laboratory resources are constrained. The MPM III has good discrimination and calibration.25 Disadvantages: The MPM excludes certain patient subsets such as cardiac surgery, myocardial infarction, and ICU readmissions, which decreases its usefulness to some ICUs.25

Page 12 of 20

Table 4. The Mortality Predictive Model (MPM)4,14

Page 13 of 20

Comparison of performance Despite having a paucity of high-quality studies comparing the performance of the various models to each, after evaluating the current evidence, the following generalisations can be made: 1. When comparing the older models (APACHE II, APACHE III, SAPS II, and SOFA) to each other, the APACHE II and III predictive scoring system were shown to be superior in one study whereas the SAPS II was shown to be superior in another.26 The accuracy of these model in predicting the actual mortality rate was also limited. As such, it has been suggested that using a combination of these scoring systems may result in an improvement in the overall predictive performance.26

2. When comparing the newer models (APACHE IV, SAPS III and MPM III) to the older models (APACHE II and III, SAPS II, MPM II and SOFA), the newer models were found to perform better. This was evident by the fact that the older models had a tendency to over-predict the mortality rate, whereas the newer models had superior calibration, accuracy and discrimination characteristics. However, despite these findings, these differences were not found to be statistically significant. A possible reason for this finding is that the choice to use one predictive scoring system over another may be due to ease of use and local preferences.27

3. In general, all of these models have very good discriminatory values with an area under the receiver operating characteristics curve between 0.80 – 0.90 while simultaneously demonstrating good calibration assessments.9 As such, no single model has been shown to be significantly superior to another with regards to predicting mortality.2,27

Page 14 of 20

Use in the low to middle-income setting With regards to the South African context, a major cause of concern is that the above- mentioned predictive scoring systems have been developed and validated predominantly in high-income countries. As a result, the accuracy of these models may be reduced given that our population and disease profile and therapeutic intervention may not mirror that seen in high-income countries. While not specific to South Africa, the performance these models to predict mortality in critically ill patients in low or middle-income countries was demonstrated to be moderate at best.19,28 One reason for this relates to the lack of calibration of these models to the population cohort. Ideally, if a predictive scoring system is to be used, it should exhibit good calibration for that population cohort. However, it has been suggested that should a model with a poor calibration demonstrate a good discriminatory ability, it may still be of benefit if it is used it to identify high-risk patients for diagnostic and/or therapeutic interventions.19,29 Another given reason is due to and the possibility of missing predictor variables due to resource constraints. This problem may be overcome by developing scoring systems with fewer and more commonly available variables (e.g. the Rwanda MPM and TropICS models) or by allowing for the substation of certain inaccessible variables with its more commonly available counterparts (e.g. substituting PaO2 for oxygen saturation).19,28 A major advantage of developing setting-specific models is that they address the above two problems. One such example is the Rwanda MPM (R-MPM), which was developed using five easily attainable variables, namely: age, heart rate and GCS at the time of ICU admission, or shock as a reason for ICU admission and suspected or confirmed infection within 24 hours of ICU admission. Despite having fewer variables, the R-MPM demonstrated better calibration, discrimination and prediction compared to the MPM when used in the same population of Rwandan ICU patients.28

Page 15 of 20

Other uses of predictive scoring systems Apart from the clinical uses describe able, predictive scoring systems do play a role in other aspects of medicine. Two key areas include its role during research and as a quality care bench mark tool. 1. Research • Scoring systems may be used in clinical trials to compare the baseline risks between comparative groups to ensure that they are similar. This is commonly used during clinical trials in patients with acute respiratory distress syndrome or sepsis whereby possible therapeutic interventions are being evaluated.2,11 2. Quality care bench mark • Predictive scoring systems help evaluate the quality of care by confirming that patients with the same or similar baseline mortality risks are being compared. An example of such practices are studies that compare ICU outcomes with other ICUs within the same hospital or in other hospitals. The implications of such findings are that policies and practices from ICUs with favourable mortality rates may then be adopted and incorporated by other units to help improve their quality or care.30,31

Page 16 of 20

Limitations The ICU is the perfect environment for using predictive scoring systems since both the population group and patient care tends to be is well defined and the most significant predictor of mortality is the severity of the illness. However, there are some limitations with regards to their use as follows:32,33 1. The scoring system may not be validated in the population group that it is being used to evaluate. Similarly, they may not be able to accurately predict outcomes for specific patient population groups (e.g. the use of APACHE II was found to be unreliable to predict outcomes in pregnant women receiving critical care) or disease processes.34 The predictiveness of the system may be reduced due to the unusual nature of a given population for which it was not designed or validated in.33

2. The predictiveness of the scoring system deteriorates over time and as such, failure to periodically update the system results in a gradual loss of discrimination and/or calibration. The net effect it that an overestimation of the predicted mortality rate may be seen.35

3. A phenomenon known as lead-time bias may occur. This was seen when patient who were transferred in from other ICUs or hospitals had a higher mortality rate than that predicted by the APACHE II scoring system. This variable, the location of treatment, was subsequently added to the APACHE scoring system but not the others mentioned above.36

4. The quality of care is better or worse than expected resulting in a lower or higher patient mortality rate. In this scenario, the model will lose its predictive accuracy as they will be a higher or lower actual survival rate due to the better or worse quality of care respectively.33

5. Unlike SAPS, MPM and SOFA, models like the APACHE require proprietary software and more data points to use, resulting in it being more burdensome, however, the integration of electronic record keeping into health systems may alleviate some of these challenges.2

6. When predicting mortality within 24 hours of admission into the ICU, the current evidence suggests that scoring systems are not yet superior to clinical judgment.37

Page 17 of 20

Conclusion and recommendations Due to the limited availability of ICU resources in South Africa it is important that we utilise multiple tools to aid in its rational use. After reviewing the literature, predictive scoring systems do have a role to play in this. The accuracy of predictive scoring systems will continue to improve with time. This is possible due to and improvements in computing power and the integration of “big data” into our heath care systems. When deciding to use predictive scoring systems, it is important for clinicians to be aware that the predicted mortality rate relates to patients within a similar cohort and not to an individual. As a result, while we may be able to predict the mortality of a group of patients with a similar score, we are unable to predict which individual patient may survive and who may die. Consequently, caution should be used when predicting mortality for individual cases. Conversely, as scoring systems allow for an objective assessment of the clinical status of the patient, they may be used to assist the clinical decision-making since they mirror the probability of mortality in a similar cohort of patients. Ultimately, predictive scoring systems should be considered as a tool to assist, rather than replace the clinician.

Page 18 of 20

References

1. Joynt GM, Gopalan DP, Argent AA, et al. The Critical Care Society of Southern Africa Consensus Guideline on ICU Triage and Rationing (ConICTri); 2019. 2. Kelley MA, Manaker S, Finlay G. Predictive scoring systems in the intensive care unit. UpToDate 2020. 3. Saranya Avr SJ. Comparison of Different Scoring Systems Used in the Intensive Care Unit. Journal of Pulmonary & Respiratory Medicine 2015; 05(04). 4. Rapsang AG, Shyam DC. Scoring systems in the intensive care unit: A compendium. Indian J Crit Care Med 2014; 18(4): 220-8. 5. Vincent JL, Moreno R. Clinical review: scoring systems in the critically ill. Critical care (London, England) 2010; 14(2): 207. 6. Nashef SA, Roques F, Hammill BG, et al. Validation of European System for Cardiac Operative Risk Evaluation (EuroSCORE) in North American cardiac surgery. Eur J Cardiothorac Surg 2002; 22(1): 101-5. 7. Kamath PS, Wiesner RH, Malinchoc M, et al. A model to predict survival in patients with end- stage liver disease. Hepatology 2001; 33(2): 464-70. 8. Zuercher M, Ummenhofer W, Baltussen A, Walder B. The use of in injury assessment: a critical review. Brain Inj 2009; 23(5): 371-84. 9. Bouch DC, Thompson JP. Severity scoring systems in the critically ill. Continuing Education in Anaesthesia Critical Care & Pain 2008; 8(5): 181-5. 10. Vincent JL, Moreno R, Takala J, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive care medicine 1996; 22(7): 707-10. 11. Kollef MH, Schuster DP. Predicting intensive care unit outcome with scoring systems. Underlying concepts and principles. Crit Care Clin 1994; 10(1): 1-18. 12. Wong DT, Knaus WA. Predicting outcome in critical care: the current status of the APACHE prognostic scoring system. Can J Anaesth 1991; 38(3): 374-83. 13. Nagar VS, Sajjan B, Chatterjee R, Parab NM. The comparison of apache II and apache IV score to predict mortality in intensive care unit in a tertiary care hospital. International Journal of Research in Medical Sciences 2019; 7(5): 6. 14. Rapsang AG, Shyam DCJIjoccmp-r, official publication of Indian Society of Critical Care Medicine. Scoring systems in the intensive care unit: a compendium. 2014; 18(4): 220. 15. VijayGanapathy S, Karthikeyan VS, Sreenivas J, Mallya A, Keshavamurthy RJI, urology c. Validation of APACHE II scoring system at 24 hours after admission as a prognostic tool in urosepsis: A prospective observational study. 2017; 58(6): 453-9. 16. Knaus WA, Wagner DP, Draper EA, et al. The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest 1991; 100(6): 1619-36. 17. Darbandsar Mazandarani P, Heydari K, Hatamabadi H, Kashani P, Jamali Danesh Y. Acute Physiology and Chronic Health Evaluation (APACHE) III Score compared to Trauma-Injury Severity Score (TRISS) in Predicting Mortality of Trauma Patients. Emerg (Tehran) 2016; 4(2): 88-91. 18. Zimmerman JE, Kramer AA, McNair DS, Malila FMJCcm. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. 2006; 34(5): 1297-310. 19. Haniffa R, Isaam I, De Silva AP, Dondorp AM, De Keizer NF. Performance of critical care prognostic scoring systems in low and middle-income countries: a systematic review. Critical care (London, England) 2018; 22(1): 18. 20. Ferreira FL, Bota DP, Bross A, Melot C, Vincent JL. Serial evaluation of the SOFA score to predict outcome in critically ill patients. JAMA 2001; 286(14): 1754-8.

Page 19 of 20

21. Lambden S, Laterre PF, Levy MM, Francois B. The SOFA score-development, utility and challenges of accurate assessment in clinical trials. Critical care (London, England) 2019; 23(1): 374. 22. Le Gall J-R, Lemeshow S, Saulnier FJJ. A new simplified acute physiology score (SAPS II) based on a European/North American multicenter study. 1993; 270(24): 2957-63. 23. Moreno RP, Metnitz PG, Almeida E, et al. SAPS 3—From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission. 2005; 31(10): 1345-55. 24. Rothen HU, Stricker K, Einfalt J, et al. Variability in outcome and resource use in intensive care units. 2007; 33(8): 1329-36. 25. Higgins TL, Teres D, Copes WS, Nathanson BH, Stark M, Kramer AAJCcm. Assessing contemporary intensive care unit outcome: an updated Mortality Probability Admission Model (MPM0-III). 2007; 35(3): 827-35. 26. Saleh A, Ahmed M, Sultan I, Abdel-lateif A. Comparison of the mortality prediction of different ICU scoring systems (APACHE II and III, SAPS II, and SOFA) in a single-center ICU subpopulation with acute respiratory distress syndrome. Egyptian Journal of Chest Diseases and Tuberculosis 2015; 64(4): 843-8. 27. Juneja D, Singh O, Nasa P, Dang R. Comparison of newer scoring systems with the conventional scoring systems in general intensive care population. Minerva anestesiologica 2012; 78(2): 194- 200. 28. Riviello ED, Kiviri W, Fowler RA, et al. Predicting Mortality in Low-Income Country ICUs: The Rwanda Mortality Probability Model (R-MPM). PLoS One 2016; 11(5): e0155858. 29. Nassar AP, Malbouisson LM, Moreno R. Evaluation of Simplified Acute Physiology Score 3 performance: a systematic review of external validation studies. Critical care (London, England) 2014; 18(3): R117. 30. Afessa B, Keegan MT, Hubmayr RD, et al. Evaluating the performance of an institution using an intensive care unit benchmark. Mayo Clin Proc 2005; 80(2): 174-80. 31. Zimmerman JE, Alzola C, Von Rueden KT. The use of benchmarking to identify top performing critical care units: a preliminary assessment of their policies and practices. Journal of critical care 2003; 18(2): 76-86. 32. Katsaragakis S, Papadimitropoulos K, Antonakis P, Strergiopoulos S, Konstadoulakis MM, Androulakis G. Comparison of Acute Physiology and Chronic Health Evaluation II (APACHE II) and Simplified Acute Physiology Score II (SAPS II) scoring systems in a single Greek intensive care unit. Critical care medicine 2000; 28(2): 426-32. 33. Patel PA, Grant BJ. Application of mortality prediction systems to individual intensive care units. Intensive care medicine 1999; 25(9): 977-82. 34. Ryan HM, Sharma S, Magee LA, et al. The Usefulness of the APACHE II Score in Obstetric Critical Care: A Structured Review. J Obstet Gynaecol Can 2016; 38(10): 909-18. 35. Nassar AP, Jr., Mocelin AO, Nunes AL, et al. Caution when using prognostic models: a prospective comparison of 3 recent prognostic models. Journal of critical care 2012; 27(4): 423 e1-7. 36. Escarce JJ, Kelley MA. Admission source to the medical intensive care unit predicts hospital death independent of APACHE II score. JAMA 1990; 264(18): 2389-94. 37. Sinuff T, Adhikari NK, Cook DJ, et al. Mortality predictions in the intensive care unit: comparing physicians with scoring systems. Critical care medicine 2006; 34(3): 878-85.

Page 20 of 20