Original article | Published 23 December 2014, doi:10.4414/smw.2014.14090 Cite this as: Swiss Med Wkly. 2014;144:w14090

SwissScoring – a nationwide survey of SAPS II assessing practices and its accuracy

Marco Previsdominia, Bernard Ceruttib, Paolo Merlanic, Mark Kaufmannd, Elisabeth van Gessele, Hans Ulrich Rothenf, Andreas Perrena a , Department of Intensive Care – Ente Ospedaliero Cantonale, Ospedale San Giovanni, Bellinzona, Switzerland b Unit of Development and Research in Medical Education, Faculty of Medicine, University of Geneva, Geneva, Switzerland c Intensive Care Unit, Department of – Ente Ospedaliero Cantonale, Ospedale Civico, Lugano, and Intensive Care Unit, University Hospital and University of Geneva, Geneva, Switzerland d Department of Anaesthesiology, University Hospital, Basel, Switzerland e Center for Interprofessional Education and Simulation, Faculty of Medicine, University of Geneva, Switzerland f Department of Intensive Care Medicine, Bern University Hospital, Inselspital, Bern, Switzerland

Summary (physician vs. nurse), experience, initial SAPS II training, or presence of a quality control system. OBJECTIVE: The first description of the simplified acute CONCLUSION:This nationwide survey revealed substan- physiology score (SAPS) II dates back to 1993, but little tial variability in the SAPS II scoring results. On average, is known about its accuracy in daily practice. Our purpose SAPS II scoring was overestimated by more than 13%, ir- was to evaluate the accuracy of scoring and the factors that respective of the profession or experience of the scorer or affect it in a nationwide survey. of the structural characteristics of the ICUs. METHODS: Twenty clinical scenarios, covering a broad range of illness severities, were randomly assigned to a Key words: SAPS II; severity score; accuracy; SwissDRG convenience sample of physicians or nurses in Swiss adult intensive care units (ICUs), who were asked to assess the Introduction SAPS II score for a single scenario. These data were com- pared to a reference that was defined by five experienced The simplified acute physiology score II (SAPS II) has researchers. The results were cross-matched with demo- been used for many years for clinical research and quality graphic characteristics and data on the training and quality measurements in European intensive care units (ICUs) control for the scoring, structural and organisational prop- [1–3]. Its original goal was to provide an estimated risk erties of each participating ICU. of hospital mortality for the patients admitted to the ICU, RESULTS: A total of 345 caregivers from 53 adult ICU which was based on given characteristics, selected from a providers completed the SAPS II evaluation of one clinical large sample of medical and surgical cases and assessed scenario. The mean SAPS II scoring was 42.6 ± 23.4, with with logistic regression analyses [4]. The score is assigned a bias of +5.74 (95%CI 2.0–9.5) compared to the reference 24 hours after admission in the ICU and ranges from 0 to score. There was no evidence of bias variation according 163 points, based on age, type of admission, presence of to the case severity, ICU size, linguistic area, profession specified chronic diseases and the worst observed value of a total of 12 clinical items connected with organ function [4, 5]. It is likely that it is still the most commonly utilised sever- List of abbreviations ity adjustment tool for research in Europe even though a GCS Glasgow scale newer severity score has been developed (SAPS III) [6, G-DRG German diagnosis related groups 7]. Moreover, the SAPS II is used to calculate the degree ICC Intraclass correlation coefficient of hospital reimbursement for intensive care unit patients ICU Intensive care unit in Germany (G-DRG) and Switzerland (SwissDRG) [8, 9]. IQR Inter-quartile range The SAPS II is also a key process indicator of the Swiss NEMS Nine equivalents of nursing manpower use score PDMS Patient data management system ICU Minimal Dataset, a quality assurance tool SAPS II Simplified acute physiology score II that is mandatory for all certified ICUs in Switzerland [10]. SAPS III Simplified acute physiology score III The vast majority of information on severity scores was SD Standard deviation generated in a research setting and is based on values that SOFA Sequential Organ Failure Assessment were recorded by specifically trained personnel. Few data SwissDRG Swiss diagnosis related groups are available on the quality of the assessed SAPS II as it

Swiss Medical Weekly · PDF of the online version · www.smw.ch Page 1 of 8 Original article Swiss Med Wkly. 2014;144:w14090

is used in the ICUs [11–13]. A small, retrospective, mul- General study design ticentre audit conducted in Southern Switzerland showed This study was based on online, self-administered ques- that the accuracy of the SAPS II scores, assessed by in- tionnaires addressed to ICU head physicians and ICU bed- tensive care nurses, is rather poor [14]. However, it is un- side personnel. Two steps of data collection, as outlined be- known whether such results can be confirmed in a larger low, were carried out between April and June 2012. sample of ICU healthcare providers with different profes- In a first survey the heads of certified adult Swiss ICUs sional and cultural backgrounds. Moreover, the process of were invited to complete an online questionnaire that was scoring may be influenced by the different organisations or available in three national languages (German, French, and structures in the ICU. Italian). The 36 items on the questionnaires included ques- The primary aim of this nationwide survey was to investig- tions on the socio-demographics and professional quali- ate the reliability and the accuracy of the caregiver-recor- fications (6 items), the ICU structure and organisation (7 ded SAPS II scores by estimating the difference between items), the SAPS II scoring procedures and training for them and the reference values established by five study au- SAPS II scoring (23 items). thors. As a secondary aim, we explored the potential link In a second phase, staff who are usually involved in SAPS between any bias and both the ICU and staff characterist- II scoring were encouraged by the heads of the ICUs to ics. participate in the SAPS II scoring online survey. If they agreed, they received a link to access the survey with a Methods 6-digit key code, known only by the statistician in charge of the analysis, which would allow matching with the centres. An online questionnaire, including 11 items (two socio- demographic items, four items about qualifications and SAPS II training, and five items about SAPS II scoring procedures) was followed by one (out of 20) randomly se- lected clinical scenario that the participant scored with the SAPS II. Every scenario described a hypothetical patient, with the reasons for admission to the ICU (Supplementary file 1) and included charts with vital parameters, laborat- ory values, ventilation details and administered medication. The scenarios were designed to depict clinical situations and sequences of events that are commonly observed in a Swiss adult ICU as well as to cover a large range of SAPS II scores. The reference values of every scenario were established by five different study authors, who as- signed the score separately. Differences were discussed un- A til consensus on the 15 items and sum-score was reached. Caregivers from centres where the SAPS II score was re- corded based on values supplied by a patient data manage- ment system (PDMS) were also given a randomly selec- ted scenario, but this was presented with a specific digital layout using either Centricity Critical Care (Version 7.0 SP3, General Electric Healthcare, Barrington, IL, USA) or MetaVision (Version 5.46.44, iMDSoft, Needham, MA, USA). The aim was to offer the participants the same sys- tem that they used in their current clinical practice. Parti- cipants were encouraged to handle their cases as usual and without external support. For every ICU, the following key indicators related to structure were retrieved from the national ICU Minimal B Dataset: average SAPS II, number of beds per unit and Figure 1 number of patients per year. A Relationship between the rater assessed SAPS II scores and the The objectives and design of this quality assurance study reference value. The dots stand for every observation made, i.e. were presented to the Executive Committee of the Swiss the score given by every participant (y-value) versus the reference Society of Intensive Care Medicine, who endorsed the pro- value (x-value). The 16 vertical grouped values stand for the reference values of the 20 cases that have been assessed by the ject. The anonymity of the participating caregivers and data participants (some cases had the identical SAPS II reference safety were ensured. The Cantonal Ethics Committee Ti- score). The continuous line was fitted with a linear mixed effect cino (6500 Bellinzona, Switzerland) approved the protocol after model selection. The dotted lines denote the 95% confidence and waived the need for informed consent. interval around the fitted line. B Bland-Altman plot showing the relationship between the difference between observed and reference values and the average Statistical analysis of reference and observed values. The dotted lines show the mean Previous studies reported an intra-class correlation coeffi- difference ±1.96 standard deviation. cient (ICC) of 0.8–0.9 for the SAPS II sum-score [11, 14,

Swiss Medical Weekly · PDF of the online version · www.smw.ch Page 2 of 8 Original article Swiss Med Wkly. 2014;144:w14090

15]. Considering an ICC (as a measure of reliability) of 0.7 Results as the lowest acceptable threshold, and aiming at an ICC of 0.85 (middle of the “average – good” range), a sample size ICU general data, practice and quality assurance used of 20 clinical scenarios that were each assessed 10 times to assign the SAPS II scores was deemed necessary for detecting the difference between In 2011, the 78 eligible adult ICUs recognised by the Swiss 0.7 and 0.85 with a power of 80% and a Type I error rate of Society of Intensive Care [22] recorded a mean SAPS II 0.05 [16]. score of 30 ± 17. Among all contacted units, 63 (81%) gave Weighted kappa values were used to establish reliability their approval for participation in the heads of ICU survey. between measures for the SAPS II items (categorical meas- The participants were 46 to 55 years old in 63% of cases, ures) [17, 18], and the intraclass correlation coefficient younger in 16% and older in 21%, respectively. The lin- (ICC) for the SAPS II Score (continuous measure) [19, guistic regions were represented as follows: 45 participants 20]. The proportion of rater agreement, as a measure of (71%) from the German part, 14 (23%) from the French re- accuracy, was calculated comparing each rater against the gion and 4 (6%) from the Italian speaking part of Switzer- reference value. Linear mixed effect models (the clinical land. A “reminder” for the SAPS II scoring was available scenarios were taken as random effects) were used to in- in 87% of the ICUs, which was activated at the 24-hour vestigate the potential links between the bias (i.e. the differ- deadline (16%), the next day (22%) or later (62%). Ap- ence between the observed value and the reference value) proximately 68% of the ICUs had a quality control system, and several structural and individual covariates using a for- but a vast majority of the ICUs (87%) had never had an ward selection (log-likelihood ratio test; see list in Supple- audit for the SAPS II scoring quality. mentary file 2). A Bland and Altman plot was used to visu- alise the bias and limits of agreement between the SAPS Caregivers who scored the SAPS II II sum scores [21]. Weighted hospital mortality was calcu- A total of 345 participants from 53 ICUs completed the lated based on the original equation [4]. Additional qual- SAPS II evaluation of one clinical scenario (table 1). They ity control checks of the results for the clinical scenarios to were mostly women and most of them worked as phys- rule out imbalances due to translation included an analysis icians (German: 91%; French: 41%; and Italian: 6%; p of variance of the SAPS sum-scores between the linguistic <0.001). Around two-thirds of the responders (63%) were regions. trained by a colleague, one-third was trained by their man- Continuous variables were expressed as the mean ± stand- ager, only 5% received structured training, and 12% ard deviation (SD), unless otherwise specified, or the medi- learned from a manual (multiple responses were allowed). an with an inter-quartile range (IQR) for highly asymmetric Overall, there were 116 participants from intensive care data. Categorical variables were presented with the number units equipped with a PDMS. of observations and percentage. A result associated with a p-value lower than 0.05 was considered significant. SAPS II reference values An independent statistician carried out the data manage- The twenty clinical scenarios were assessed by five study ment and the statistical analyses. Data collection was per- authors. The mean SAPS II score was 35 ± 21, and the ® formed via SurveyMonkey , Palo Alto, CA, USA. All ana- minimum-maximum (median) was 6–79 (27). The agree- ® lyses were performed with TIBCO Spotfire S+ 8.1 for ment for the sum scores among the five authors was excel- Windows, TIBCO Software Inc., Palo Alto, CA, USA.

Table 1: Characteristics of the study participants scoring the SAPS II. Characteristic Categories Scoring participants (n = 345, 53 ICUs) Age ≤45 years 273 (79%) 46–55 years 60 (17%) ≥56 years 11 (3%) Linguistic region of ICU German 210 (61%) French 56 (16%) Italian 79 (23%) Gender Male 161 (47%) Female 183 (53%) Position – Qualification Consultant 81 (23%) Resident physician 137 (40%) RN with certificate for 93 (27%) Other 33 (10%) Experience with the SAPS II ≤3 months 63 (18%) >3, ≤12 months 70 (20%) >12 months 207 (60%) Frequency of the SAPS II scoring On daily basis 144 (42%) At least once per week 111 (32%) Less than once per week 84 (24%) Data in n (%). RN = registered nurse. Missing values: Age 1; Linguistic region 0; Gender 1; Position-Qualification 1; Experience with the SAPS II 5; Frequency of the SAPS II scoring 6.

Swiss Medical Weekly · PDF of the online version · www.smw.ch Page 3 of 8 Original article Swiss Med Wkly. 2014;144:w14090

lent (ICC = 0.941). Table 2 shows the authors’ reliability acceptable for benchmarking, but seems excessive for clin- for the single items assessed. ical research. Since 2012, the SAPS II has become a component of the SAPS II scoring of the clinical scenarios ICU specific codification (SwissDRG system) for severe Every clinical scenario was assessed between 10 and 26 patients who have a high resource use. For this purpose the times (Supplementary file 1). Among the 345 responders, score that is assessed at the end of the first 24 hours is ad- the average SAPS II score was 42.56 ± 23.38, with an ICC ded to the cumulative result of the NEMS score, which is of 0.79, a bias of +5.74 (95%CI 2.00 to 9.48) compared to an instrument to quantify nursing workload at the ICU level the reference values (Supplementary file 3) and no eviden- [23] and is computed (varying between 0 and 56 points) at ce of variation according to the case severity (p = 0.195; the end of each eight-hour shift during the entire ICU stay. fig. 1). The (weighted) mean predicted mortality was 33.4 This type of codification is relevant for cost weight only ± 33.0% against 27.1 ± 31.6% of the reference value (p = above 500 points [9]. Thus, in terms of the current coding 0.011, difference = 6.3% ± 20%). There was no evidence practices for SwissDRG, the bias may be considered as ir- of association between the bias and the ICU size and other relevant. Nevertheless, more extensive use of the SAPS II structural or organisational ICU features (table 3). Similar cannot be recommended (e.g., integration of the everyday results were found for the demographic characteristics of SAPS II score – without the GCS – as in the German G- the participants (table 4), type of initial SAPS II training (p DRG-System) [8]. = 0.164), and the method used for scoring (p = 0.759). The mean weighted kappa value was 0.47, which, based Only 7.8% (27/345) of rated cases matched the reference on the criteria proposed by experts, grades the reliability value for each item. Bilirubin, temperature and chronic dis- of SAPS II scoring among our participants as fair to mod- ease were the most accurately scored items (93%, 93%, erate [17, 18]. In accordance with the existing knowledge, and 91%, respectively), whereas the lowest agreement was the less reliable variables were those that require calcula- found for urinary output and the tions (oxygenation ratio and urinary output), judgement or (63% and 64%). The mean weighted kappa was 0.47 with implementation of some definition on the part of the rater best values for age (0.77) and lowest for urinary output and (Glasgow Coma Scale) [24]. For instance, errors for diur- sodium (0.20 and 0.28, Supplementary file 4). esis were caused by miscalculation or by the neglected ex- trapolation of the total urinary output when patients were Discussion discharged from the ICU before the 24–hour deadline. The Glasgow coma scale, a well known source of error, pro- The main finding of this study was an overall overestim- duced most of the overestimation of the sum score as result ation of the SAPS II by ICU personnel. Theoretically, in of an incorrect assessment of the sedated patients with terms of mortality prediction the change due to a bias of 5 [25–27]. points will depend on the actual score and on the compos- Interestingly, the observed bias was independent of the par- ition of the analysed sample. In the very low range (SAPS ticipants’ personal characteristics and of the ICU’s structur- II ≤20) and in the highest range of SAPS II (≥80), the im- al or organisational features. Our results add to the findings pact of the bias will be minimal, whereas in the middle of a retrospective multicentre audit about nurse-registered range (30–60 points) it will be definitely higher. Therefore, SAPS II performance [14]. the relevance clearly depends on the spectrum of acute ill- Some variability is inherent to scoring, approximately ness. In our study this generated an increase of the pre- 10–15%, as was shown for the APACHE II by testing dicted mortality of 6.3%, which is, in our opinion, barely the intra-individual variability [28]. Nevertheless, our data show that there is room for improvement, particularly when

Table 2: Reliability across the authors for the single items of the SAPS II score. Item Kappaa Mean agreementb Perfect agreementc Heart rate 0.544 80% 45% Systolic blood pressure 0.683 89% 65% Temperature 0.767 96% 80% Oxygenation 0.699 87% 65% Urinary output 0.782 96% 80% Urea 0.812 85% 90% Leucocytes 0.836 96% 85% Potassium 0.763 97% 90% Sodium 0.543 94% 75% Bicarbonate 0.671 91% 70% Bilirubin 1.000 100% 100% Glasgow Coma Scale 0.583 90% 70% Age 0.900 96% 85% Chronic diseases 0.789 96% 85% Type of admission 0.877 96% 80% a Weighted Fleiss' Kappa. b Mean proportion of agreement among the 5 authors versus the reference value. c Percentage of total agreement among the 5 authors versus the reference value.

Swiss Medical Weekly · PDF of the online version · www.smw.ch Page 4 of 8 Original article Swiss Med Wkly. 2014;144:w14090

compared to a study conducted in nine Dutch ICUs, which (APACHE) II scoring decreased when strict guidelines and showed a non-significant bias of 0.4 [12]. In those ICUs, a training programme are implemented [29]. the NICE registry was applied, which has implemented Likewise, the Swiss Society of Intensive Care Medicine many procedures (e.g., data dictionary with clear defini- might consider modifying the regulations for the units’ cer- tions of items, mandatory training, automatic quality check, tification. They may include the requirement of a clear de- and automatic selection of variables by a computer al- scription of the local responsibility for the quality of the gorithm) that are aimed at improving the quality of the key process indicators, for example by assigning a single data. Another study showed that the inter-observer variab- trained representative per ICU with the task of educating ility in Acute Physiology and Chronic Health Evaluation the ICU staff. New collaborators should receive a mandat-

Table 3: Relationship between the scoring bias (scored vs reference sum scores) and structural ICU data. Characteristic Categories Bias p-valuea Mean annual SAPS II in 2011 (points) ≤29 5.0 ± 12.9 0.431 >29 and ≤32 5.1 ± 13.5 >32 6.6 ± 14.6 Number of beds per unit ≤8 5.6 ± 13.5 0.884 9–15 6.5 ± 13.9 >15 2.9 ± 12.9 Number of patients/year ≤800 6.0 ± 14.2 0.537 801–1,200 6.4 ± 12.6 >1,200 2.9 ± 14.0 Number of senior physicians (FTE) ≤1 7.6 ± 12.8 0.533 >1 and ≤3 4.9 ± 13.2 >3 5.6 ± 14.3 Number of residents per day shift (FTE) ≤1 5.5 ± 12.5 0.233 >1 and ≤2 6.0 ± 14.8 >2 4.1 ± 12.9 Presence of dedicated resident during evening shift Yes 5.0 ± 13.5 0.410 No 5.7 ± 13.7 Presence of dedicated resident during night shift Yes 5.6 ± 13.9 0.685 No 5.3 ± 13.4 ICU affiliation independent 6.2 ± 13.1 0.133 non independent 3.8 ± 15.0 SAPS II data acquisition method semi-automatic/automatic by PDMS 2.3 ± 18.8 0.459 manual 5.8 ± 13.1 Presence of quality control Yes 6.3 ± 13.7 0.998 No 4.8 ± 14.1 Presence of feed-back mechanism Yes 5.6 ± 14.3 0.360 No 9.6 ± 14.9 a Log likelihood ratio test. FTE = full time equivalent; PDMS = patient data management system.

Table 4: Relationship between the scoring bias (scored vs. reference sum scores) and personal characteristics. Characteristic Categories Bias p-valuea Gender Female 6.6 ± 14.2 0.206 Male 4.1 ± 12.6 Age ≤45 yrs 5.3 ± 13.8 0.652 46–55 yrs 5.2 ± 12.4 ≥56 yrs 8.8 ± 13.8 Profession Senior physician 3.7 ± 10.8 0.328 Resident physician 4.7 ± 14.6 RN with certificate for Critical care nursing 8.0 ± 13.4 Other 6.4 ± 13.4 Language German 4.0 ± 13.7 0.209 French 7.6 ± 13.0 Italian 7.7 ± 13.3 Experience with SAPS II ≤3 months 2.8 ± 15.7 0.758 >3, ≤12 months 5.2 ± 13.8 >12 months 6.0 ± 12.4 Frequency of SAPS II scoring On a daily basis 4.4 ± 12.6 0.720 At least once per week 6.0 ± 14.9 Less than once per week 5.8 ± 12.5 a Log likelihood ratio test. RN = registered nurse.

Swiss Medical Weekly · PDF of the online version · www.smw.ch Page 5 of 8 Original article Swiss Med Wkly. 2014;144:w14090

ory, specific introduction and should be provided with of- the caregivers, method used for data acquisition and struc- ficial manuals containing all SAPS II definitions. Further- tural characteristics of the ICUs. This fact might hamper more, the frequency of training or case-based-discussions the scores’ validity in terms of quality improvement, in the ICUs could be determined. Additionally, the quality benchmarking and clinical research.A structured national of the collected data should be constantly verified, such as programme reaching all involved professionals, including by mandatory and documented spot checks. Automatic re- training sessions, official manuals and quality checks, trieval of variables from a PDMS is feasible and may con- should be contemplated by the Swiss Society of Intensive tribute to an increase in the scoring accuracy, depending Care Medicine. on the applied validation procedures [12, 30]. In fact, this already happens in most larger hospitals. However, com- pared to manual acquisition of data, it introduces new prob- Acknowledgement: The authors are most grateful to the Swiss lems due to higher sample frequency (e.g. blood pressure ICU heads and all other participants of the surveys. They would measurements), that have been shown to result in high- also like to express their special appreciation to Dr. Marianne Zumbrunn, Basel, for her contribution. Endorsement and er scores and a lower standardised mortality ratio, poten- financial support of this project by the Swiss Society of tially biasing the comparison between hospitals with differ- Intensive Care Medicine are recognised. ent practices [31]. Funding / potential competing interests: This study was This study has several limitations to be considered before supported by a grant from the Research Fund of the Swiss its results are reflected on the daily clinical practice. First, Society of Intensive Care Medicine. the clinical scenarios were expressly created for the study and their number was limited to twenty, which was based Correspondence: Marco Previsdomini, MD, Intensive Care on power analysis. The number of scoring difficulties in- Unit, Ospedale Regionale Bellinzona e Valli, CH-6500 serted in the scenarios might differ from that of real clinical Bellinzona, Switzerland, marco.previsdomini[at]eoc.ch cases, therefore our results could be regarded more as ex- perimental than of clinical relevance. Second, their mean References SAPS II sum-score was 5 points higher than that observed 1 Moreno RP, Hochrieser H, Metnitz B, Bauer P, Metnitz PGH. Charac- in Switzerland in 2011, and they could not reproduce the terizing the risk profiles of intensive care units. Intensive Care Med. real complexity of the Swiss ICU patient population. Third, 2010;36(7):1207–12. the various characteristics of SAPS II items were not 2 Minne L, Eslami S, de Keizer N, de Jonge E, de Rooij SE, Abu-Hanna evenly distributed, which may result in additional selection A. Effect of changes over time in the performance of a customized bias. Therefore, we cannot exclude that other vignettes SAPS-II model on the quality of care assessment. Intensive Care Med. would have produced slightly different results. The use 2011;38(1):40–6. of mixed effect models, however, should at least partially 3 Vosylius S, Sipylaite J, Ivaskevicius J. Evaluation of intensive care unit performance in Lithuania using the SAPS II system. Eur J Anaesthesi- account for this. Fourth, we assumed that everyone sur- ol. 2004;21(8):619–24. veyed was skilled in retrieving some information from the 4 Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute charts, even if not expressly noted in the case description. Physiology Score (SAPS II) based on a European/North American mul- However, some participants may have been better able to ticenter study. JAMA. 1993;270(24):2957–63. find the information needed if they had personally wit- 5 SAPS II Calculator [Internet]. 2014. Available from: ht- nessed the sequence of events. Fifth, we could not verify if tp://clincalc.com/IcuMortality/SAPSII.aspx all participants are regularly involved in SAPS II scoring. 6 Metnitz PGH, Moreno RP, Almeida E, Jordan B, Bauer P, Campos RA, In fact, one-quarter declared to deal with this task less than et al. SAPS 3 – From evaluation of the patient to evaluation of the in- once a week and this might cause a wider inter-rater vari- tensive care unit. Part 1: Objectives, methods and cohort description. Intensive Care Med. 2005;31(10):1336–44. ability. Finally, 15 cases contained patient charts that were 7 Moreno RP, Metnitz PGH, Almeida E, Jordan B, Bauer P, Campos presented in a classical way, whereas 5 cases had a “digital” RA, et al. SAPS 3 – From evaluation of the patient to evaluation of like layout. Only participants working in ICUs equipped the intensive care unit. Part 2: Development of a prognostic model with a PDMS were provided with a “digital” chart (accur- for hospital mortality at ICU admission. Intensive Care Med. acy was not different for this subgroup), but we cannot rule 2005;31(10):1345–55. out that some participants struggled with unfamiliar graph- 8 Deutsches Institut für Medizinische Dokumentation und Information, ic details. OPS Version 2014, Kapitel 8, Nichtoperative Therapeutische Mass- nahmen 8–98, 2014, Dokumentationsvorgaben zur Erfassung der In- We believe that the participation of 345 professionals (i.e. tensivmedizinischen Komplexbehandlung. Available from: ht- about one-third of all potential responders) from 53 dif- tp://www.dimdi.de/static/de/klassi/faq/ops/kapitel_8/ops-anleitung-in- ferent Swiss ICUs offers a reasonably good snapshot of tensivmedizin-8009.pdf the national practices because these results are representat- 9 Klassifikationen BM. Schweizerische Operationsklassifikation ive for a vast proportion of the Swiss ICUs. However, the (CHOP). Statistik BF, editor. 2013 Jul 31. Available from: ht- tp://www.bfs.admin.ch/bfs/portal/de/index/infothek/nomenklaturen/ rather limited participation of the French-speaking region blank/blank/chop/02/05.html (only 16% of scorers; compared to 23% of the Swiss popu- 10 Rothen HU, Kaufmann M. Ein wichtiges Instrument zur Qualitäts- lation) may interfere with drawing reliable conclusions for sicherung in der Intensivmedizin. The Medical Journal. 2009;2:18–21. this specific part of the country. 11 Strand K, Strand LI, Flaatten H. The interrater reliability of SAPS II and In conclusion, this nationwide survey suggests that the ac- SAPS 3. Intensive Care Med. 2010;36(5):850–3. curacy of the SAPS II scores calculated by the ICU caring 12 Arts D, de Keizer N, Scheffer G-J, de Jonge E. Quality of data collected staff might only be moderate producing a significant bi- for severity of illness scores in the Dutch National Intensive Care as.This was independent of the profession, experience of Evaluation (NICE) registry. Intensive Care Med. 2002;28(5):656–9.

Swiss Medical Weekly · PDF of the online version · www.smw.ch Page 6 of 8 Original article Swiss Med Wkly. 2014;144:w14090

13 Grønlykke L, Brandstrup SLR, Perner A. Data from clinical database 23 Miranda DR, Moreno R, Iapichino G. Nine equivalents of nursing man- on septic are valid. Dan Med J. 2012;59(10):A4522. power use score (NEMS). Intensive Care Med. 1997;23(7):760–5. 14 Perren A, Previsdomini M, Perren I, Merlani P. Critical Care Nurses In- 24 Arts DGT, de Keizer NF, Vroom MB, de Jonge E. Reliability and accur- adequately Assess SAPS II Scores of Very Ill Patients in Real Life. Crit acy of Sequential Organ Failure Assessment (SOFA) scoring. Crit Care Care Res Pract. 2012;2012:919106. Med. 2005;33(9):1988–93. 15 Strand K, Søreide E, Aardal S, Flaatten H. A comparison of SAPS II 25 Chen LM, Martin CM, Morrison TL, Sibbald WJ. Interobserver variab- and SAPS 3 in a Norwegian intensive care unit population. Acta An- ility in data collection of the APACHE II score in teaching and commu- aesthesiol Scand. 2009;53(5):595–600. nity hospitals. Crit Care Med. 1999;27(9):1999–2004. 16 Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for 26 Holt AW, Bury LK, Bersten AD, Skowronski GA, Vedig AE. Prospect- reliability studies. Stat Med. 1998;17(1):101–10. ive evaluation of residents and nurses as severity score data collectors. 17 Landis JR, Koch GG. An application of hierarchical kappa-type statist- Crit Care Med. 1992;20(12):1688–91. ics in the assessment of majority agreement among multiple observers. 27 Goldhill DR, Sumner A. APACHE II, data accuracy and outcome pre- Biometrics. 1977;33(2):363–74. diction. Anaesthesia. 1998;53(10):937–43. 18 Fleiss JL. Measuring nominal scale agreement among many raters. Psy- 28 Polderman KH, Christiaans HM, Wester JP, Spijkstra JJ, Girbes AR. chological Bulletin. 1971;76(5):378–82. Intra-observer variability in APACHE II scoring. Intensive Care Med. 19 Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater re- 2001;27(9):1550–2. liability. Psychol Bull. 1979;86(2):420–8. 29 Polderman KH, Jorna EM, Girbes AR. Inter-observer variability in 20 McGraw KO, Wong SP. Forming inferences about some intraclass cor- APACHE II scoring: effect of strict guidelines and training. Intensive relation coefficients. Psychological Methods. American Psychological Care Med. 2001;27(8):1365–9. Association; 1996;1(1):30–46. 30 Bosman RJ, Oudemane van Straaten HM, Zandstra DF. The use of in- 21 Altman DG, Bland JM. Measurement in Medicine: the Analysis of tensive care information systems alters outcome prediction. Intensive Method Comparison Studies. The statistician. 1983;32:307–17. Care Med. 1998;24(9):953–8. 22 Anerkannte Intensivstationen. 2013 Oct 23. Available from: 31 Suistomaa M, Kari A, Ruokonen E, Takala J. Sampling rate causes http://www.sgi-ssmi.ch/index.php/liste-der-anerkannten-is.html bias in APACHE II and SAPS II scores. Intensive Care Med. 2000;26(12):1773–8.

Swiss Medical Weekly · PDF of the online version · www.smw.ch Page 7 of 8 Original article Swiss Med Wkly. 2014;144:w14090

Figures (large format)

A

B

Figure 1 A Relationship between the rater assessed SAPS II scores and the reference value. The dots stand for every observation made, i.e. the score given by every participant (y-value) versus the reference value (x-value). The 16 vertical grouped values stand for the reference values of the 20 cases that have been assessed by the participants (some cases had the identical SAPS II reference score). The continuous line was fitted with a linear mixed effect after model selection. The dotted lines denote the 95% confidence interval around the fitted line. B Bland-Altman plot showing the relationship between the difference between observed and reference values and the average of reference and observed values. The dotted lines show the mean difference ±1.96 standard deviation.

Swiss Medical Weekly · PDF of the online version · www.smw.ch Page 8 of 8