Improving surgical outcomes through adoption of evidence-based process measures: Intervention specific or associated with overall hospital quality?

Benjamin S. Brooke, MD,a Robert A. Meguid, MD, MPH,a Martin A. Makary, MD, MPH,a,c Bruce A. Perler, MD, MBA,a Peter J. Pronovost, MD, PhD,b,c and Timothy M. Pawlik, MD, MPH,a Baltimore, MD

Background. The Leapfrog Group aims to improve surgical outcomes through promoting hospital adoption of procedure-specific process measures, although it is unclear whether compliance reflects a hospital’s overall quality. The purpose of this study was to evaluate whether implementation of Leapfrog’s standard for routine b-blockade was associated with reductions in mortality after open abdominal aortic aneurysm (AAA) repair alone versus other high-risk operations. Methods. Using a 2:1 matched case-control study design, hospitals that had not adopted the b-blockade standard (n = 72) were compared with hospitals that had implemented this Leapfrog standard (n = 36). Leapfrog survey data were linked to patient outcomes in the California OSHPD database from 2000 to 2005. Random-effects Poisson regression models were used to evaluate in-hospital mortality over time for patients undergoing AAA repair versus esophagectomy, hepatectomy, pancreatectomy, colectomy, gas- trectomy, and pulmonary lobectomy. Results. A total of 6,199 AAA repairs, 2,780 esophagectomies, 2,544 hepatectomies, 2,909 pancrea- tectomies, 57,795 colectomies, 6,267 gastrectomies, and 10,210 lobectomies were analyzed. AAA- associated mortality significantly declined in hospitals that adopted the b-blocker standard (relative risk [RR]: 0.49; 95% confidence interval [CI]: 0.24--0.97; P < .05). Implementation of this Leapfrog standard had no effect on reducing adjusted mortality rates for other high-risk operations, including esophagectomy (RR: 0.70; 95% CI: 0.25--1.89), hepatectomy (RR: 1.16; 95% CI: 0.32--4.29), pan- createctomy (RR: 0.76; 95% CI: 0.28--2.02), colectomy (RR: 1.12; 95% CI: 0.86--1.44), gastrectomy (RR: 1.17; 95% CI: 0.57--2.43), and lobectomy (RR: 0.98; 95% CI: 0.46--2.08) (all P > .05). Conclusion. Compliance with peri-operative b-blockade resulted in a significant reduction in mortality after open AAA repair over time, but it had no crossover effect on mortality associated with other high-risk operations in the same hospital. These data suggest that improvements in outcomes resulting from the adoption of evidence-based process measures are procedure specific and do not necessarily reflect overall hospital quality. (Surgery 2010;147:481-90.)

From the Departments of Surgerya and Anesthesiology/Critical Care Medicine,b Johns Hopkins University School of Medicine, Baltimore; and Department of Health Policy and Management,c Johns Hopkins University School of Public Health, Baltimore, MD

AN ESTIMATED 50,000 IN-HOSPITAL DEATHS occur annu- peri-operative adverse events are known to be pre- ally in the United States among patients who have ventable with the reliable application of evidence- undergone operative procedures.1 Many of these based medicine (EBM) practices by hospitals and health care providers. This fact has been high- Supported by Grant 1KL2RR025006-01 from the National lighted by several reports released by the Institute Center for Research Resources (NCRR), a component of the of Medicine over the past decade, which have pro- National Institutes of Health (NIH), and the NIH Roadmap for Medical Research. jected that thousands of patient lives might be Accepted for publication October 7, 2009. saved if patients underwent operations at a hospi- tal that adheres to EBM.2,3 Reprint requests: Timothy M. Pawlik, MD, MPH, Department of Surgery, Johns Hopkins Hospital, 600 North Wolfe Street, Based on these findings, several nationwide Halsted 614, Baltimore, MD 22187-6681. E-mail: tpawlik1@ initiatives have been formed that aim to refer jhmi.edu. surgical patients selectively to hospitals that adhere 0039-6060/$ - see front matter to EBM practices. One of the largest such pro- Ó 2010 Mosby, Inc. All rights reserved. grams is the Leapfrog Group Hospital Quality and doi:10.1016/j.surg.2009.10.037 Patient Safety Initiative, which was started in 2000

SURGERY 481 482 Brooke et al Surgery April 2010 by a group of business and health care leaders.4 A number of hospitals surveyed. The Leapfrog data cornerstone of the Leapfrog program has been the are based on surveys sent to the Corporate Exec- promotion of evidence-based standards for high- utive Officer or head administrator of 337 acute- risk operations, including abdominal aortic care hospitals in urban regions of California, with aneurysm (AAA) repair, esophagectomy, pancre- the collected data representing self-reported infor- atic resection, coronary artery bypass grafting mation regarding hospital demographics and an- (CABG), aortic valve replacement, and bariatric nual compliance with each of the Leapfrog Group surgery. The Leapfrog standards include encourag- hospital quality and safety standards. A total of 212 ing patient referral to hospitals that meet defined targeted hospitals returned Leapfrog surveys and case volume thresholds, as well as giving incentives were available for review. for hospitals to promote implementation of evi- In all, 36 California hospitals were identified in dence-based process measures for specific proce- which a hospital policy for routine peri-operative b- dures.5 Although the benefits of meeting EBM blocker use during AAA repair was in place starting standards on surgical outcomes have been noted in 2003. To meet Leapfrog criteria for this process by several recent analyses, most hospitals do not measure, at least 80% of patients who underwent meet case volume standards and overall compli- elective AAA repair at a given hospital must have ance with the adoption of evidence-based process been on b-blocker therapy during their hospitali- measures remains low.5,6 zation as well as at the time of discharge. California It is well recognized that numerous barriers may hospitals that met this evidence-based standard limit or prevent health care organizations from were matched by total hospital admission volume adopting new evidence-based practices into hospi- (1:2) to 72 control hospitals that returned Leap- tal policy.7-9 Hospitals that overcome these obsta- frog surveys declaring noncompliance with routine cles may therefore have unique characteristics or b-blocker use. To maintain homogeneity, only institution-wide programs dedicated toward quality hospitals that used open AAA repair were identi- improvement. Indeed, the adoption of evidence- fied and used for the purpose of analyses. To based process measures may serve as a surrogate determine whether improvements in outcomes marker for a hospital’s commitment toward EBM among hospitals willing to adopt evidence-based and/or improving operative processes of care. process measures were procedure specific (eg, Prior studies have suggested that hospitals with b-blockade for AAA peri-operative outcomes) or low mortality rates for 1 operation tend to have reflective of hospital-wide quality improvements lower mortality rates for other operations based over time, the peri-operative mortality outcomes of on shared evidence-based processes of care.10 As 6 other high-risk operations were determined. such, we sought to determine whether improve- Specifically, the peri-operative outcomes of pa- ments in outcomes among hospitals willing to tients who underwent elective esophagectomy, adopt evidence-based process measures were spe- pancreatectomy, hepatectomy, colectomy, gastrec- cific for the procedures to which they apply or tomy, and pulmonary lobectomy in the same whether the adoption of such evidence-based mea- California hospitals were assessed. All data on sures are reflective of more global hospital-wide patients who underwent any of these elective improvements over time. To address this question, operative procedures at the identified hospitals we evaluated whether hospitals that adopted the were obtained from the California Office of Leapfrog evidence-based process measure for rou- Statewide Health Planning and Development tine b-blocker use during AAA repair also experi- (OSHPD) database for the years between 2000 enced improved outcomes over time for other and 2005. The data were then linked by OSHPD high-risk operations. Such information is impor- identification number to the Leapfrog Group tant for determining markers of hospital quality survey results. International Classification of and may help ensure that operative patients are Diseases, 9th Revision procedure codes were referred to centers with the best outcomes. used to identify open AAA repair (38.34, 38.36, 38.44, 38.64, 39.25, and 39.52), esopha- METHODS gectomy (42.40, 42.41, 42.42, and 43.99), pan- Hospital and patient data. Response data ob- createctomy (52.70, 52.51, 52.52, 52.53, 52.59, tained from the Leapfrog Group Hospital Quality and 55.26), hepatectomy (50.22 and 50.30), and Safety Surveys sent to California hospitals colectomy (45.7, 45.71, 45.72, 45.73, 45.74, annually between 2001 and 2005 were reviewed. 45.75, 45.76, 45.79, and 45.8), gastrectomy California was the first state-wide region to be (43.5, 43.6, 43.7, 43.8, 43.89, 43.9, and 43.99), targeted by Leapfrog, and it contains the largest and pulmonary lobectomy (32.3 and 32.4) Surgery Brooke et al 483 Volume 147, Number 4 procedures from the OSHPD database. The pro- cedure codes for endovascular AAA repair (39.71) and nonelective operations were excluded from analyses. The Johns Hopkins University School of Medicine Institutional Review Board approved this study. Study design. Hospital characteristics and in- hospital mortality for all operative procedures were compared over consecutive time periods as follows: (1) the 3-year period (2000--2002) prior to the release of the Leapfrog Group process measure standard for routine b-blocker use; and (2) the 3-year period (2003--2005) after California hospitals were either compliant (treatment group) or non- b compliant (control group) with the -blocker Fig 1. Location of California hospitals that complied standard. The hospital-level covariates that were with the Leapfrog Group standard for routine b-blocker analyzed included the number of total annual use during AAA repair (gray circles) as compared with admissions, number of floor admissions, number control hospitals (white circles) that failed to implement of intensive care unit admissions, number of this process measure. staffed floor beds, number of licensed floor beds, number of staffed intensive care unit beds, num- on in-hospital mortality following AAA repair was ber of licensed intensive care unit beds, hospital calculated as the mortality rate ratio for hospitals membership status in health organizations, pres- that complied with the Leapfrog b-blocker process ence of an Accreditation Council for Graduate measure over hospitals that did not comply with Medical Education (ACGME) accredited general this standard (control group) for each time surgery or vascular surgery residency training pro- period. A ratio of mortality rate ratios was then gram, and total number of elective operations calculated to assess the impact of implementing performed at each hospital (open AAA repairs, this evidence-based process measure over time. esophagectomies, pancreatectomies, hepatecto- P values less than a = .05 (2-sided) were considered mies, colectomies, gastrectomies, and lobecto- to be significant for all statistical tests and models. mies), as well as whether hospitals met Leapfrog Stata 10.0 statistical software (Stata Corporation, volume standards for AAA repair ($50 cases/ College Station, TX) was used for all analyses. year), esophagectomy ($13 cases/year), and pan- createctomy ($11 cases/year). Patient-level varia- RESULTS bles that were analyzed included age, sex, race, Hospital characteristics. The California hospi- ethnicity, insurance status, and Charlson Comor- tals that reported compliance with the Leapfrog bidity Index score. The main outcome measure b-blockade process measure (n = 36) were distrib- was in-hospital mortality after elective surgery. uted evenly among major metropolitan areas Statistical analysis. Hospital characteristics and throughout the state compared with control hospi- in-hospital mortality before and after Leapfrog tals (n = 72; Fig 1). Case and control hospitals were AAA standards were compared between each of matched equally based on hospital-level variables, the defined groups using Chi-square tests for including the mean number of total admissions, categorical variables and analysis of variance or mean number of intensive care unit admissions, in- the Student paired t tests for continuous variables tensive care unit beds, and floor beds (Table I). that were normally distributed. The Wilcoxon Similarly, the proportion of hospitals that be- signed rank test was used to compare non-normally longed to a health system organization or that distributed data between time periods. Both fixed- had an ACGME surgical training program did effects and random-effects Poisson regression not vary between the 2 groups (Table I). models with an intercept for each hospital were With regard to hospital-level volume characteris- used to estimate the impact of meeting Leapfrog’s tics, no significant differences in the proportion of b-blockade standard on in-hospital mortality for case and control hospitals that met Leapfrog group each operative procedure before and after adjust- procedure volume criteria for AAA repair, esopha- ing for hospital and patient-level confounders.11 gectomy, or pancreatectomy were noted (Table I). These models account for clustering of outcomes In addition, no significant differences were found within hospitals. The effect of Leapfrog standards in the median number of open AAA repair, 484 Brooke et al Surgery April 2010

Table I. Characteristics of California hospitals (n = 108) stratified by compliance with Leapfrog b-blocker process measure Variable b-blocker (n = 36) Control (n = 72) P value Total admissions, mean (± SD) 14.4 (± 7.1) 13.5 (± 5.7) .48 Intensive care unit admissions, mean (± SD) 1.6 (± 1.4) 1.8 (± 1.5) .58 Floor beds, mean (± SD) 219.7 (± 103.2) 209.6 (± 104.8) .64 Intensive care unit beds, mean (± SD) 25.1 (± 14.1) 27.9 (± 25.2) .54 Member of health system organization, number (%) 30 (83.3) 58 (81.7) .83 ACGME surgical training program, number (%) 4 (11.1) 6 (8.3) .64 Met Leapfrog AAA repair volume criteria 3 (8.3) 14 (19.4) .14 Met Leapfrog pancreatectomy volume criteria* 6 (16.7) 10 (13.8) .62 Met Leapfrog esophagectomy volume criteriay 3 (8.3) 6 (8.3) .99 *Hospitals fulfilled Leapfrog evidence-based hospital referral standard by averaging 11 of more pancreatectomy procedures per year. yHospitals fulfilled Leapfrog evidence-based hospital referral standard by averaging 13 or more esophagectomy procedures per year. SD, Standard deviation.

Table II. Median annual volume of high-risk operations performed in California hospitals stratified by compliance with Leapfrog b-blocker process measure Procedure Beta-blocker (n = 36) Control (n = 72) P value Open AAA repair, median (IQR) 15 (10–26) 16 (9–33) .78 Pancreatectomy, median (IQR) 18 (8–37) 12 (5–24) .10 Hepatectomy, median (IQR) 6 (2–16) 4 (2–12) .65 Esophagectomy, median (IQR) 14 (8–29) 15 (5–23) .32 Colectomy, median (IQR) 79 (66–106) 83 (50–116) .76 Gastrectomy, median (IQR) 8 (5–11) 9 (5–12) .99 Pulmonary resection, median (IQR) 14 (7–22) 15 (5–24) .88

IQR, Interquartile range. esophagectomy, pancreatectomy, hepatectomy, col- hospitals compliant with the b-blocker process ectomy, gastrectomy, and pulmonary lobectomy measure. No significant differences were found cases performed at hospitals that adopted the Leap- in age, sex, race, insurance status, or Charlson Co- frog b-blocker standard versus those hospitals that morbidity Index Score variables among patients did not adopt this standard (Table II). who underwent open AAA repair at either case Patient characteristics. Regarding patient-level or control hospitals (Table III). In comparison, volume, a total of 2,021 (33%) patients underwent the patient populations that underwent other open AAA repair, 893 (32%) underwent esopha- high-risk operations in the same hospitals were gectomy, 1,175 (40%) underwent pancreatectomy, found to be more heterogeneous when the same 1,075 (42%) underwent hepatectomy, 18,065 patient-level variables were compared. (31%) underwent colectomy, 1,993 (32%) under- In-hospital mortality. The average in-hospital went gastrectomy, and 3,350 (33%) underwent death rates per year were calculated across the pulmonary lobectomy at hospitals adopting the entire 6-year time period (2000--2005). Each of the Leapfrog b-blocker standard between 2000 and 7 operative procedures were associated with a high- 2005. In comparison, 4,178 (67%) open AAA risk of peri-operative mortality (open AAA repair: repairs, 1,887 (68%) esophagectomies, 1,734 4.2%; pancreatectomy: 5.1%; hepatectomy: 3.6%; (60%) pancreatectomies, 1,469 (58%) hepatecto- esophagectomy: 6.0%; colectomy: 4.3%; gastrec- mies, 39,730 (69%) colectomies, 4,274 (68%) tomy: 6.3%; and pulmonary lobectomy: 2.7%). gastrectomies, and 6,860 (67%) pulmonary After stratifying hospitals by whether hospitals lobectomies were performed during the same had adopted the Leapfrog b-blocker standard, time period at control hospitals that failed to baseline in-hospital mortality rates for each proce- implement the b-blockade process measure. The dure were calculated for the time period prior to characteristics of patients that underwent each of the time this process measure was promoted (2000 the 7 high-risk procedures are shown in Table III, to 2002). No significant differences were found in stratified by whether cases were performed in baseline crude mean mortality rates for hospitals Surgery Brooke et al 485 Volume 147, Number 4

Table III. Characteristics of patients who underwent high-risk operations in California hospitals stratified by compliance with b-blocker process measure Open AAA Pancreatectomy Esophagectomy Hepatectomy Colectomy Gastrectomy Lobectomy b: b: b: b: b: b: b: n = 2,022; n = 1,175; n = 893; n = 1,075; n = 18,065; n = 1,993; n = 3,656; control: control: control: control: control: control: control: Variable n = 4,182 n = 1,734 n = 1,887 n = 1,469 n = 39,730 n = 4,274 n = 7,404 Male sex: P value NS NS NS NS NS .05 NS b-blocker, number (%) 1,587 (79) 562 (48) 645 (72) 529 (49) 8,557 (47) 1,096 (55) 1,807 (49) Control, number (%) 3,279 (78) 841 (49) 1,322 (70) 707 (48) 18,391 (46) 2,211 (52) 3,592 (49) Age, P value NS NS NS .05 NS .05 .05 b-blocker, mean (± SD) 73.3 (± 7.7) 61.9 (± 15.3) 64.8 (± 14.7) 53.8 (± 20.4) 67.1 (± 16.4) 67.8 (± 14.0) 65.9 (± 15.8) Control, mean (± SD) 73.3 (± 7.8) 62.6 (± 13.7) 65.9 (± 12.5) 58.5 (± 15.9) 68.0 (± 15.2) 66.5 (± 14.5) 68.0 (± 12.5) Race: P value NS .05 NS NS .05 .05 .05 Caucasian b-blocker, number (%) 1,852 (92) 779 (66) 657 (74) 737 (69) 13,870 (77) 1,192 (60) 2,970 (81) Control, number (%) 3,830 (92) 1,269 (73) 1,405 (74) 1,021 (70) 32, 054 (81) 2,777 (65) 6,239 (84) Black b-blocker, number (%) 51 (3) 111 (9) 29 (3) 35 (3) 1,037 (6) 152 (8) 198 (6) Control, number (%) 109 (3) 113 (7) 80 (4) 54 (3) 2,296 (6) 289 (7) 318 (4) Latino b-blocker, number (%) 103 (5) 180 (15) 135 (15) 134 (12) 1,564 (9) 414 (21) 276 (8) Control, number (%) 180 (4) 228 (13) 242 (13) 194 (13) 4,954 (12) 620 (15) 399 (5) Other b-blocker, number (%) 16 (1) 105 (9) 72 (8) 169 (16) 1,594 (8) 235 (11) 212 (5) Control, number (%) 63 (2) 124 (7) 160 (8) 200 (14) 426 (1) 588 (13) 448 (6) Insurance: P value NS .05 NS .05 NS .05 .05 Medicare b-blocker, number (%) 1,587 (79) 510 (43) 450 (50) 331 (31) 9,403 (52) 1,055 (53) 1,952 (53) Control, number (%) 3,158 (76) 779 (45) 915 (49) 459 (31) 21,171 (53) 2,242 (52) 4,236 (57) Medicaid b-blocker, number (%) 38 (2) 53 (5) 49 (5) 77 (7) 831 (5) 150 (8) 178 (5) Control, number (%) 100 (3) 137 (8) 100 (5) 134 (9) 1,970 (5) 378 (9) 375 (5) Private b-blocker, number (%) 369 (18) 568 (48) 364 (41) 631 (59) 7,245 (40) 732 (37) 1,451 (40) Control, number (%) 864 (21) 748 (43) 795 (42) 752 (51) 15,184 (39) 1,480 (35) 2,585 (35) Self-pay b-blocker, number (%) 28 (1) 44 (4) 30 (4) 36 (3) 590 (3) 56 (3) 74 (2) Control, number (%) 60 (1) 69 (4) 77 (4) 124 (8) 1,401 (3) 174 (4) 207 (3) Charlson index: P value NS NS .05 .05 NS .05 .05 #1 b-blocker, number (%) 822 (41) 140 (12) 83 (9) 211 (20) 7,787 (43) 355 (18) 401 (12) Control, number (%) 1,686 (40) 187 (11) 234 (12) 272 (19) 17,593 (44) 896 (21) 844 (12) 2 b-blocker, number (%) 678 (34) 125 (11) 204 (23) 112 (10) 3,397 (19) 385 (19) 954 (28) Control, number (%) 1,390 (33) 149 (9) 474 (25) 108 (7) 7,073 (18) 798 (19) 1,754 (26) 3 b-blocker, number (%) 276 (14) 187 (16) 101 (11) 138 (13) 1,665 (9) 214 (11) 826 (25) Control, number (%) 588 (14) 302 (17) 203 (11) 217 (15) 3,643 (9) 509 (12) 1,822 (27) $4 b-blocker, number (%) 246 (12) 723 (62) 505 (57) 614 (57) 5,216 (29) 1,039 (52) 1,169 (35) Control, number (%) 518 (12) 1,096 (63) 976 (52) 872 (59) 11,421 (29) 2,071 (48) 2,440 (35) b-blocker, Hospitals compliant with Leapfrog b-blocker standard. that complied with b-blocker process measure P = .84), or pulmonary lobectomy (2.78 vs 2.87, compared with control hospitals for patients who respectively; P = .859). However, hospitals that underwent AAA repair (4.35 vs 3.90, respectively; complied with the b-blocker process measure P = .52), pancreatectomy (4.51 vs 5.79, respectively; were found to have lower baseline unadjusted P = .30), esophagectomy (6.33 vs 6.61, respectively; in-hospital mortality rates for patients who 486 Brooke et al Surgery April 2010

Fig 2. Unadjusted mean in-hospital death rates for seven high-risk operations performed in California hospitals over 6- year time period (2000--2005). High-risk operations that were assessed including open AAA repair (A), esophagectomy (B), pancreatectomy (C), hepatectomy (D), colectomy (E), gastrectomy (F), and pulmonary lobectomy (G). Hospitals stratified by whether they reported compliance (dark circles) versus noncompliance (white circles) with the b-blocker pro- cess measure during elective AAA repair on Leapfrog surveys after 2003. Error bars represent ± standard error of the mean (SEM). underwent hepatectomy (2.38 vs 5.30, respectively; Temporal trends in operative mortality were P < .05), colectomy (3.77 vs 4.61, respectively; evaluated in both hospital groups for each of these P < .05), and gastrectomy (4.70 vs 8.01, respec- 7 high-risk procedures during consecutive time tively; P < .05). periods (Fig 2). As shown in Fig 2, A, no significant Surgery Brooke et al 487 Volume 147, Number 4 difference in mortality rates was found for open Table IV. Poisson regression rate ratio estimates of AAA repair among hospitals for the 3-year time pe- in-hospital mortality for high-risk operations riod (2000--2002) prior to release of the Leapfrog performed at California hospitals stratified by b-blocker standard in 2003. A decrease in open whether routine b-blocker use was implemented AAA mortality was observed, however, in the pe- after 2003 riod between 2004 and 2005 among hospitals Ratio of rate P that implemented the b-blocker process measure. Hospitals ratios (95% CI) value In contrast, hospitals that failed to adopt the b-blocker standard did not exhibit a decrease in Hospitals that 1.00 (reference) did not adopt AAA peri-operative mortality. routine b-blocker use A more attenuated trend toward decreasing in- Hospitals that hospital mortality over time was observed in insti- adopted routine tutions compliant with b-blocker use for other b-blocker use in 2003 high-risk procedures, including esophagectomy AAA repair (Fig 2, B), pancreatectomy (Fig 2, C), and gastrec- Fixed effects unadjusted 0.67 (0.40–1.12) .129 tomy (Fig 2, F), when compared with unadjusted Fixed effects adjusted 0.42 (0.20–0.92) .030 data from control hospitals during the same time Random effects unadjusted 0.72 (0.42–1.25) .153 periods. However, no differences in mean unad- Random effects adjusted 0.49 (0.24–0.97) .042 justed in-hospital death rates over time were found Hepatectomy for patients undergoing hepatectomy (Fig 2, D), Fixed effects unadjusted 1.03 (0.87–1.22) .71 Fixed effects adjusted 1.01 (0.71–1.44) .95 colectomy (Fig 2, E), or lobectomy (Fig 2, G) Random effects unadjusted 1.13 (0.40–3.78) .13 when hospital groups were compared during the Random effects adjusted 1.16 (0.32–4.29) .36 same time periods. Esophagectomy To explore the temporal effect of hospital Fixed effects unadjusted 0.95 (0.83–1.09) .47 adoption of Leapfrog’s b-blocker process measure Fixed effects adjusted 0.91 (0.75–1.11) .37 on in-hospital mortality for each high-risk opera- Random effects unadjusted 0.75 (0.38–1.49) .42 tion, Poisson regression models were used to fit Random effects adjusted 0.70 (0.25–1.89) .47 both fixed and random intercepts to estimate Pancreatectomy adjusted mortality rates among individual hospitals Fixed effects unadjusted 1.13 (0.92–1.38) .25 over time. Using both fixed-effects estimates (rel- Fixed effects adjusted 1.10 (0.79–1.52) .59 ative risk [RR]: 0.40; 95% confidence interval [CI]: Random effects unadjusted 1.04 (0.53–2.06) .90 Random effects adjusted 0.76 (0.28–2.02) .58 0.20--0.92; P = .30) and random-effects estimates Colectomy (RR: 0.49; 95%CI: 0.24--0.97; P = 0.42) to adjust Fixed effects unadjusted 1.32 (1.11–1.57) <.05 for hospital- and patient-level confounders as well Fixed effects adjusted 1.11 (0.85–1.46) .43 as clustering, a greater than 50% reduction oc- Random effects unadjusted 1.32 (1.12–1.58) <.05 curred in mortality after open AAA repair in hospi- Random effects adjusted 1.12 (0.86–1.44) .40 tals compliant with peri-operative b-blocker usage Gastrectomy as compared with control hospitals. In contrast, Fixed effects unadjusted 1.11 (0.69–1.79) .66 no significant differences were found in crude or Fixed effects adjusted 1.13 (0.50–2.53) .77 adjusted in-hospital mortality rate ratio estimates Random effects unadjusted 1.21 (0.76–1.93) .43 among the other 6 high-risk operations indepen- Random effects adjusted 1.17 (0.57–2.43) .66 dent of whether the hospital had adopted the Lobectomy Fixed effects unadjusted 1.08 (0.68–1.71) .76 Leapfrog peri-operative b-blocker standard process Fixed effects adjusted 0.93 (0.42–2.04) .85 measure (Table IV). Random effects unadjusted 1.08 (0.68–1.71) .75 Random effects adjusted 0.98 (0.46–2.08) .96 DISCUSSION The possibility of suffering a life-threatening complication is a risk faced by 30 million patients outcomes.5,12 Previously, we reported that Califor- undergoing major operative procedures in the nia hospitals that adopted routine peri-operative United States each year.1 One strategy that is being b-blocker use in patients who underwent open promoted heavily to reduce peri-operative morbid- AAA repair had a significant reduction in hospital ity and mortality involves the selective referral of mortality compared with control hospitals.6 It re- operative patients to hospitals that comply with ev- mained unclear, however, whether the effect of idence-based standards, including process mea- b-blocker use was treatment specific or whether sures believed to be associated with improved some of the beneficial effect was attributable to 488 Brooke et al Surgery April 2010 institution-wide quality improvements reflected by use on changing operative outcomes relative to con- the adoption of evidence-based process measures. trol hospitals, and thereby it separated the effect of As such, we sought to investigate in the current this process measure from its role as a proxy study whether the implementation of b-blocker measure of baseline hospital quality. use for AAA repair was associated with institu- Several studies have argued that the hospital- tion-wide quality improvements for other high- level outcomes for 1 high-risk operation may be risk operative procedures over time. In fact, we predictive of outcomes for other complex proce- found the same level of improvement among ad- dures.10,19 Such data suggest that certain hospitals justed in-hospital mortality rates for other high- have a higher baseline level of quality that may be risk operations (eg, esophagectomy, hepatectomy, generalized across other areas of care throughout pancreatectomy, colectomy, gastrectomy, and lo- that institution. This concept is based on the belief bectomy) performed in the same California hospi- that the shared elements of structures and pro- tals regardless of whether the hospital had adopted cesses of care across certain individual hospitals the b-blocker process measure. These results are may relate to overall operative performance. In- important as our data strongly suggest that im- deed, hospital-level quality standards may explain provements in mortality outcomes over time result- in part our finding that baseline unadjusted mor- ing from hospital compliance with this process tality rates for several high-risk operations were sig- measure are procedure specific and do not neces- nificantly lower in hospitals that subsequently sarily reflect advances in overall hospital quality. adopted b-blocker use compared with control hos- A wide spectrum of hospital-level factors has pitals. These data may also imply that hospitals will- been identified that may influence the quality of ing to adopt Leapfrog’s recommendations also patient care and can help explain variations in participated in other operative quality initiatives. operative mortality. Such variables include the size Regardless of baseline mortality rates, however, or location of an institution, the status as a teaching the affect of hospital compliance with the facility or tertiary referral center, and hospital-level b-blocker process measure was limited to patients and individual surgeon-level case volume for the who underwent open AAA repair after controlling specific case in question.12-15 Avedis Donabedian for patient and hospital confounders and did not proposed a model for assessing health care quality significantly effect other operations. Consistent based on categorizing these different factors into with our findings, the recent POISE trial demon- structures, processes, and outcomes.16 This model strated that vascular surgery patients benefited is invoked commonly in the literature to describe the most from the peri-operative use of b-blockers how different variables relate and how each can compared with patients who underwent other ma- act independently to influence operative out- jor intra-abdominal procedures.20 Hence, these comes.17,18 Although the Donabedian model is data suggest that inferences about the quality of useful as a conceptual framework, it fails to discrim- hospital care derived from compliance with proce- inate the relative influence of each of these varia- dure-specific process measures should not be bles and the degree to which confounding or generalized to other patient types. interactions among the variables may occur. For There has been increasing interest in identify- example, it is difficult to distinguish the effective- ing which factors determine whether a hospital is ness of a process measure such as peri-operative willing to adopt evidence-based process measures. b-blocker use on operative outcomes from other in- Such information is important as recent studies trinsic institutional-level attributes such as hospital have shown that overall compliance with process size, case volume, or intensive care unit physician measures known to reduce morbidity and mortal- staffing. In the current study, we addressed these is- ity remains relatively low.21 A common perception sues by using a case-control design that controlled is that only large hospitals or hospitals that are for both hospital size and the number of annual part of large health care networks have the infra- high-risk procedures performed in each institution. structure and resources to comply with evidence- We also employed random-effects longitudinal based process measures. This theory is supported, models to give a more conservative estimate of the in part, by evidence from a recent study showing association between compliance with the b-blocker high-volume hospitals were more apt to comply process measure and the variation in mortality out- with evidence-based process measures aimed at pa- comes within individual hospitals over time. Fur- tients with cardiovascular disease.22 In our analysis thermore, the current study is informative in that of urban California hospitals, however, the adop- the analyses estimated the adjusted temporal effect tion of b-blocker use was undertaken primarily by associated with adopting peri-operative b-blocker hospitals that would not be considered high- Surgery Brooke et al 489 Volume 147, Number 4 volume centers for AAA repair. Furthermore, hos- well captured by these types of administrative data pitals that complied with b-blocker use did not sets and have been found to be reliable.25 The var- have significantly higher case volumes for AAA re- iation in mortality rates among hospitals across time pair or any of the other 6 high-risk operations may also have been influenced by unmeasured fac- when compared with control hospitals. In an era tors not controlled for in this study. Such factors in- where patient referral is increasingly driven by clude any changes in peri-operative management pay-for-performance and operative outcomes, that are found more commonly in hospitals that low-volume hospitals may actually have more in- adopted the b-blocker policy than control hospitals, centive to implement evidence-based process mea- which would have acted to inflate or confound the sures to compete with higher volume medical treatment effect attributed to adopting this EBM centers. Furthermore, it may be easier to imple- standard. Similarly, it was unknown whether case ment evidence-based practices in smaller, less com- or control hospitals employed b-blockers or any plex hospitals. other evidence-based process measures during the One question that faces all health care organi- peri-operative period for any of the high risk oper- zations---regardless of size---is deciding what spe- ations beyond open AAA repair. It is unlikely, how- cific process measures should be implemented to ever, that the results of this study were differentially achieve the best operative outcomes. Many have biased by unmeasured variables for all of the oper- relied on pay-for-performance programs and qual- ative procedures analyzed. Finally, data were not ity improvement initiatives such as the Leapfrog available to validate the administration of b-blockers Group, Surgical Care Improvement Project or compliance with the Leapfrog policy for their (SCIP), and the Centers for Medicare & Medicaid peri-operative usage. Services Hospital Quality Initiative to identify evi- In conclusion, hospitals can improve operative dence-based process measures.4, 23,24 The SCIP ini- outcomes through compliance with evidence- tiative, in particular, has promoted a broad range based process measures directed at specific proce- of prophylactic peri-operative measures including dures such as the peri-operative use of b-blockers those to prevent routine deep vein thrombosis, sur- in patients who undergo AAA repair. Compliance gical site infections, postoperative pneumonia, and with this process measure, however, is not a surro- the use of b-blockers in high-risk patients who un- gate marker for overall hospital quality and does derwent noncardiac surgery to prevent adverse car- not seem to predict improved outcomes for other diac events. These peri-operative process measures high-risk operations performed in the same insti- can generally be standardized across patients who tution. These data support ongoing efforts by undergo a range of operative procedures. By com- quality initiatives such as Leapfrog to define and parison, the Leapfrog Group approach has been to promote evidence-based process measures for spe- promote compliance with process measures tar- cific procedures among hospitals of all sizes. geted at patients who undergo a few select high- risk procedures. Although few process measures REFERENCES have yet to be defined for specific operations to 1. Merrill CT, Elixhauser A. Procedures in U.S. hospitals, date, this is likely to change in the future as 17 2003. HCUP fact book no. 7. AHRQ Publication No. 06- more evidence becomes available. As shown in 0039. Rockville, MD: Agency for Healthcare Research and the current study, broad hospital-level process mea- Quality; 2006. Available from: http://www.ahrq.gov/data/ sures are inadequate in predicting quality out- hcup/factbk7/. comes in other unrelated areas of care. Our 2. Kohn LTCJ, Donaldson MS. To err is human: building a safer health system institute of medicine. Washington, DC: findings emphasize that it will be critical to expand National Academy of Sciences; 2000. the list of both general and specific process mea- 3. McClellan MB, McGinnis JM, Nabel EG, Olsen LM. Evi- sures to reflect the actual operative procedures dence-based medicine and the changing nature of health- that a given patient is undergoing. care: IOM roundtable on evidence-based medicine. Our longitudinal study has several limitations. Washington, DC: Institute of Medicine; 2008. 4. The Leapfrog Group for Patient Safety. Available from: First, only 63% of California hospitals responded to www.leapfroggroup.org. Leapfrog surveys during the study time period, 5. Birkmeyer JD, Dimick JB. Potential benefits of the new which limits the representability of results because Leapfrog standards: effect of process and outcomes mea- of an increased potential for selection bias. More- sures. Surgery 2004;135:569-75. over, the outcomes obtained from the retrospective 6. Brooke BS, Perler BA, Dominici F, Makary MA, Pronovost PJ. Reduction of in-hospital mortality among California analysis of administrative data such as the California hospitals meeting Leapfrog evidence-based standards for OSHPD may also be prone to bias. However, our abdominal aortic aneurysm repair. J Vasc Surg 2008;47: primary endpoint---in-hospital deaths---are generally 1155-6. 490 Brooke et al Surgery April 2010

7. Berwick DM. Disseminating innovations in health care. 17. Birkmeyer JD, Dimick JB. Understanding and reducing JAMA 2003;289:1969-75. variation in surgical mortality. Annu Rev Med 2009;60: 8. Glasgow RE, Lichtenstein E, Marcus AC. Why don’t we see 405-15. more translation of health promotion research to practice? 18. Birkmeyer JD, Dimick JB, Birkmeyer NJ. Measuring the Rethinking the efficacy-to-effectiveness transition. Am J quality of surgical care: structure, process, or outcomes? J Public Health 2003;93:1261-7. Am Coll Surg 2004;198:626-32. 9. Glasgow RE, Emmons KM. How can we increase translation 19. Urbach DR, Baxter NN. Does it matter what a hospital is of research into practice? Types of evidence needed. Annu ‘‘high volume’’ for? Specificity of hospital volume-outcome Rev Public Health 2007;28:413-33. associations for surgical procedures: analysis of administra- 10. Dimick JB, Staiger DO, Birkmeyer JD. Are mortality rates for tive data. BMJ 2004;328:737-40. different operations related?: implications for measuring 20. Devereaux PJ, Yang H, Yusuf S, Guyatt G, Leslie K, Villar JC, the quality of noncardiac surgery. Med Care 2006;44:774-8. et al. Effects of extended-release metoprolol succinate in 11. Diggle P, Heagerty P, Liang K-Y, Zeger SL. Analysis of longi- patients undergoing non-cardiac surgery (POISE trial): a tudinal data. Oxford (UK): Oxford University Press; 2002. randomised controlled trial. Lancet 2008;371:1839-47. p. 169-89. 21. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, De- 12. Birkmeyer NJ, Birkmeyer JD. Strategies for improving surgi- Cristofaro A, et al. The quality of health care delivered cal quality---should payers reward excellence or effort? N to adults in the United States. N Engl J Med 2003;348: Engl J Med 2006;354:864-70. 2635-45. 13. Birkmeyer JD, Stukel TA, Siewers AE, Goodney PP, Wenn- 22. Williams SC, Koss RG, Morton DJ, Schmaltz SP, Loeb JM. berg DE, Lucas FL. Surgeon volume and operative mortality Case volume and hospital compliance with evidence-based in the United States. N Engl J Med 2003;349:2117-27. processes of care. Int J Qual Health Care 2008;20:79-87. 14. Birkmeyer JD, Siewers AE, Finlayson EV, Stukel TA, Lucas 23. Centers for Medicare and Medicaid Services. Process of care FL, Batista I, et al. Hospital volume and surgical mortality measures. Available from: www.cms.hhs.gov/Hospital in the United States. N Engl J Med 2002;346:1128-37. QualityInits/01_Overview.asp#TopOfPage. 15. Meguid RA, Brooke BS, Chang DC, Sherwood JT, Brock MV, 24. Surgical Care Improvement Project Core Measure Set. Yang SC. Are surgical outcomes for lung cancer resections Available from: www.jointcommission.org/Performance improved at teaching hospitals? Ann Thorac Surg 2008;85: Measurement/PerformanceMeasurement/. 1015-25. 25. Nathan H, Pawlik TM. Limitations of claims and registry 16. Donabedian A. The quality of care. How can it be assessed? data in surgical oncology research. Ann Surg Oncol 2008; JAMA 1988;260:1743-8. 15:415-23.