Users' Guides to the Medical Literature

THE MEDICAL LITERATURE

Users’ Guides to the Medical Literature XVIII. How to Use an Article Evaluating the Clinical Impact of a Computer-Based Clinical Decision Support System

Adrienne G. Randolph, MD, MSc The chair of medicine keeps prom- Randomized Trial of ‘Corollary Or- R. Brian Haynes, MD, PhD ising to install computers to help man- ders’ to Prevent Errors of Omission.” age all of this information, but she is The abstract of this article concludes Jeremy C. Wyatt, MD limited by the budget squeeze. She that “physician work stations, linked to Deborah J. Cook, MD, MSc needs proof that computerization will a comprehensive electronic medical Gordon H. Guyatt, MD, MSc improve patient care to justify such a record, can be an efficient means for major expense. She asks you to help. decreasing errors of omissions and CLINICAL SCENARIO You remember reading, in one of the improving adherence to practice guide- 1 It is 7 AM, and medical rounds are many journals piled up at home, about lines.” starting on university hospital ward how computers can be used to pro- You order the full article over the In- 1 3B. In the past 24 hours of your resi- vide decision support leading to im- ternet from Loansome Doc. In this study dency, you have transferred 2 criti- proved patient outcomes. If you can conducted on the inpatient general medi- cally ill patients to the intensive care show that computers improve patient cal wards of an inner-city public hospi- unit; accepted 11 patients to your care, maybe the hospital administra- tal, 6 independent services (red service, medical service; examined and revised tion will see the expense as an invest- green service, etc) cared for the inpa- medication orders for 22 patients; ment that could reduce costs. tients. Each service included a faculty in- placed 9 intravascular catheters; writ- ternist, a senior resident, and 2 interns. ten 35 notes; and reviewed, catego- THE SEARCH A different physician team rotated onto rized, and acted on more than 300 When you get home that night, you each service every 6 weeks, and during new pieces of laboratory and radiol- connect to the Internet and decide to a year, 8 different teams worked on each ogy data. You were planning to ask search the medical literature using In- service. At the beginning of the study, the the infectious disease specialist about ternet Grateful Med from the US Na- investigators randomly allocated 3 of the a patient, but he seems very busy, and tional Library of Medicine. You type 6 services to the intervention group, the broad-spectrum antibiotic regi- http://igm.nlm.nih.gov/ into your browser men you prescribed should suffice. and choose MEDLINE. You quickly re- Author Affiliations: Departments of Pediatrics and An- You were just told that you ordered alize that you don’t know what search esthesia, Children’s Hospital and Harvard Medical School, Boston, Mass (Dr Randolph); Departments of total parenteral nutrition for the terms to use. You enter decision then Clinical Epidemiology and Biostatistics and Medicine, wrong patient. While deciding which click the button for Find MeSH/Meta McMaster University, Hamilton, Ontario (Drs Haynes, Cook, and Guyatt); and School of Public Policy, Uni- patient should receive parenteral Terms. From the 31 Medical Subject versity College London, London, England (Dr Wyatt). nutrition, you realize that the calcula- Headings terms offered, you choose de- The original list of members (with affiliations) appears in the first article of this series (JAMA. 1993; tions for the amino acid concentration cision making, computer-assisted; 270:2093-2095). A list of new members appears in are erroneous. After the first 5 min- therapy, computer assisted; diagnosis, the 10th article of the series (JAMA. 1996;275:1435- utes of your first patient presentation, computer-assisted; drug therapy, com- 1439). The following members contributed to this article: Anne Holbrook, MD, PharmD, MSc; Virginia the senior physician asks you details puter-assisted, specifying that they are Moyer, MD, MPH; W. Scott Richardson, MD; David from the patient’s past medical his- the major topics of the article. You limit L. Sackett, MD, MSc. Corresponding Author and Reprints: Gordon H. Guy- tory. You wish you could refer to your your search to randomized controlled att, MD, MSc, McMaster University Health Sciences admission note, but you couldn’t trials in English during the years 1995 Centre, 1200 Main St W, Room 2C12, Hamilton, On- tario, Canada L8N 3Z5. access it before your rounds because a to 1998. Browsing through the 45 ab- Users’ Guides to the Medical Literature Section Editor: utilization review clerk had the chart. stracts from the search, you choose “A Drummond Rennie, MD, Deputy Editor (West), JAMA.

©1999 American Medical Association. All rights reserved. JAMA, July 7, 1999—Vol 281, No. 1 67 USERS’ GUIDES TO THE MEDICAL LITERATURE which had access to a computer-based ibility, display, and accessibility of in- or patterns that might require action on clinical decision support system (CDSS); formation in the patient’s chart may the part of the care provider.4 A simple the other 3 services served as controls and produce benefits that can sometimes be example of an alert is the starred (*) or did not have access to a CDSS. Teams related to the care of an individual pa- highlighted item (with H or L mark- were randomly assigned to the interven- tient. However, medical literature da- ing or with BOLD or changed colors on tion and control services. The CDSS re- tabases and ordinary patient charting the screen) that alerts the clinician to sponded to trigger orders by suggesting systems do not filter and abstract in- values that are out of range on com- corollary orders needed to detect or ame- formation from detailed clinical data. puterized laboratory printouts and dis- liorate adverse reactions and allowed We use the term CDSS to describe soft- play screens. Alerting systems draw at- physicians to accept or reject these sug- ware designed to directly aid in clini- tention to events as they occur. gestions. TABLE 1 shows examples of cor- cal decision making about individual Reminder systems notify clinicians of ollary orders and their trigger orders. patients. Specifically, detailed indi- important tasks that need to be done vidual patient data are input into a com- before an event occurs. An outpatient CLINICAL COMPUTER puter program that sorts and matches clinic reminder system may generate a SYSTEMS them using programs or algorithms in list of immunizations that each pa- Clinicians who manage the care of pa- a knowledge base, resulting in the gen- tient on the daily schedule requires. Al- tients are dependent on computers. eration of patient-specific assess- though the technical rules that gener- Laboratory data management soft- ments or recommendations for clini- ate alerts and reminders are often ware, pharmacy information manage- cians.2 TABLE 2 shows functions of simple, alerting the right person in a ment systems, applications for track- decision support systems developed for timely fashion is quite complex. ing patient location through admission the following medical purposes: alert- When the clinician has made a de- and discharge, mechanical ventila- ing, reminding, critiquing, interpret- cision and the computer evaluates that tors, and oxygen saturation measure- ing, predicting, diagnosing, assisting, decision and generates an appropriate- ment devices are among the many types and suggesting.3 ness rating or alternative suggestion, the of computerized systems that have be- Many alerting, reminding, and cri- decision support approach is called cri- come an integral part of the modern tiquing systems are based on simple tiquing. The distinction between as- hospital. These devices and systems if-then rules that tell the computer what sisting and critiquing decision sup- capture, transform, display, or ana- to do when a certain event occurs. Alert- port programs is that assisting programs lyze data for use in clinical decision ing systems monitor a continuous sig- help formulate the clinical decision, making. Using computers to search the nal or stream of data and generate a whereas critiquing programs have no medical literature or to improve the leg- message (an alert) in response to items part in suggesting the order or plan but evaluate the plan, after it is entered, against an algorithm in the com- Table 1. Example Trigger and Corollary Orders puter.3 Critiquing systems are com- Trigger Orders Corollary Orders monly applied to physician order en- Heparin infusion Platelet count once before heparin starts, then every 24 h try. For example, a clinician entering Activated partial thromboplastin time at start, again 6 h after a dosage change an order for a blood transfusion may re- Prothrombin time once before heparin started ceive a message stating that the pa- Hemoglobin at start of therapy, then every morning Test stools for occult blood while administering heparin Table 2. Functions of Computer-Based Intravenous fluids Place a saline lock when intravenous fluids are discontinued Clinical Decision Support Systems Narcotics (class II) Docusate if not taking any other stool softener or laxative Function Example Nonsteroidals Creatinine level (if not 1 in previous 10 d); SMA-12,* blood urea nitrogen counted as equivalent Alerting Highlighting out-of-range laboratory values Aminoglycosides Peak and trough levels after dosage changes and every week Reminding Reminding the clinician to Creatinine level twice per week (every Monday and Thursday) schedule a mammogram Critiquing Rejecting an electronic order Warfarin sodium Prothrombin time each morning Interpreting Interpreting the Amphotericin B Creatinine level twice per week (every Monday and Thursday) electrocardiogram Magnesium level (twice per week while receiving therapy) Predicting Predicting risk of mortality from a severity-of-illness score Electrolytes (twice per week while receiving therapy) Diagnosing Listing a differential diagnosis for Acetaminophen (650 mg by mouth 30 min before each dose) a patient with chest pain Assisting Tailoring the antibiotic choices Diphenhydramine hydrochloride (50 mg 30 min before each amphotericin for liver transplantation and dose) renal failure *SMA-12 indicates sequential multiple analyzer, measuring glucose, blood urea nitrogen, uric acid, calcium, phospho- Suggesting Generating suggestions for rus, total protein, albumin, cholesterol, total bilirubin, alkaline phosphatase, serum glutamic oxaloacetic transami- adjusting the mechanical nase, and lactate dehydrogenase. ventilator

68 JAMA, July 7, 1999—Vol 281, No. 1 ©1999 American Medical Association. All rights reserved. USERS’ GUIDES TO THE MEDICAL LITERATURE tient’s hemoglobin level is above the account the patient’s renal function, Table 3. Using Articles Describing transfusion threshold, and the clini- drug allergies, the site of infection, the Computer-Based Clinical Decision Support cian must justify the order by stating epidemiology of organisms in patients Systems (CDSSs) an indication, such as active bleed- with this infection at this hospital over Are the results of the study valid? ing.5 Getting the attention of the per- many years, the efficacy of the antibi- Was the method of participant allocation appropriate? son who can take action is one of the otic regimen, and the cost of therapy. Was the control group uninfluenced by the most difficult aspects of making alert- A system that instructs caregivers about CDSS? Aside from the CDSS, were the groups ing, reminding, and critiquing sys- how to manage the ventilation of pa- treated equally? tems effective. tients with adult respiratory distress What were the results? 11 What was the effect of the CDSS? The automated interpretations of syndrome is another example. Can you apply the computer-based CDSS in electrocardiogram readings6 and the The primary reason to invest in com- your clinical setting? outcome predictions generated by se- puter support is to improve quality of What elements of the CDSS are required? 7 verity-of-illness scoring systems are ex- care. If a computer system purports to Is the CDSS exportable to a new site? amples of decision support systems aid clinical decisions, enhance patient Is the CDSS likely to be accepted by clinicians in your setting? used for interpreting and predicting, re- care, and improve outcomes, then it Do the benefits of the CDSS justify the spectively. These systems filter and ab- should be subject to the same rules of risks and costs? stract detailed clinical data and gener- testing as any other health care inter- ate a report characterizing the meaning vention with similar claims. In this ar- of the data (eg, anterior myocardial in- ticle, we describe how to use articles low-up were explained. The purpose of farction).6 that evaluate the clinical impact of a our discussion in this article is to high- Computer-aided diagnostic sys- CDSS. While the focus of a CDSS may light issues of particular importance in tems assist the clinician with the pro- be restricted to diagnosis or progno- the evaluation of a CDSS. cess of differential diagnosis.8 When the sis, we will limit our discussion to the Was the Method of Participant Al- electrocardiogram results are not de- situation in which the CDSS is de- location Appropriate? The validity of finitive, computer systems that try to signed to change clinician behavior and the observational study designs often distinguish between myocardial infarc- patient outcome. Many iterative steps used to evaluate a CDSS is limited. The tion and other sources of chest pain can are involved in developing, evaluat- most common observational design is sometimes outperform a clinician.9 ing, and improving a CDSS before it can the before-after study design, in which These types of systems require perti- progress beyond the laboratory envi- investigators compare outcomes be- nent patient information, such as signs, ronment and pilot-testing phase and be fore a technology is implemented (us- symptoms, past medical history, labo- allowed to have a wider impact on phy- ing a historic control group) with those ratory values, and demographic char- sicians and patients. These evalua- after the system is implemented. The acteristics. The programs start gener- tions involve social science methods for validity of this approach is threatened ating hypotheses, often prompt the user evaluating human behavior and com- by the possibility that changes over time for more information, and ultimately puter science methods for evaluating (called secular trends) in patient mix provide a diagnosis or a list of possible technological safety and robustness.4 or in aspects of health care delivery may diagnoses ranked probabilistically. We limit our discussion to mature sys- result in changes in behavior that ap- Computerized patient management tems that have surpassed initial evalu- pear to be attributable to the CDSS. systems are complex programs that ation and are being implemented to Consider a CDSS that assisted physi- make suggestions about the optimal de- change physician behavior and pa- cians with antibiotic ordering10 in the cision based on the information cur- tient outcome. late 1980s and was associated with im- rently known by the system. These provements in the cost-effectiveness of types of systems are often integrated Are the Results antibiotic ordering over the next 5 years. into the physician ordering process. Af- of the Study Valid? Changes in the health care system, inter collecting information on specific When clinicians examine the effect of cluding the advent of managed care, patient variables, the assistant pro- a CDSS on patient management or out- were occurring simultaneously dur- gram tailors the order to the patient come, they should use the same crite- ing that time. To control for secular based on prior information in the da- ria appropriate for any other interven- trends, the computerized antibiotic tabase regarding appropriate dosages or tion (TABLE 3), whether it be a drug, a practice guideline study investiga- by implementing specified protocols. rehabilitation program, or an ap- tors10 compared antibiotic prescribing The Antibiotic Assistant10 is a CDSS that proach to diagnosis or screening.12 In practices with those of other nonfed- implements guidelines to assist physi- our Users’ Guide to prevention and eral US acute care hospitals for the du- cians with ordering antibiotics. This sys- therapy,13 the importance of random as- ration of the study. tem recommends the most cost- signment, blinding of patients and out- One type of time-series design, in effective antibiotic regimen taking into come assessors, and complete fol- which the intervention is turned on and

©1999 American Medical Association. All rights reserved. JAMA, July 7, 1999—Vol 281, No. 1 69 USERS’ GUIDES TO THE MEDICAL LITERATURE off multiple times, has been used to con- this study is only 2 (2 teams), the like- valid. However, although investiga- trol for potential secular trends. Al- lihood of imbalance despite random- tors did not randomly assign house staff though this provides some protection ization is very large. to services, they conducted their analy- against bias, random allocation of pa- When investigators randomize phy- sis at the individual house staff level, tients to a concurrent control group re- sicians and health care teams, obtain- comparing 45 intervention physicians mains the strongest study design for ing a sample of sufficient size can be with 41 control physicians. They took evaluating therapeutic or preventive in- difficult. If only a few health care teams no steps to ensure that the character- terventions.13 Use of historical con- are available, stratification of these teams istics of house staff on the interven- trols may lead to a higher tendency to according to important prognostic fac- tion and control teams were similar, see positive results. A comparison of the tors can reduce potential imbalances. If leaving the study open to biases from 2 types of studies used to evaluate the there are many known risk factors, in- baseline differences in house staff per- same antihypertensive drugs revealed vestigators can pair health care teams ac- formance. Moreover, the use of indi- that 80% of historically controlled stud- cording to their similarities and ran- vidual house staff instead of the team ies suggested that the new drugs were domly allocate the intervention within as the unit of analysis may have led to effective, whereas only 20% of random- each matched pair.20 In addition, inves- false precision in estimating the im- ized controlled trials confirmed this re- tigators can use statistical methods de- pact of the intervention because of a sult.14 Randomized controlled trials veloped specifically for analyzing stud- falsely inflated sample size. have been successfully used to evaluate ies using cluster randomization.21 In the study by Overhage et al,1 in- more than 70 CDSSs.2,15-17 There is one other issue regarding vestigators excluded 6 physicians from An important issue for CDSS evalu- randomization to which clinicians the intervention group because those ation is the unit of allocation. Investi- should attend. If some clinicians as- physicians received fewer than 5 sug- gators in clinical trials usually random- signed to CDSS fail to receive the in- gestions about corollary orders. This de- ize patients. When evaluating the effect tervention, should these clinicians be cision violates the intention-to-treat of a CDSS on patient care, the inter- included in the analysis? principle and risks introducing bias, be- vention is usually aimed at changing the The answer, counterintuitive to cause physicians on the control side who decision making of the clinician, so in- some, is yes. Randomization can ac- received fewer than 5 suggestions were vestigators may randomize individual complish the goal of balancing groups included. Fortunately, the small num- clinicians or clinician clusters such as with respect to both known and un- ber of excluded physicians were mostly health care teams, hospital wards, or known determinants of outcome only off-service physicians covering night outpatient practices.18 A common mis- if patients (or clinicians) are analyzed calls for 1 or 2 nights and not actually take made by investigators is to ana- in the groups to which they are ran- service team members, so the contribu- lyze their data as if they had random- domized. Deleting or moving patients tion of such physicians to the compari- ized patients rather than clinicians. This after randomization compromises or de- son of CDSS and control is small. is called a unit of analysis error.19 stroys the balance that randomization Was the Control Group Uninflu- To highlight the problem, we will use is designed to achieve. An analysis in enced by the CDSS? One problem with an extreme example. Investigators ran- which patients are included in the performing a controlled trial randomize study participants to ensure that groups to which they were random- domizing a CDSS across patients is the treatment and control groups are bal- ized, whether or not they received the difficulty in controlling for contam- anced with respect to important pre- intervention, is called intention to treat.13 ination of the control group by the in- dictors of outcome. Randomization of- In the study by Overhage et al,1 dur- tervention. Strickland and Hasson22 ran- ten fails to balance groups if sample size ing the course of a year, there were 36 domly allocated patients to have is small. Consider a study in which an teams randomly assigned to 18 CDSSs changes in their level of mechanical investigator randomizes one team of cli- and 18 control services. House staff ventilator support either directed by a nicians to a CDSS and another to stan- were required to write all orders and computer protocol and implemented dard practice. During the course of the were used as the unit of analysis. Each through a physician or directed by the study, each team sees 10 000 patients. service admitted patients in sequence, physician independently. Because the If the investigator analyzes the data as so that all 6 services received equal same physicians and respiratory thera- if patients were individually random- numbers of patients. A total of 86 house pists who used the computer protocol ized, the sample size appears huge (the staff physicians who each received more managed the care of patients not as- unit of analysis error19). However, it is than 5 corollary orders during the study signed to the protocol, it is possible cli- very plausible, perhaps even likely, that cared for 2181 different patients dur- nicians remembered and applied pro- the 2 teams’ performance differed at the ing 2955 different admissions. tocol algorithms in control patients. start and that this difference persisted Random assignment of teams to When the control group is influenced through the study independent of the CDSS and non-CDSS services in- by the intervention, the effect of the CDSS. Because the base sample size in creases our belief that the results are CDSS may be diluted. Contamination

70 JAMA, July 7, 1999—Vol 281, No. 1 ©1999 American Medical Association. All rights reserved. USERS’ GUIDES TO THE MEDICAL LITERATURE may spuriously decrease, or even elimi- different interpretations of marginal find- dents through their teaching. Faculty nate, a true intervention effect. ings or differential encouragement dur- could rotate with different house staff One method of preventing expo- ing performance tests.23 Blinding also di- on different rotations during the study, sure of the control group to the CDSS minishes the placebo effect,13 which, in further complicating this situation. To is to assign individual clinicians to use the case of CDSS, may be the tendency allow for this clustering of physicians or not use the CDSS. This is often prob- of patients or clinicians to ascribe posi- within teams, the investigators used lematic because of cross-coverage of pa- tive attributes to use of a computer work- generalized estimating equations to tients. Comparing the performance of station.4 Although blinding the clini- control for potential cointervention. wards or hospitals that do or do not use cians, patients, and study personnel to the CDSS is another possibility. Unfor- the presence of the computer-based What Are the Results? tunately, it usually is not feasible to en- CDSS may prevent this type of bias, What Is the Effect of the CDSS? A CDSS roll a sufficient number of hospitals in blinding is sometimes not possible. is often aimed at preventing adverse a study to avoid the problem we de- Interventions other than the treat- events or health outcomes or at improv- scribed earlier—when sample size is ment being studied that can influence ing compliance with a treatment regi- small, randomization may fail to en- the outcome are called cointerven- men. (See our Users’ Guide for preven- sure prognostically similar groups. tions. They frequently occur because tion or therapy13 for a discussion of In the study by Overhage et al,1 phy- most patients receive multiple thera- relative risk and relative risk reduc- sicians whose teams were assigned to pies aimed at improving their out- tions, risk differences and absolute risk a control service had the CDSS guide- come. A problem arises when cointer- reductions, and confidence intervals.) In lines available on paper but did not re- ventions are differentially applied to the the study by Overhage et al,1 interven- ceive assistance when ordering. To con- treatment and control groups. This situ- tion physicians ordered the corollary or- trol for the risk that cross-coverage of ation is more likely to arise in un- ders suggested by the CDSS much more patients could expose the control group blinded studies, particularly if the use frequently than control physicians spon- to the CDSS, the investigators had the of very effective nonstudy treatments taneously ordered them. This was true chief medical resident construct the is permitted at physicians’ discre- when measured by immediate compli- residents’ evening call schedule to sepa- tion.13 Clinicians’ concerns regarding ance (46.3% vs 21.9%; relative in- rate coverage for patients based on pa- lack of blinding are ameliorated if increase, 2.11; PϽ.0001), 24-hour com- tients’ study status. If switches in the vestigators describe permissible cointer- pliance (50.4% vs 29.0%; relative schedule were made, control physi- ventions and their differential use increase, 1.74; PϽ.0001), or hospital- cians provided call coverage only for and/or standardize cointerventions24 to stay compliance (55.9% vs 37.1%; rela- non-CDSS patients, and intervention ensure that their application is similar tive increase, 1.51; PϽ.0001). Because physicians covered only CDSS pain both treatment and control groups. the numerators and denominators are tients. Furthermore, to avoid contami- It is also important to ensure that the not reported for the total numbers of cor- nation that could occur if interven- evaluation of the outcome for each ollary orders complied with and not tion physicians cared for control group is not biased. In some studies, the complied with for each group, we can- patients, the computer suggested or- computer system may be used as a data not calculate the confidence intervals for ders only when the patient had been as- collection tool to evaluate the out- the risk difference for the increase in signed to a physician in the CDSS come in the CDSS group. The “data compliance. However, because the P val- group, and corollary order sugges- completeness bias” can occur when the ues are very small, we know that the tions were suppressed if the patient was information system is used to log epi- lower boundary of the confidence in- assigned to the control group. If phy- sodes in the treatment group and a terval is appreciably greater than 1, and sicians returned for a second rotation manual system is used to log episodes the confidence interval is therefore rela- and changed study status, the investi- in the non-CDSS group.4 Because the tively narrow. gators excluded data from their sec- computer may log more episodes than Length of stay and hospital charges ond rotation. All of these efforts were the manual system, it may appear that did not differ significantly. Pharma- to prevent contamination of the con- the CDSS group had more events, cists made 105 interventions with the trol group by the CDSS. which could bias the outcome in favor CDSS group of physicians and 156 with Aside From the CDSS, Were the of or against the CDSS group. To pre- control physicians (2-tailed P = .003) for Groups Treated Equally? The results vent this bias, outcomes should be errors considered to be life-threaten- of studies evaluating interventions aimed logged similarly in both groups. ing, severe, or significant. at therapy or prevention are more be- In the study by Overhage et al,1 al- lievable if patients, their caregivers, and though faculty were proscribed from Can You Apply the CDSS study personnel are blind to the treat- writing orders except during emergen- in Your Clinical Setting? ment.13 Unblinded study personnel who cies, physicians practiced within teams, What Elements of the CDSS Are Re- are measuring outcomes may provide and the faculty influenced the resi- quired? Investigators should specify the

©1999 American Medical Association. All rights reserved. JAMA, July 7, 1999—Vol 281, No. 1 71 USERS’ GUIDES TO THE MEDICAL LITERATURE intervention that they are evaluating. site, it has to be able to integrate with developed on the basis of potential us- Two of the major elements of a CDSS, existing software, users at the new site ers’ capabilities and limitations, the us- the logic and the computer interface must be able to maintain the system, and ers’ tasks, and the environment in which used to present the logic, could each be users must accept the system. Double- those tasks are performed.26 One of the evaluated as a separate intervention. charting occurs when systems require main difficulties with alerting systems However, sometimes it is not possible staff (usually nurses) to enter the data is notifying the individual with deci- to separate these 2 elements and achieve twice—into the computer and again on sion-making capability as rapidly as the same result. For example, we men- a flow sheet. Systems that require possible that there is an abnormal labo- tioned a randomized controlled trial double-charting increase staff time de- ratory value or other potential prob- that compared a computerized proto- voted to documentation, frustrate us- lem. A group of investigators tried a col for managing patients diagnosed as ers, and divert time that could be de- number of different alerting methods, having adult respiratory distress syn- voted to patient care. In general, such from a highlighted icon on the com- drome with standard clinical care us- systems fail in clinical use. puter screen to a flashing yellow light ing extracorporeal carbon dioxide Successful systems usually have au- placed on the top of the computer.27 removal as rescue therapy.11 The com- tomatic electronic interfaces to exist- These investigators later gave the nurses puterized protocol group without res- ing data-producing systems. Unfortu- pagers to alert them to abnormal labo- cue therapy did as well as the rescue nately, building interfaces to diverse ratory values.28 The nurses could then therapy group. Was this due to the logic computer systems is often challenging decide how to act on the information in the protocol, the use of the com- and sometimes impossible. and when to alert the physician. puter, or both interacting together? The program described in the study To ensure user acceptance, users To test whether the computer is by Overhage et al1 was implemented us- must feel that they can depend on the needed requires that one group apply ing the Regenstrief Medical Record Sys- system to be available whenever they the protocol logic as written on paper tem developed at Indiana University need it. The amount of downtime and the other group use the same logic School of Medicine. This system pro- needed for data backup, troubleshoot- implemented by the computer. Some- vides an electronic medical record sys- ing, and upgrading should be mini- times the protocol logic is so complex tem and a physician order entry system. mal. The response time must be fast, that use of a computer may be re- While it may be possible to use the data integrity must be maintained, and quired for implementation. knowledge built into the system in a data redundancy must be minimized. The CDSS may have a positive im- health care environment in which the If systems have been functioning at pact for unintended reasons. The im- patient population is similar, the infer- other sites for a period of time, major pact of structured data collection forms ence engine used to compare the rules problems or software bugs may have and performance evaluations (the with the order entered into the data- been eradicated, decreasing down- Checklist Effect and the Feedback Ef- base is not easily exported to other lo- time and improving acceptance. Inves- fect,4 respectively) on decision making cations. If, after critically appraising the tigators should also assess the amount can equal that of computer-generated ad- article, you are convinced that a CDSS of training required for users to feel vice.25 The CDSS intervention itself may for implementing guidelines would be comfortable with the system. If users be administered by research personnel useful, you would need sufficient re- become frustrated, system perfor- or by paid clinical staff who receive scant sources to rebuild the system at your mance will be suboptimal. mention in the published report but own site. Many computer programs may func- without whom the impact of the sys- Is the CDSS Likely to Be Accepted tion well at the site where the pro- tem is seriously undermined. by Clinicians in Your Setting? A CDSS gram was developed; unfortunately, the The CDSS in the study by Overhage may not be accepted if the clinicians dif- staff at your own institution may have et al of corollary orders1 and in the adult fer in important ways from those who objections to the approaches taken else- respiratory distress syndrome study11 participated in the study. The choice of where. For example, an expert system had 3 components: a knowledge base evaluative group may limit the gener- for managing ventilated patients who defining which corollary orders were re- alizability of the conclusions if recruit- have adult respiratory distress syn- quired for each trigger order, a data- ment is based upon enthusiasm, demo- drome may use continuous positive base that stored the trigger orders, and graphics, or a zest for new technology. airway pressure trials to wean patients an inference engine that compared the Clinicians in a new setting may be sur- off the ventilator, whereas clinicians at database with the knowledge base when prised when their colleagues do not use your institution may prefer pressure- a trigger order was received and sent a a CDSS with the same avidity as the support weaning. Syntax, laboratory list of suggested corollary orders to the original participants. coding and phrasing of diagnoses, and computer terminal for display. The user interface is an important therapeutic interventions can vary Is the CDSS Exportable to a New component of the effectiveness of a markedly among institutions. Custom- Site? For a CDSS to be exported to a new CDSS. The CDSS interface should be izing the application to the environ-

72 JAMA, July 7, 1999—Vol 281, No. 1 ©1999 American Medical Association. All rights reserved. USERS’ GUIDES TO THE MEDICAL LITERATURE ment may not be feasible, and addi- ware. Indeed, it can be very difficult to that preventive interventions improve tional expense may be invoked when estimate the costs of purchasing or patient health outcomes. mapping vocabulary to synonyms un- building and implementing an inte- Fortunately, evaluation of health care less a mechanism to do so is already grated CDSS. processes will adequately infer benefit programmed into the system. To en- Are CDSSs Beneficial? Human per- if the care processes being monitored sure user acceptance, the needs and formance may improve when partici- are already known to improve out- concerns of users should be consid- pants are aware that their behavior is comes.31 We could conclude that a ered, and users should be included in being observed (the Hawthorne ef- CDSS that increased the frequency with decision making and implementation fect30); the same behavior may not be which aspirin, ␤-blockers, and angio- stages. exhibited when the monitoring of out- tensin-converting enzyme inhibitors The logic in the Regenstrief Order comes has stopped. Taking into ac- were administered to appropriate Entry system1 was based on the exper- count the influence of a study environ- patients after myocardial infarction was tise of a hospital committee of staff phy- ment, a recently updated17 published beneficial, because large, well-designed sicians and pharmacists. Although the systematic review of studies assessing randomized trials have demonstrated investigators used reference texts, the CDSSs used in inpatient and outpa- the benefit of these 3 interventions. Un- degree to which they applied an evi- tient clinical settings by health care pro- fortunately, the link between pro- dence-based approach is unclear. Use viders2 showed that the majority of cesses and outcomes is often weak or of solid evidence29 from the literature CDSSs studied were beneficial. The re- unknown. could enhance clinician acceptance by view assessed patient-related out- The study by Overhage et al1 was able convincing physicians that the rules comes (eg, mortality, length of hospi- to demonstrate that a physician work- positively affect patient outcomes. How- tal stay, decrease in infections) or health station, when linked to an order entry ever, gaining consensus even with evi- care process measures (eg, compli- system able to run a series of rules, is dence-based practices can be difficult ance with reminders or with evidence- an efficient means for decreasing er- and a method for gaining consensus based processes of care). A total of 68 rors of omission and improving adher- must be integrated into the local pro- prospective trials using concurrent con- ence to practice guidelines. It is un- cesses and culture of care. Further- trol groups have reported the effects of clear how many of the rules in the more, physicians will need some time using CDSSs on drug dosing, diagno- system were based on solid evidence to become acquainted with any new sys- sis, preventive care, and active medi- and thus how likely it is that compli- tem, especially an order entry system. cal care. Forty-three (66%) of 65 stud- ance with rules will improve out- When the study by Overhage et al be- ies showed that CDSSs improved comes. Therefore, it is unclear whether gan, all physicians on the medical wards physician performance. These in- the benefits are worth the cost of pur- had been entering all inpatient orders di- cluded 9 of 15 studies on drug dosing chasing, configuring, installing, and rectly into physician workstations for 12 systems, 1 of 5 studies on diagnostic maintaining the CDSS. months. Because the order entry pro- aids, 14 of 19 preventive care systems, gram was developed over time and re- and 19 of 26 studies evaluating CDSSs RESOLUTION fined by user input, it was tailored to the for active medical care. Six (43%) of 14 OF THE SCENARIO needs of the clinicians at that hospital. studies showed that CDSSs improved A computer-based CDSS evaluation in- Whether this system would be easily ac- patient outcomes, 3 studies showed volves the interplay between 3 ele- cepted in a new environment by clini- no benefit, and the remaining studies ments: 1 or more human intermediar- cians who had nothing to do with its de- lacked sufficient power to detect a clini- ies, an integrated computerized system velopment is open to question. cally important effect. and its interface, and the knowledge in Do the Benefits of the CDSS Jus- Health care processes are more of- the decision support system. This tify the Risks and Costs? Does the re- ten evaluated than patient health out- makes evaluation of a computer- port reveal the behind-the-scenes comes because process events occur based CDSS a complex undertaking. costs? The real cost of the CDSS is usu- more frequently. For example, a trial Systematic reviews32 of the impact of a ally much higher than the initial hard- designed to show a 25% improvement CDSS on provider behavior and pa- ware, software, interface, training, (from 50% to 62.5%) in the propor- tient outcome have shown evidence of maintenance, and upgrade costs (which tion of patients who are compliant with benefit.2,15-17 Because the evaluation pro- may not be in the report). Often the a certain medication regimen would cess for these reviews was not stan- CDSS is designed and maintained by need to enroll 246 patients per group. dardized, it is difficult to compare the staff whose actions are critical to the A trial designed to show that this medi- results. success of the intervention. An insti- cation reduces mortality by 25% (from We have described a process of evalu- tution might not want to pay for the 5% to 3.75%) would need to enroll 4177 ating articles that aims to measure the time of such people in addition to the patients per group. Furthermore, long impact of a computer-based CDSS on cost of the computer software and hard- follow-up periods are required to show provider decisions or patient out-

©1999 American Medical Association. All rights reserved. JAMA, July 7, 1999—Vol 281, No. 1 73 USERS’ GUIDES TO THE MEDICAL LITERATURE comes. Despite the complexity of evalu- mask the effect of the CDSS. When us- Even if the study is valid and a posi- ation, clinicians can use basic prin- ing multilevel designs (composed of the tive effect is shown, CDSSs have spe- ciples of evidence-based care to evaluate physician or physician group and their cial applicability issues that must be CDSSs. A study evaluating a CDSS is respective patients) investigators should considered. Is the computer essential more believable if there is a concur- treat the physician or group, not the pa- to deployment of the knowledge in the rent control group with a random al- tients, as the unit of analysis. Because CDSS? Can the CDSS be exported to a location of subjects. Randomization of most studies evaluating CDSSs are not new site? Will clinicians accept the clinicians by clusters can prevent the blinded, we stressed the importance of CDSS? And, finally, is it possible to ac- cross-contamination of the control controlling for cointerventions that curately evaluate the cost of the CDSS group by the intervention that could could bias the outcome. when assessing risks and benefits?

REFERENCES

1. Overhage JM, Tierney WM, Zhou XH, McDonald tio ventilation and extracorporeal CO2 removal for adult 22. Strickland JH Jr, Hasson JH. A computer- CJ. A randomized trial of “corollary orders” to pre- respiratory distress syndrome. Am J Respir Crit Care controlled ventilator weaning system: a clinical trial. vent errors of omission. J Am Med Inform Assoc. 1997; Med. 1994;149(2 pt 1):295-305. Chest. 1993;103:1220-1226. 4:364-375. 12. Spiegelhalter DJ. Evaluation of clinical decision- 23. Guyatt GH, Pugsley SO, Sullivan MJ, et al. Effect 2. Johnston ME, Langton KB, Haynes RB, Mathieu A. aids, with an application to a system for dyspepsia. of encouragement on walking test performance. Tho- Effects of computer-based clinical decision support sys- Stat Med. 1983;2:207-216. rax. 1984;39:818-822. tems on clinician performance and patient outcome: 13. Guyatt GH, Sackett DL, Cook DJ, for the Evidence- 24. Morris AH, East TD, Wallace CJ, et al. Standard- a critical appraisal of research. Ann Intern Med. 1994; Based Medicine Working Group. Users’ guides to the ization of clinical decision making for the conduct of 120:135-142. medical literature, II: how to use an article about credible clinical research in complicated medical en- 3. Pryor TA. Development of decision support sys- therapy or prevention, A: are the results of the study vironments. Proc AMIA Annu Fall Symp. October tems. Int J Clin Monitoring Computing. 1990;7:137- valid? JAMA. 1993;270:2598-2601. 1996:418-422. 146. 14. Sacks H, Chalmers TC, Smith H Jr. Randomized 25. Adams ID, Chan M, Clifford PC, et al. Computer 4. Friedman CP, Wyatt JC. Evaluation Methods in versus historical controls for clinical trials. Am J Med. aided diagnosis of acute abdominal pain: a multicen- Medical Informatics. New York, NY: Springer- 1982;72:233-240. tre study. BMJ. 1986;293:800-804. Verlag; 1997. 15. Shea S, DuMouchel W, Bahamonde L. A meta- 26. Salvemini AV. Improving the human-computer in- 5. Lepage EF, Gardner RM, Laub RM, Golubjatnikov analysis of 16 randomized controlled trials to evalu- terface: a human factors engineering approach. MD OK. Improving blood transfusion practice: role of a ate computer-based clinical reminder systems for pre- Comput. 1998;15:311-315. computerized hospital information system. Transfu- ventive care in the ambulatory setting. JAmMed 27. Bradshaw KE, Gardner RM, Pryor TA. Develop- sion. 1992;32:253-259. Inform Assoc. 1996;3:399-409. ment of a computerized laboratory alerting system. 6. Weinfurt PT. Electrocardiographic monitoring: an 16. Balas EA, Austin SM, Mitchell JA, Ewigman BG, Comput Biomed Res. 1989;22:575-587. overview. J Clin Monitoring. 1990;6:132-138. Bopp KD, Brown GD. The clinical value of computer- 28. Tate KE, Gardner RM, Scherting K. Nurses, pag- 7. Knaus WA, Wagner DP, Draper EA, et al. The ized information services: a review of 98 randomized ers, and patient-specific criteria: three keys to im- APACHE III prognostic system: risk prediction of hos- clinical trials. Arch Fam Med. 1996;5:271-278. proved critical value reporting. Proc Annu Symp Com- pital mortality for critically ill hospitalized adults. Chest. 17. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects put Applications Med Care. October 1995:164-168. 1991;100:1619-1636. of computer-based clinical decision support systems 29. Guyatt GH, Sackett DL, Sinclair JC, et al. Users’ 8. Berner ES, Webster GD, Shugerman AA, et al. Per- on physician performance and patient outcomes: a sys- guides to the medical literature, IX: a method for grad- formance of four computer-based diagnostic systematic review. JAMA. 1998;280:1339-1346. ing health care recommendations. JAMA. 1995;274: tems. N Engl J Med. 1994;330:1792-1796. 18. Cornfield J. Randomization by group: a formal 1800-1804. 9. Kennedy RL, Harrison RF, Burton AM, et al. An ar- analysis. Am J Epidemiol. 1978;108:100-102. 30. Roethligsburger FJ, Dickson WJ. Management and tificial neural network system for diagnosis of acute 19. Whiting-O’Keefe QE, Henke C, Simborg DW. the Worker. Cambridge, Mass: Harvard University myocardial infarction (AMI) in the accident and emer- Choosing the correct unit of analysis in medical care Press; 1939. gency department: evaluation and comparison with experiments. Med Care. 1984;22:1101-1114. 31. Mant J, Hicks N. Detecting differences in quality serum myoglobin measurements. Comput Methods 20. Klar N, Donner A. The merits of matching in com- of care: the sensitivity of measures of process and out- Programs Biomed. 1997;52:93-103. munity intervention trials: a cautionary tale. Stat Med. come in treating acute myocardial infarction. BMJ. 10. Evans RS, Pestotnik SL, Classen DC, et al. A computer- 1997;16:1753-1764. 1995;311:793-796. assisted management program for antibiotics and other 21. Thompson SG, Pyke SD, Hardy RJ. The design and 32. Oxman AD, Cook DJ, Guyatt GH, for the Evi- antiinfective agents. N Engl J Med. 1998;338:232-238. analysis of paired cluster randomised trials: an appli- dence-Based Medicine Working Group. Users’ guides 11. Morris AH, Wallace CJ, Menlove RL, et al. Ran- cation of meta-analysis techniques. Stat Med. 1997; to the medical literature, VI: how to use an overview. domized clinical trial of pressure-controlled inverse ra- 16:2063-2079. JAMA. 1994;272:1367-1371.