Critical Appraisal of the Evidence: Part I an Introduction to Gathering, Evaluating, and Recording the Evidence

By Ellen Fineout-Overholt, PhD, RN, FNAP, FAAN, Bernadette Mazurek Melnyk, PhD, RN, CPNP/PMHNP, FNAP, FAAN, Susan B. Stillwell, DNP, RN, CNE, and Kathleen M. Williamson, PhD, RN

Critical Appraisal of the Evidence: Part I An introduction to gathering, evaluating, and recording the evidence.

This is the fifth article in a series from the Arizona State University College of Nursing and Health Innovation’s Center for the Advancement of Evidence - Based Practice. Evidence-based practice (EBP) is a problem-solving approach to the delivery of health care that integrates the best evidence from studies and patient care data with clinician expertise and patient preferences and values. When delivered in a context of caring and in a supportive organizational culture, the highest quality of care and best patient outcomes can be achieved. The purpose of this series is to give nurses the knowledge and skills they need to implement EBP consistently, one step at a time. Articles will appear every two months to allow you time to incorporate information as you work toward implementing EBP at your institution. Also, we’ve scheduled “Chat with the Authors” calls every few months to provide a direct line to the experts to help you resolve questions. Details about how to participate in the next call will be pub- lished with September’s Evidence-Based Practice, Step by Step.

n May’s evidence-based prac- database’s own indexing lan- library subscription or those tice (EBP) article, Rebecca R., guage, or controlled vocabulary, flagged as “free full text” by a Iour hypothetical staff nurse, matched the keywords or syn- database or journal’s Web site. and Carlos A., her hospital’s ex- onyms, those terms were also Others are available through in- pert EBP mentor, learned how to searched. At the end of the data- terlibrary loan, when another search for the evidence to answer base searches, Rebecca and Car- hos pital library shares its articles their clinical question (shown los chose to retain 18 of the 18 with Rebecca and Carlos’s hospi- here in PICOT format): “In hos studies found in PubMed; six of tal library. pitalized adults (P), how does a the 79 studies found in CINAHL; Carlos explains to Rebecca that rapid response team (I) compared and the one study found in the the purpose of critical appraisal with no rapid response team (C) Cochrane Database of System- isn’t solely to find the flaws in a affect the number of cardiac ar atic Reviews, because they best study, but to determine its worth rests (O) and unplanned admis answered the clinical question. to practice. In this rapid critical sions to the ICU (O) during a As a final step, at Lynne’s rec- appraisal (RCA), they will review threemonth period (T)?” With ommendation, Rebecca and Car- each study to determine the help of Lynne Z., the hospi- los conducted a hand search of • its level of evidence. tal librarian, Rebecca and Car- the reference lists of each study • how well it was conducted. los searched three databases, they retained looking for any rele- • how useful it is to practice. PubMed, the Cumulative Index vant studies they hadn’t found in Once they determine which of Nursing and Allied Health their original search; this process studies are “keepers,” Rebecca Literature (CINAHL), and the is also called the ancestry method. and Carlos will move on to the Cochrane Database of Systematic The hand search yielded one ad- final steps of critical appraisal: Reviews. They used keywords ditional study, for a total of 26. evaluation and synthesis (to be from their clinical question, in- discussed in the next two install- cluding ICU, rapid response RAPID CRITICAL APPRAISAL ments of the series). These final team, cardiac arrest, and un The next time Rebecca and Car- steps will determine whether planned ICU admissions, as los meet, they discuss the next overall findings from the evi- well as the following synonyms: step in the EBP process—critically dence review can help clinicians failure to rescue, never events, appraising the 26 studies. They improve patient outcomes. medical emergency teams, rapid obtain copies of the studies by Rebecca is a bit apprehensive response systems, and code printing those that are immedi- because it’s been a few years since blue. Whenever terms from a ately available as full text through she took a research class. She [email protected] AJN ▼ July 2010 ▼ Vol. 110, No. 7 47 shares her anxiety with Chen M., new EBP team, Carlos provides and the Boston University Medi- a fellow staff nurse, who says Rebecca and Chen with a glossary cal Center Alumni Medical Li- she never studied research in of terms so they can learn basic brary [http://medlib.bu.edu/ school but would like to learn; research terminology, such as sam bugms/content.cfm/content/ she asks if she can join Carlos ple, independent variable, and de ebmglossary.cfm#R].) and Rebecca’s EBP team. Chen’s pendent variable. The glossary Determining the level of evi- spirit of inquiry encourages Re- also defines some of the study de- dence. The team begins to divide becca, and they talk about the signs the team is likely to come the 26 studies into categories ac- opportunity to learn that this across in doing their RCA, such cording to study design. To help project affords them. Together as systematic review, randomized in this, Carlos provides a list of they speak with the nurse man- controlled trial, and cohort, qual- several different study designs ager on their medical–surgical itative, and descriptive studies. (see Hierarchy of Evidence for unit, who agrees to let them use (For the definitions of these terms Intervention Studies). Rebecca, their allotted continuing educa- and others, see the glossaries pro- Carlos, and Chen work together tion time to work on this project, vided by the Center for the Ad- to determine each study’s design after they discuss their expecta- vancement of Evidence-Based by reviewing its abstract. They tions for the project and how its Practice at the Arizona State Uni- also create an “I don’t know” outcome may benefit the patients, versity College of Nursing and pile of studies that don’t appear the unit staff, and the hospital. Health Innovation [http://nursing to fit a specific design. When they Learning research terminol- andhealth.asu.edu/evidence-based- find studies that don’t actively ogy. At the first meeting of the practice/resources/glossary.htm] answer the clinical question but

Hierarchy of Evidence for Intervention Studies

Type of evidence Level of evidence Description

Systematic review or I A synthesis of evidence from all relevant randomized controlled trials. meta-analysis Randomized con II An experiment in which subjects are randomized to a treatment group trolled trial or control group. Controlled trial with III An experiment in which subjects are nonrandomly assigned to a out randomization treatment group or control group. Case-control or IV Case-control study: a comparison of subjects with a condition (case) cohort study with those who don’t have the condition (control) to determine characteristics that might predict the condition. Cohort study: an observation of a group(s) (cohort[s]) to determine the development of an outcome(s) such as a disease.

Systematic review of V A synthesis of evidence from qualitative or descriptive studies to qualitative or descrip answer a clinical question. tive studies Qualitative or de- VI Qualitative study: gathers data on human behavior to understand why scriptive study and how decisions are made. Descriptive study: provides background information on the what, where, and when of a topic of interest. Expert opinion or VII Authoritative opinion of expert committee. consensus

Adapted with permission from Melnyk BM, Fineout-Overholt E, editors. Evidence-based practice in nursing and healthcare: a guide to best practice [forthcoming]. 2nd ed. Philadelphia: Wolters Kluwer Health/Lippincott Williams and Wilkins.

48 AJN ▼ July 2010 ▼ Vol. 110, No. 7 ajnonline.com Critical Appraisal Guide for Quantitative Studies 1. Why was the study done? • Was there a clear explanation of the purpose of the study and, if so, what was it? 2. What is the sample size? • Were there enough people in the study to establish that the findings did not occur by chance? 3. Are the instruments of the major variables valid and reliable? • How were variables defined? Were the instruments designed to measure a concept valid (did they measure what the researchers said they measured)? Were they reliable (did they measure a concept the same way every time they were used)? 4. How were the data analyzed? • What statistics were used to determine if the purpose of the study was achieved? 5. Were there any untoward events during the study? • Did people leave the study and, if so, was there something special about them? 6. How do the results fit with previous research in the area? • Did the researchers base their work on a thorough literature review? 7. What does this research mean for clinical practice? • Is the study purpose an important clinical issue?

may inform thinking, such as or a meta-analysis—is the most appraisal process (to appear in descriptive research, expert opin- reliable and the best evidence to future installments of this series). ions, or guidelines, they put them answer their clinical question. Creating a study evaluation aside. Carlos explains that they’ll Using a critical appraisal table. Carlos provides an online be used later to support Rebecca’s guide. Carlos recommends that template for a table where Re- case for having a rapid response the team use a critical appraisal becca and Chen can put all the team (RRT) in her hospital, sh checklist (see Critical Appraisal data they’ll need for the RCA. ould the evidence point in that Guide for Quantitative Studies) Here they’ll record each study’s direction. to help evaluate the 15 studies. essential elements that answer the After the studies—including This checklist is relevant to all three questions and begin to ap- those in the “I don’t know” studies and contains questions praise the 15 studies. (To use this group—are categorized, 15 of about the essential elements of template to create your own eval- the original 26 remain and will research (such as, purpose of the uation table, download the Eval be included in the RCA: three study, sample size, and major uation Table Template at http:// systematic reviews that include variables). links.lww.com/AJN/A10.) one meta-analysis (Level I evi- The questions in the critical ap dence), one randomized con- praisal guide seem a little strange EXTRACTING THE DATA trolled trial (Level II evidence), to Rebecca and Chen. As they re- Starting with level I evidence two cohort studies (Level IV evi- view the guide together, Carlos studies and moving down the dence), one retrospective pre- explains and clarifies each ques- hierarchy list, the EBP team takes post study with historic controls tion. He suggests that as they try each study and, one by one, finds (Level VI evidence), four preex- to figure out which are the essen- and enters its essential elements perimental (pre-post) interven- tial elements of the studies, they into the first five columns of tion studies (no control group) focus on answering the first three the evaluation table (see Table (Level VI evidence), and four EBP questions: Why was the study 1; to see the entire table with implementation projects (Level done? What is the sample size? all 15 studies, go to http://links. VI evidence). Carlos reminds Are the instruments of the major lww.com/AJN/A11). The team Rebecca and Chen that Level I variables valid and reliable? The discusses each element as they evidence—a systematic review remaining questions will be ad- enter it, and tries to determine if of randomized controlled trials dressed later on in the critical it meets the criteria of the critical [email protected] AJN ▼ July 2010 ▼ Vol. 110, No. 7 49 - Criteria for for Criteria activating RRT Appraisal: Appraisal: toWorth Practice Note: • Findings Data Data Analysis Measure - ment HMR HMR CR of rates UICUA

1 AP 1 ICU or ED RN Major Variables Studied Studied Variables Major (and Their Definitions) • DV1: HMR (unexpected deaths, excluding DNRs) (excluding CR DV2: DNRs) DV3: UICUA IV: RRT protocol for RRT IV: 6 months • IV: RRT RRT IV: DV1: HMR DV2: CR RRT IV: DV1: HMR RRT IV: DV1: HMR DV2: CR

Intervention group group Intervention (n = 12) Control group (n = 11) Sample/Setting • Setting: Australia Attrition: none N = 18 studies studies 18 = N Setting: acute care hos - pitals; 13 adult, 5 peds no. beds: NR Average Attrition: NR N = 2 studies 24 adult hospitals Attrition: NR N = 8 studies no. beds: 500 Average Attrition: NR N = 23 hospitals no. beds: 340 Average •

Searched 5 databases from 1950-2008, and from literature” “grey MD conferences studies only Included with a control group Searched 6 databases from 1990-2006 2 but all Excluded RCTs Searched 3 databases from 1990-2005 studies only Included with a control group Design/Method SR on RRT of effect Purpose: HMR and CR • • • • SR (Cochrane review) RRT of effect Purpose: on HMR • SR on RRT of effect Purpose: HMR and CR • RCT on RRT of effect Purpose: CR, HMR, and UICUA Conceptual Conceptual Framework None None None None

First Author (Year) 2010;170(1):18-26. Chan PS, et al. Arch Intern Med McGaughey J, et al. al. et J, McGaughey Cochrane Database Syst Rev 2007;3: CD005529. BD, et al.Winters Crit Care Med 2007;35(5): 1238-43. al. et K, Hillman Lancet 2005; 365(9477): 2091-7. tality rates; ICU = intensive care unit; IV = independent variable; MD = medical doctor; NR = not reported; Peds = pediatric; RCT = randomized controlled trial; RN = registered nurse; RRT = rapid response team; SR = systematic review; UICUA = unplanned ICU admissions. Table 1. Evaluation Table, Phase 1. I Evaluation Table, Table Shaded columns indicate where data will be entered in future installments of the series. AP = attending physician; CR = cardiopulmonary arrest or code rates; DNR = do not resuscitate; DV = dependent variable; ED = emergency department; HMR: hospital-wide mor

50 AJN ▼ July 2010 ▼ Vol. 110, No. 7 ajnonline.com appraisal guide. These elements— suggests they leave the column in. findings, not to compare them such as purpose of the study, sam- He says they can further discuss with other like studies. Rebecca ple size, and major variables—are this point later on in the process realizes that she enjoys this kind typical parts of a research report when they synthesize the studies’ of conversation, in which she and should be presented in a pre findings. As Rebecca and Chen and Chen have a voice and can dictable fashion in every study review each study, they enter its contribute to a deeper under- so that the reader understands citation in a separate reference list standing of how research impacts what’s being reported. so that they won’t have to create practice. As Rebecca and Chen continue to enter data into the table, they begin to see similarities and Usually the important information in a study differences across studies. They mention this to Carlos, who tells can be found in the abstract. them they’ve begun the process of synthesis! Both nurses are en- couraged by the fact that they’re learning this new skill. As the EBP team continues to this list at the end of the process. The MERIT trial is next in the review the studies and fill in the The reference list will be shared stack of studies and it’s a good evaluation table, they realize that with colleagues and placed at the trial to use to illustrate this phase it’s taking about 10 to 15 minutes end of any RRT policy that re- of the RCA process. Set in Aus- per study to locate and enter the sults from this endeavor. tralia, the MERIT trial1 examined information. This may be because Carlos spends much of his whether the introduction of an when they look for a description time answering Rebecca’s and RRT (called a medical emergency of the sample, for example, it’s Chen’s questions concerning how team or MET in the study) would important that they note how the to phrase the information they’re reduce the incidence of cardiac sample was obtained, how many entering in the table. He suggests arrest, unplanned admissions to patients are included, other char- that they keep it simple and con- the ICU, and death in the hospi- acteristics of the sample, as well sistent. For example, if a study tals studied. See Table 1 to follow as any diagnoses or illnesses the indicated that it was implement- along as the EBP team finds and sample might have that could be ing an RRT and hoped to see a enters the trial data into the table. important to the study outcome. change in a certain outcome, the Design/Method. After Rebecca They discuss with Carlos the like- nurses could enter “change in and Chen enter the citation infor- lihood that they’ll need a few ses- [the outcome] after RRT” as the mation and note the lack of a con sions to enter all the data into the purpose of the study. For studies ceptual framework, they’re ready table. Carlos responds that the examining the effect of an RRT to fill in the “Design/Method” more studies they do, the less on an outcome, they could say as column. First they enter RCT time it will take. He also says the purpose, “effect of RRT on for randomized controlled trial, that it takes less time to find the [the outcome].” Using the same which they find in both the study information when study reports words to describe the same pur- title and introduction. But MERIT are clearly written. He adds that pose, even though it may not have is called a “cluster-randomised usually the important informa- been stated exactly that way in controlled trial,” and cluster is a tion can be found in the abstract. the study, can help when they term they haven’t seen before. Rebecca and Chen ask if it compare studies later on. Carlos explains that it means that would be all right to take out Rebecca and Chen find it frus- hospitals, not individuals or pa- the “Conceptual Framework” trating that the study data are tients, were randomly assigned to column, since none of the stud- not always presented in the same the RRT. He says that the likely ies they’re reviewing have con- way from study to study. They reason the researchers chose to ceptual frameworks (which help ask Carlos why the authors or randomly assign hospitals is that guide researchers as to how a journals wouldn’t present similar if they had randomly assigned study should proceed). Carlos information in a similar manner. individual patients or units, oth- replies that it’s helpful to know Carlos explains that the purpose ers in the hospital might have that a study has no framework of publishing these studies may heard about the RRT and poten- underpinning the research and have been to disseminate the tially influenced the outcome. [email protected] AJN ▼ July 2010 ▼ Vol. 110, No. 7 51 To randomly assign hospitals the RRTs were activated and pro- continue the work—as long as (instead of units or patients) to vided their protocol for calling the Carlos is there to help. the intervention and comparison RRTs. However, these elements In applying these principles groups is a cleaner research de- might be helpful to the EBP team for evaluating research studies sign. later on when they make decisions to your own search for the evidence to answer your PICOT question, remember that this series can’t contain all the available Keep the data in the table consistent by using information about research meth odology. Fortunately, there are simple, inclusive terminology. many good resources available in books and online. For example, to find out more about sample size, which can affect the likeli- To keep the study purposes about implementing an RRT in hood that researchers’ results oc consistent among the studies in their hospital. So that they can cur by chance (a random finding) the RCA, the EBP team uses inclu- come back to this information, rather than that the intervention sive terminology they developed they place it in the last column, brought about the expected out- after they noticed that different “Appraisal: Worth to Practice.” come, search the Web using terms trials had different ways of de- After reviewing the studies to that describe what you want to scribing the same objectives. Now make sure they’ve captured the know. If you type sample size they write that the purpose of the essential elements in the evalua- findings by chance in a search en- MERIT trial is to see if an RRT tion table, Rebecca and Chen still gine, you’ll find several Web sites can reduce CR, for cardiopulmo- feel unsure about whether the in- that can help you better under- nary arrest or code rates, HMR, formation is complete. Carlos stand this study essential. for hospital-wide mortality rates, reminds them that a system-wide Be sure to join the EBP team and UICUA for unplanned ICU practice change—such as the in the next installment of the se- admissions. They use those same change Rebecca is exploring, that ries, “Critical Appraisal of the terms consistently throughout the of implementing an RRT in her Evidence: Part II,” when Rebecca evaluation table. hospital—requires careful consid- and Chen will use the MERIT Sample/Setting. A total of 23 eration of the evidence and this is trial to illustrate the next steps hospitals in Australia with an only the first step. He cautions in the RCA process, complete average of 340 beds per hospi- them not to worry too much the rest of the evaluation table, tal is the study sample. Twelve about perfection and to put their and dig a little deeper into the hospitals had an RRT (the inter- efforts into understanding the studies in order to detect the vention group) and 11 hospitals information in the studies. He re- “keepers.” ▼ didn’t (the control group). minds them that as they move on Major Variables Studied. The to the next steps in the critical Ellen Fineout-Overholt is clinical profes independent variable is the vari- appraisal process, and learn even sor and director of the Center for the Advancement of Evidence-Based Practice able that influences the outcome more about the studies and proj- at Arizona State University in Phoenix, (in this trial, it’s an RRT for six ects, they can refine any data in where Bernadette Mazurek Melnyk is dean and distinguished foundation months). The dependent vari the table. Rebecca and Chen feel professor of nursing, Susan B. Stillwell able is the outcome (in this case, uncomfortable with this uncer- is clinical associate professor and pro HMR, CR, and UICUA). In this tainty but decide to trust the program coordinator of the Nurse Educator Evidence-Based Practice Mentorship trial, the outcomes didn’t include cess. They continue extracting Program, and Kathleen M. Williamson do-not-resuscitate data. The RRT data and entering it into the table is associate director of the Center for the was made up of an attending phy even though they may not com- Advancement of Evidence-Based Practice. Contact author: Ellen Fineout-Overholt, sician and an ICU or ED nurse. pletely understand what they’re [email protected]. While the MERIT trial seems entering at present. They both to perfectly answer Rebecca’s realize that this will be a learn- REFERENCE PICOT question, it contains ele- ing opportunity and, though the 1. Hillman K, et al. Introduction of ments that aren’t entirely relevant, learning curve may be steep at the medical emergency team (MET) system: a cluster-randomised con such as the fact that the research- times, they value the outcome of trolled trial. Lancet 2005;365(9477): ers collected information on how improving patient care enough to 2091-7.

52 AJN ▼ July 2010 ▼ Vol. 110, No. 7 ajnonline.com