By Ellen Fineout-Overholt, PhD, RN, FNAP, FAAN, Bernadette Mazurek Melnyk, PhD, RN, CPNP/PMHNP, FNAP, FAAN, Susan B. Stillwell, DNP, RN, CNE, and Kathleen M. Williamson, PhD, RN Critical Appraisal of the Evidence: Part I An introduction to gathering, evaluating, and recording the evidence. This is the fifth article in a series from the Arizona State University College of Nursing and Health Innovation’s Center for the Advancement of Evidence - Based Practice. Evidence-based practice (EBP) is a problem-solving approach to the delivery of health care that integrates the best evidence from studies and patient care data with clinician expertise and patient preferences and values. When delivered in a context of caring and in a supportive organizational culture, the highest quality of care and best patient outcomes can be achieved. The purpose of this series is to give nurses the knowledge and skills they need to implement EBP consistently, one step at a time. Articles will appear every two months to allow you time to incorporate information as you work toward implementing EBP at your institution. Also, we’ve scheduled “Chat with the Authors” calls every few months to provide a direct line to the experts to help you resolve questions. Details about how to participate in the next call will be pub- lished with September’s Evidence-Based Practice, Step by Step. n May’s evidence-based prac- database’s own indexing lan- library subscription or those tice (EBP) article, Rebecca R., guage, or controlled vocabulary, flagged as “free full text” by a Iour hypothetical staff nurse, matched the keywords or syn- database or journal’s Web site. and Carlos A., her hospital’s ex- onyms, those terms were also Others are available through in- pert EBP mentor, learned how to searched. At the end of the data- terlibrary loan, when another search for the evidence to answer base searches, Rebecca and Car- hos pital library shares its articles their clinical question (shown los chose to retain 18 of the 18 with Rebecca and Carlos’s hospi- here in PICOT format): “In hos- studies found in PubMed; six of tal library. pitalized adults (P), how does a the 79 studies found in CINAHL; Carlos explains to Rebecca that rapid response team (I) compared and the one study found in the the purpose of critical appraisal with no rapid response team (C) Cochrane Database of System- isn’t solely to find the flaws in a affect the number of cardiac ar- atic Reviews, because they best study, but to determine its worth rests (O) and unplanned admis- answered the clinical question. to practice. In this rapid critical sions to the ICU (O) during a As a final step, at Lynne’s rec- appraisal (RCA), they will review three-month period (T)?” With ommendation, Rebecca and Car- each study to determine the help of Lynne Z., the hospi- los conducted a hand search of • its level of evidence. tal librarian, Rebecca and Car- the reference lists of each study • how well it was conducted. los searched three databases, they retained looking for any rele- • how useful it is to practice. PubMed, the Cumulative Index vant studies they hadn’t found in Once they determine which of Nursing and Allied Health their original search; this process studies are “keepers,” Rebecca Literature (CINAHL), and the is also called the ancestry method. and Carlos will move on to the Cochrane Database of Systematic The hand search yielded one ad- final steps of critical appraisal: Reviews. They used keywords ditional study, for a total of 26. evaluation and synthesis (to be from their clinical question, in- discussed in the next two install- cluding ICU, rapid response RAPID CRITICAL APPRAISAL ments of the series). These final team, cardiac arrest, and un- The next time Rebecca and Car- steps will determine whether planned ICU admissions, as los meet, they discuss the next overall findings from the evi- well as the following synonyms: step in the EBP process—critically dence review can help clinicians failure to rescue, never events, appraising the 26 studies. They improve patient outcomes. medical emergency teams, rapid obtain copies of the studies by Rebecca is a bit apprehensive response systems, and code printing those that are immedi- because it’s been a few years since blue. Whenever terms from a ately available as full text through she took a research class. She [email protected] AJN ▼ July 2010 ▼ Vol. 110, No. 7 47 shares her anxiety with Chen M., new EBP team, Carlos provides and the Boston University Medi- a fellow staff nurse, who says Rebecca and Chen with a glossary cal Center Alumni Medical Li- she never studied research in of terms so they can learn basic brary [http://medlib.bu.edu/ school but would like to learn; research terminology, such as sam- bugms/content.cfm/content/ she asks if she can join Carlos ple, independent variable, and de- ebmglossary.cfm#R].) and Rebecca’s EBP team. Chen’s pendent variable. The glossary Determining the level of evi- spirit of inquiry encourages Re- also defines some of the study de- dence. The team begins to divide becca, and they talk about the signs the team is likely to come the 26 studies into categories ac- opportunity to learn that this across in doing their RCA, such cording to study design. To help project affords them. Together as systematic review, randomized in this, Carlos provides a list of they speak with the nurse man- controlled trial, and cohort, qual- several different study designs ager on their medical–surgical itative, and descriptive studies. (see Hierarchy of Evidence for unit, who agrees to let them use (For the definitions of these terms Intervention Studies). Rebecca, their allotted continuing educa- and others, see the glossaries pro- Carlos, and Chen work together tion time to work on this project, vided by the Center for the Ad- to determine each study’s design after they discuss their expecta- vancement of Evidence-Based by reviewing its abstract. They tions for the project and how its Practice at the Arizona State Uni- also create an “I don’t know” outcome may benefit the patients, versity College of Nursing and pile of studies that don’t appear the unit staff, and the hospital. Health Innovation [http://nursing to fit a specific design. When they Learning research terminol- andhealth.asu.edu/evidence-based- find studies that don’t actively ogy. At the first meeting of the practice/resources/glossary.htm] answer the clinical question but Hierarchy of Evidence for Intervention Studies Type of evidence Level of evidence Description Systematic review or I A synthesis of evidence from all relevant random ized controlled trials. meta-analysis Randomized con- II An experiment in which subjects are randomized to a treatment group trolled trial or control group. Controlled trial with- III An experiment in which subjects are nonrandomly assigned to a out randomization treatment group or control group. Case-control or IV Case-control study: a comparison of subjects with a condition (case) cohort study with those who don’t have the condition (control) to determine characteristics that might predict the condition. Cohort study: an observation of a group(s) (cohort[s]) to determine the development of an outcome(s) such as a disease. Systematic review of V A synthesis of evidence from qualitative or descrip tive studies to qualitative or descrip- answer a clinical question. tive studies Qualitative or de- VI Qualitative study: gathers data on human behavior to understand why scriptive study and how decisions are made. Descriptive study: provides background information on the what, where, and when of a topic of interest. Expert opinion or VII Authoritative opinion of expert committee. consensus Adapted with permission from Melnyk BM, Fineout-Overholt E, editors. Evidence-based practice in nursing and healthcare: a guide to best practice [forthcoming]. 2nd ed. Philadelphia: Wolters Kluwer Health/Lippincott Williams and Wilkins. 48 AJN ▼ July 2010 ▼ Vol. 110, No. 7 ajnonline.com Critical Appraisal Guide for Quantitative Studies 1. Why was the study done? • Was there a clear explanation of the purpose of the study and, if so, what was it? 2. What is the sample size? • Were there enough people in the study to establish that the findings did not occur by chance? 3. Are the instruments of the major variables valid and reliable? • How were variables defined? Were the instruments designed to measure a concept valid (did they measure what the researchers said they measured)? Were they reliable (did they measure a concept the same way every time they were used)? 4. How were the data analyzed? • What statistics were used to determine if the purpose of the study was achieved? 5. Were there any untoward events during the study? • Did people leave the study and, if so, was there something special about them? 6. How do the results fit with previous research in the area? • Did the researchers base their work on a thorough literature review? 7. What does this research mean for clinical practice? • Is the study purpose an important clinical issue? Adapted with permission from Melnyk BM, Fineout-Overholt E, editors. Evidence-based practice in nursing and healthcare: a guide to best practice [forthcoming]. 2nd ed. Philadelphia: Wolters Kluwer Health/Lippincott Williams and Wilkins. may inform thinking, such as or a meta-analysis—is the most appraisal process (to appear in descriptive research, expert opin- reliable and the best evidence to future installments of this series). ions, or guidelines, they put them answer their clinical question. Creating a study evaluation aside. Carlos explains that they’ll Using a critical appraisal table. Carlos provides an online be used later to support Rebecca’s guide.