Critical Appraisal of Common Beliefs about and Barriers Evidence to Critical Appraisal A Focus on  It takes too much time Intervention/Treatment Studies  It is too difficult Bernadette Mazurek Melnyk, PhD, CPNP/PMHNP, FAANP, FAAN Associate Vice President for Health Promotion  It takes a researcher or expert in University Chief Wellness Officer EBP to do it well Dean and Professor, College of Nursing  You have to be a “whiz” at statistics Professor of Pediatrics & Psychiatry, College of Editor, Worldviews on Evidence-Based Nursing

Critical Appraisal: Three Key Questions in The Bottom Line Critical Appraisal  Are the findings valid (as close to the Being able to determine if findings truth as possible)? from a valid study can be generalized to your patient(s) and factoring those  Are the findings important (reliable) findings into decision-making with - What is the impact of the intervention your patients (size of the effect)?

 Are the findings clinically relevant/ applicable to my patient(s)?

Critical Appraisal of an Intervention Three Major Criteria for or Treatment Study Establishing Causality The purpose of this type of  The independent variable (the treatment study is to determine whether or intervention) must precede the a causal relationship exists dependent variable or outcome in time between an intervention or  There must be a relationship between the treatment and an outcome independent and dependent variable

 This relationship can not be explained by other extraneous factors

1 Explanations for Differences Critical Appraisal of a Therapy or Between Study Groups in Intervention Trial: The Basics Intervention Studies Are the results valid? Five Major Questions  Extraneous/confounding variables 1. Were subjects randomly assigned to treatment groups and was the random  Differences between how the assignment concealed from the individuals groups were treated during the enrolling subjects? 2. Was the follow-up sufficiently long to study the effects of the treatment and were all  The treatment or intervention patients accounted for at the end of the worked! study?

Critical Appraisal of a Therapy or Are the results valid? Intervention Trial Additional Questions 3. Were patients analyzed in the group to •Were patients and providers kept blind which they were assigned? to treatment? 4. Was the control group appropriate? •Were the groups treated equally aside 5. Were the instruments used to measure from the experimental treatment? the outcome variables valid and reliable? •Were the groups equal at the beginning of the study?

What are the results of the study Reduction and are they important?

What was the magnitude of the treatment effect?  Proportion of risk for bad (RRR) outcomes in the intervention Absolute Risk Reduction (ARR) group compared to the (NNT) unexposed control group Number Needed to Harm (NNH) Effect sizes How precise was the treatment effect? Confidence intervals

2 Outcome of Smoking Cessation at Six Months Following Acupuncture Therapy Relative Risk Reduction (RRR) or Placebo (sham acupuncture)

OUTCOME RRR ={|Re-Ru|/Ru} X 100 EXPOSURE Yes No (Not Total (Quit) quit) Re (EER) = 5/(5+23) = .179 Yes 5 23 28 (Acupuncture) Ru (CER) = 4/(4+23) = .148

No 4 23 27 (Sham acupuncture) RRR = {[|.179-.148|]/.148}X 100 = 21%

Total 9 46 55

Absolute Risk Reduction (ARR) Absolute Risk Reduction (ARR)

 The absolute difference between ARR =Ru-Re X 100 the unexposed and exposed groups’ risks (i.e., occurrence in the Re (EER) = 5/(5+23) = .179 unexposed/ control group subtracted from the occurrence in Ru (CER) = 4/(4+23) = .148 the exposed/intervention group) ARR = (|.179-.148|) X 100 = 3%

Number Needed to Treat (NNT) Number Needed to Harm (NNH)

The number of patients that need to be treated to The number of patients that need to be achieve one additional favorable outcome. treated to achieve one negative outcome. NNT = 1/ARR ARI (absolute risk increase) = {Re – Ru} X 100 NNT = 1/.03 = 33 NNH = 1/ARI 33 people would need to treated with acupuncture NNH = 1/.149-.178 = 35 to promote one person’s smoking cessation 35 people would need to be treated with acupuncture to promote continuation of smoking

3 Effect Size Effect Size

An estimate of how large the treatment effect is,  Calculated as the mean of the that is how well the intervention worked in the experimental group minus the mean of experimental group in comparison to the control the control group divided by the pooled group or average standard deviation

The larger the effect size, the stronger are the  If found to be so large that it is unlikely experimental intervention’s effects to have occurred by chance, then we say it is statistically significant

Statistical Significance versus Effect Sizes for Interventions Clinical Meaningfulness

Statistical significance: the results obtained in a .2 = Small effect study are unlikely to be caused by chance Statistical significance is largely dependent upon .5 = Medium effect power and the number of subjects in a study

.8 = Large effect The larger the sample, the greater the power and probability of detecting significant differences between study groups, even when effect sizes are small

Randomized Controlled Trial Randomized Controlled Trial

In teens, what is the effect of imagery versus deep 50 – 40 breathing on state anxiety level? = .5 Medium Effect 20 20= Experimental Group 20= Control Group This is clinically meaningful but not statistically significant Findings of the independent t-test revealed p=.15

4 Important Take Home Message P value

 The probability of an event given the When evaluating intervention assumption that there is no true trials, the focus should be on the difference between intervention and control magnitude of effects instead of  Unlike CIs, p values do not tell us statistically significant differences anything about the precision of the measures or the size of the effect

Confidence Intervals (CI) Confidence Intervals (CI)

 The range in which the real answer lies with  95% CIs = range in which can a given degree of certainty (usually 95%) expect the “true” result to lie  Describe an interval around the estimated 95% of the time effect point (you can get results within this interval, if you replicate their intervention)  Meaning that in 1 in 20 studies,  A 95% CI for the mean is sometimes on average, the real value will lie loosely described as having a 95% outside the CI limits probability of containing the true mean

Confidence Intervals (CI) How Precise is the Treatment Effect?  The wider the CI, the less confident we are about the true result Confidence Interval: The precision of the effect  CI width is greatly dependent on sample size A NNT of 5 with a 95% confidence interval of 3-8  If CI for RR or OR contains 1, then would mean that we have 95% confidence that the the results are not statistically true NNT value is between 3 and 8 significant  If CI for other statistics contain 0, then results are NS

5 Will the Results From this Study Help Me in Caring for My Patients?

 Are the results applicable to my patient(s)?

 Were all clinically important outcomes considered?

 What are the risks and benefits of the treatment?

 Is the treatment feasible in my clinical setting?

 What are my patient’s values and expectations for both the outcome that is trying to be prevented and the treatment itself?

Exercise in Are the Study Findings Valid? Rapid Critical Appraisal of a RCT  Were the subjects randomly assigned to the experimental and control  Melnyk, B.M. & colleagues (2004). groups? Creating opportunities for parent  Were the follow-up assessments empowerment: Program effects on the conducted long enough to fully study mental health/coping outcomes of the effects of the intervention? critically ill young children and their  Did at least 80% of the subjects mothers. Pediatrics, 113(4), e597-607 complete the study?

Are the Study Findings Valid? Are the Study Findings Valid?

  Was random assignment concealed Were the subjects and providers kept from the individuals who were first blind to study group? enrolling subjects into the study?  Were the instruments used to  Were the subjects analyzed in the measure the outcomes valid and group to which they were randomly reliable? assigned?  Were the subjects in each of the  Was the control group appropriate? groups similar on demographic and clinical variables?

6 What are the Results of the Child Outcomes Study and are They Important? BASC (Externalizing Behaviors) COPE Control Effect  How large is the intervention or Size* treatment effect (NNT, NNH, Effect n M n M Size, Level of Significance)? Time 4 73 98.6 57 98.8 .07  How precise is the intervention or Time 7 49 99.6 41 107.9 .34 treatment (Confidence interval)? Time 8 55 100.0 34 103.6 .17 Time 9 43 97.3 25 111.6 .62** *Small effect=.2; Medium=.5; Large=.8 **p<.05

Will the Results Help Me in Will the Results Help Me in Caring for my Patients? Caring for my Patients?

 Are the results applicable to my  Is the treatment feasible in my clinical patients? setting?  Were all clinically important outcomes measured?  What are my patients/family’s values  What are the risks and benefits of the and expectations for the outcome and treatment? the treatment itself?

Helpful References on Critical Appraisal Contact Information

 Melnyk, B.M. & Fineout-Overholt, E. (2014). Evidence-based [email protected] practice in nursing & healthcare. A guide to best practice (3rd Edition). Philadelphia: Lippincott Williams & Wilkins.

 Melnyk, B.M., & Morrison-Beedy, D. (2012). Designing, 614-292-4844 Conducting, Analyzing and Funding Intervention Research. A Practical Guide for Success. New York, NY: Springer Publishing Company. Follow me on Twitter @bernmelnyk

Slide Presentation Copyright, 2015

7