Critical Appraisal of Scientific Literature

André Valdez, PhD Stanford Health Care Stanford University School of Medicine Learning Objectives

 Understand the major components of a scientific article

 Learn about some of the most common study designs and analyses

 Become more familiar with literature appraisal tools

Confidential – For Discussion Purposes Only 2 Why Does This Matter?

 Greater understanding of the literature = better advocacy for EBP

 Increased literacy on a given clinical topic and gaps in knowledge

 Increased awareness of a given study’s potential and its limitations

 Grounding of clinical decision making in sound scientific reasoning

Confidential – For Discussion Purposes Only 3 Sections of a Scientific Article – Abstract

 Introduction

 Objectives/hypothesis

 Methods

 Results

 Discussion/Conclusion

 Two general types: structured and unstructured − Structured contains headings for each major section of a study − Unstructured does not contain headings

Confidential – For Discussion Purposes Only 4 Sections of a Scientific Article – Introduction

 Helps the reader become familiar with the topic

 Refers to recent literature and clearly maps out study’s rationale

 Supports central argument with citations

 What question(s) does the study attempt to answer?

 Why is the study design appropriate?

(duPrel et al. 2009)

Confidential – For Discussion Purposes Only 5 Sections of a Scientific Article – Methods

 Highlights data for appraisal of study’s validity

 Describes all phases of planning, study sample, procedures, and statistical tests

 Lays out study design, which is paramount

 Precise description of inclusion/exclusion criteria

 Thoroughly describes how data were collected, measurement scales of variables, etc.

 Sample size calculation

(duPrel et al. 2009)

Confidential – For Discussion Purposes Only 6 Sections of a Scientific Article – Results

 Clear and objective presentation of findings

 Summary statistics on all variables

 Inferential statistics on variables of interest

 Tables and figures to improve clarity

(duPrel et al. 2009)

Confidential – For Discussion Purposes Only 7 Sections of a Scientific Article – Discussion

 Compares findings with what’s currently known

 What has the study added to the existing body of knowledge?

 What conclusions may the reader draw?

 Do the interpretations follow logically from the results?

 Analyzes study limitations

 Statistical significance not the same as clinical relevance

(duPrel et al. 2009) Confidential – For Discussion Purposes Only 8 Common Study Designs

 Quality Improvement/Process/Performance Evaluation − Set up to enhance clinical practice at a local level (not generalized) − Often but not always quasi/non-experimental (no random assignment)  Uncontrolled pre-post comparison  Controlled pre-post comparison  Qualitative experience  Cost-effectiveness  Human Subject Research − Set up to generate new knowledge (generalizable) − Often but not always experimental (random assignment)  Randomized controlled trial  Quasi/non-experimental/observational − Case-control − Cohort − Longitudinal − Retrospective chart review

(Portela et al. 2015; Topics in Biostatistics 2007) Confidential – For Discussion Purposes Only 9

(Portela et al. 2015) Common Statistical Analyses

 Variable scale: continuous (parametric: assumptions about data distribution) − Independent-samples t-test − Paired t-test − Analysis of Variance − Linear regression  Variable scale: categorical − Chi-square test − Logistic regression  Variable scale: ordinal (non-parametric: distribution free) − Signed-rank test − Rank-sum test − Kruskal-Wallis test

(http://stattrek.com/) Confidential – For Discussion Purposes Only 10 Literature Appraisal Tools – Johns Hopkins

Evidence Levels & Quality Guides

Evidence Levels Quality Guides Level I A High quality: Consistent, generalizable results; sufficient sample size Experimental study, randomized controlled trial for the study design; adequate control; definitive conclusions; (RCT) consistent recommendations based on comprehensive literature Systematic review of RCTs, with or without review that includes thorough reference to scientific evidence meta-analysis Level II B Good quality: Reasonably consistent results; sufficient sample size for Quasi-experimental study the study design; some control, fairly definitive conclusions; Systematic review of a combination of RCTs and reasonably consistent recommendations based on fairly quasi-experimental, or quasi-experimental comprehensive literature review that includes some reference to studies only, with or without meta-analysis scientific evidence

Level III C Low quality or major flaws: Little evidence with inconsistent results; Non-experimental study insufficient sample size for the study design; conclusions cannot be Systematic review of a combination of RCTs, drawn quasi-experimental and non-experimental studies, or non-experimental studies only, with or without meta-analysis Qualitative study or systematic review with or without a meta-synthesis

(http://www.hopkinsmedicine.org/evidence-based-practice/jhn_ebp.html)

Confidential – For Discussion Purposes Only 11 Literature Appraisal Tools – Johns Hopkins

Evidence Summary Table

Sample, Evidence Article Author & Evidence Study findings that help answer the EBP Sample Size & Limitations Level & # Date Type question Setting Quality

 N/A

 N/A

 N/A

 N/A

 N/A

 N/A

 N/A

 N/A

 N/A

(http://www.hopkinsmedicine.org/evidence-based-practice/jhn_ebp.html)

Confidential – For Discussion Purposes Only 12 Literature Appraisal Tools – STROBE

(http://www.strobe-statement.org/index.php?id=available-checklists)

Confidential – For Discussion Purposes Only 13 Literature Appraisal Tools – STROBE

✔

✔ ✔ ✔

✔

✔

(http://www.strobe-statement.org/index.php?id=available-checklists)

Confidential – For Discussion Purposes Only 14 Literature Appraisal Tools – STROBE

✔

✔ ✔ ✔

(http://www.strobe-statement.org/index.php?id=available-checklists)

Confidential – For Discussion Purposes Only 15 Summary

 Understanding and appraising scientific literature is crucial for effective EBP

 Learning about study designs and analyses will help you interpret findings

 Using appraisal tools is a good way to organize your literature review

 Critical appraisal promotes critical thinking about your topic

Confidential – For Discussion Purposes Only 16