2016 Pharmacotherapy Specialty Review Course Research Design, Evidence‐Based Medicine and Statistical Analysis

Home , Number needed to harm, Relative risk, Relative risk reduction

Linda S. Tyler, Pharm.D., FASHP Chief Pharmacy Officer University of Utah Hospitals and Clinics Salt Lake City, Utah [email protected]

Learning Objectives

At the end of the presentation, the pharmacist should be able to

 Interpret biomedical literature with regard to study design, methodology, statistical analysis, and significance and applicability of reported data and conclusions.  Summarize key points from the most current pharmacy practice literature.  Explain the use of evidence based treatment guidelines and protocols.

Format: This session will present articles used as the case examples for reviewing common study design and statistical issues. The session will use audience response system and self‐reflection questions to engage the audience in the key concepts.

Premise: Participants in this course are pharmacists who have clinical practices in health system care settings. Participants will have had some previous statistics and literature evaluation courses and experiences. This session will serve as a review and help to identify areas that merit further study in preparation for the board exam.

Introduction  Have measures been taken to prevent  Is the reason for conducting the study competing interventions that may influence discussed? the results?  Are the study objectives clearly defined?  Are exclusion criteria clearly defined?  Is the null hypothesis clear?  Have adequate measures been taken to prevent information bias? Methodology  Are data sources used appropriate and likely to For each of the following questions, assess how this have the appropriate information? might influence the results or affect the validity of  What is the quality of the data? the study  Have the issues related to recall bias been adequately addressed? (case‐control, or Selection bias retrospective follow‐up study)  Have adequate measures been taken to prevent selection bias? All study designs  Is the study population adequately defined?  Have adequate measures been taken to  How were subjects selected? What are the prevent measurement bias? inclusion criteria? Are the selection  What measures were used to evaluate the procedures clearly defined? outcomes of the study? Case‐control:  Are they adequately described?  How were cases selected?  Were the measures used appropriate?  How were controls selected?  Were objective measures used?  Are the controls comparable to the cases?  Are the measures reproducible?  Was bias introduced in the selection process?  Were subjects observed for a sufficient length Follow‐up/cross‐sectional: of time?  How was the study population selected?  Have adequate measures been taken to  Was bias introduced in the selection process? prevent observer bias? Experimental:  Are the observers specified?  Were subjects randomly selected?  Have measures been taken to prevent inter‐  Did all qualified subjects have an equal chance observer variation? of being admitted to the study?  Are the treatment groups comparable? Experimental studies (other issues)  Are pertinent patient specific data provided?  Were subjects randomized? (i.e. healthy subjects vs patients, sex, age,  Are randomization procedures appropriate and concurrent disease states, concurrent therapy, clearly defined? [Allocation bias] race, weight or other pertinent information)  Are the interventions well described?  Have adequate measures been taken to  Is the study blinded? Are blinding procedures prevent classification bias? appropriate?  Does the study use specific definitions for the  Were specific data on drug regimens given study parameters? including dose, dosage form, duration of  How were patients classified for entrance into administration, time of dose in relationship to the study? Do they have the disease of meals? interest? (case‐control, experimental)  Were all study drugs given in appropriate doses and regimens? Observational Studies (other issues)  Are both groups comparable, and treated in  Is the severity of disease described? the same manner, except for the intervention?  How were the risk factors classified?  Were the measures adequate to insure or  How were the outcomes classified? evaluate compliance?  Have adequate measures been taken to  Were there any competing therapies that prevent confounding bias? would have influenced the results?

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 2  If the study is a crossover trial, was the Statistical Analyses washout period adequate between  Are appropriate descriptive statistics interventions? presented? [i.e. measure of central tendency (median, mean, mode), spread of the data Statistical Analyses (range), variation in the data (SD)]  Have the authors described the statistical  Are p values and confidence intervals analyses to be used in the study? specified?  Are the statistical tests appropriate for the type  Are the inferential statistical tests applied of data (nominal, ordinal, continuous)? appropriately?  Is the sample size determination information  Are statistical analyses meaningful? included?  Have appropriate significance levels been Discussion/ Conclusions established?  Are the author’s conclusions appropriate based  Is the power of the study described? on the data presented?  Are the results statistically significant? Overall methodology  Are the results clinically significant?  Based on the methodology, is the study likely  Does the author discuss objectively the to have external validity? limitations to the study?  Is the study sample representative of the  Are the conclusions consistent with the general population? purpose of the study?  Were the interventions practical?  Can the conclusions be extrapolated to the (experimental) population in general?

Results Overall  Do the title and abstract appropriately reflect Patients studied the content of the study?  Are the numbers of patients specified?  Does the author cite mostly primary literature?  Can all patients be accounted for? Is the article referenced appropriately?  Are the numbers of dropouts given? Are the  Who sponsored the study? reasons for dropping out described?  What is the reputation of the journal? Is it peer (experimental, follow‐up) reviewed?  Were sufficient numbers of patients studied?  Are there editorials available that discuss the  Were patient demographics presented? article? (companion editorials or editorials that  Do the groups look similar based on come out later) demographics?  In general, what are the study’s strengths and weaknesses? Data presentation  Does the study have internal validity?  Are data presented for all measurements  Does the study have external validity? Is it specified in the methodology? relevant to your problem/situation/practice?  Are data presented objectively?  Are data clear and understandable?

Study Design Features Examples / Applications Potential Problems Descriptive Describes an observation  Case study, case series, new service or  Not generalizable No comparison groups program, educational intervention, some  “Anecdotal” types of surveys  Often first information available  Ideas for future studies  Some areas this is the only type of information available: e.g. Toxicology/ poisonings, drugs in pregnancy Case‐Control  Identify cases with the  Often first comparative study design applied  Selection bias: How were cases identified? (Observational, disease of interest to new diseases or outbreaks How were controls identified? comparison groups) (outcome)  Used to study “rare” diseases  Classification bias: Were risk and  Identify controls  Can evaluate multiple risk factors outcomes appropriately classified? Also called: retrospective without the outcome  In general, relatively easy and less expensive  Information bias: adequacy of information  Look back in time to to conduct  Information bias: recall bias assess the risk factors  Highly susceptible to confounding bias  Bonferroni effect: If you evaluate enough things, by chance, one of them is bound to be significant  Weakest cause and effect relationship

Follow‐up  Identify a study  Considered the strongest study design of the  Selection bias: How was the study (Observational, population observational studies; comes the closest to population selected? comparison groups)  Exclude individuals ascertaining cause and effect relationships  Classification bias (as above) with the outcome of  Because data are collected as the study  Hawthorne effect: Would participants Also called: cohort, interest progresses and according to the study design, behave differently because they were longitudinal, prospective  Classify according to data are more consistent with decreased watched more closely? risk factor issues of adequacy of information  Surveillance bias: Is one group watched  Follow over time and  Have a denominator and time frame! More more closely than the other? assess if they develop likely to predict incidence  How does the group change over time? the outcome of  In general, more difficult to do and more  Attrition bias: Who was lost to follow‐up? interest. expensive  If a retrospective follow‐up, then issues with information bias as above Study Design Features Examples / Applications Potential Problems Cross‐sectional  Identify a study  Often an initial study design used to see how  “Chicken and egg” problems: Hardest to

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 4 (Observational, population common an event is demonstrate time/order relationships comparison groups)  Classify into groups  Surveillance bias and Hawthorne effect  Susceptible to confounding bias based on outcome unlikely problems  Selection bias: How was the study  Classify risk factors  Attrition bias unlikely population selected?  “Slice of life”: study  In general, relatively easy and less expensive  Classification bias (as above) occurs in the present to conduct  Have a denominator! Can predict prevalence.

Experimental  Investigator decides  Used to evaluate the efficacy of new drugs;  Selection bias: How were patients (Investigator intervention, how patients will be compare the efficacy of two different selected for the study? comparison groups) selected and assigned treatment regimens  Allocation bias: How were they allocated Parallel to groups  Considered the “Cadillac” of study designs to the treatment/intervention groups? Cross‐over/  Patients must consent  Able to “control” for more variables  Observer bias: Did the observers sequential to participate  Able to blind introduce bias to the study? Were the  Once assigned to  Most likely to demonstrate cause and effect groups observed differently? groups, individuals relationships  Measurement bias: Variation, reliability receive or participate  Considered the more difficult to conduct and and validity in some sort of most expensive  Compliance bias treatment or  Attrition bias intervention  Crossover: Was the washout sufficient?  Investigator conducts Did the subjects change over time? measurements to  Evaluate generalizability assess outcome

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 5 Statistical Tests Commonly Used in the Literature Type of Data Two Independent Two related samples Three or More Samples (Paired or Cross‐over) Independent Samples Nominal Data: Yes/no data, All or nothing Chi‐squared for 2 independent McNemar Test Chi‐Squared for k Examples: samples (1) independent Responded or not Fishers exact samples(2) ADR or not Alive or Dead Ordinal Data: Ranked data; not mathematically equal intervals Mann‐Whitney U test Sign test Kruskal Wallis one Examples: Wilcoxon rank sum test Wilcoxon signed rank way analysis of Likert scales (1‐5: Strongly agree; agree, neutral, Kolmogorov‐ Smirnov test variance (ANOVA) disagree, strongly disagree) Interval or continuous data: Data on a mathematical scale Student’s t‐test (3) Paired t‐test Analysis of variance Examples: (for parametric data) (ANOVA) (4) Serum creatinine Mann Whitney U test Blood Pressure (for nonparametric data.) Visual analog scale (VAS) Has descriptors for the ends, but not the points in the middle. Respondents can pick any value in between. (On a scale of 1‐10, 1 being no pain, 10 being the worst pain imaginable, what is your level of pain?) (1) Chi‐squared: The expected frequency of each cell must be at least 5. If n is less than 20, this test cannot be used. If n is between 20 and 40, you will need to calculate the expected frequency. If n is greater than 40, the expected frequency is this is usually greater than 5 but you may still want to calculate, especially if there are unequal numbers in the groups. Expected frequency is the value one would “expect” if the distribution was equal between the groups. It is calculated by the following equation. Expected frequency of a cell: = [(total of row) x (total of column)]/total n

(2) For multiple comparisons, all cells must have an expected frequency of at least 1, and greater than 20% of the cells must have an expected frequency of 5. If these requirements are not met, the test cannot be used. However, there reallye ar no alternative tests to use in this situation.

(3) The Student’s t‐test requires that several assumptions be true: the variance of each groups should be similar, both groups must have normal distribution. The sample size must be “large”. The last one is a controversial point. If the n of the study is less than 20, then the t‐test should not be used. If the total n for the study is greater than 40, a t‐test could probably be used, especially if the two samples were fairly equal in size. For total sample sizes of 20 – 40, it is controversial—being conservative, the t‐test should probably not be used.

(4) For groups of 3 or more, you will sometimes see in the literature, the application of multiple t‐tests (compare group A and B, then B and C, and finally A and C). This would be considered inappropriate. The ANOVA test works by seeing if a difference exists. If one exists, then other statistical tests are applied to detect between which groups the difference appears. These additional tests are called multiple comparison procedures or post hoc tests.

Portions of the table adapted from: Elenbaas RM, Elenbaas JK, Cuddy PG: Evaluating the medical literature Part II: Statistical analysis, Ann Emerg Med 1983;12:610‐620. ©2015, Linda S. Tyler, Pharm.D., Salt Lake City, UT. Used with permission.

In plain English (which for statistical reasons we can’t say out loud), we are trying to determine if Drug A is at least similar or better than Drug B. Just want to be sure Drug A is not much worse than Drug B. We must always state that we are trying to determine if Drug A is non‐inferior to Drug B.

Uses a different null hypothesis: Drug A is not non‐inferior to Drug B.

Can’t claim superiority right off from a non‐inferiority trial; but can do non‐inferiority then a superiority analysis in the same study on the same data, but it needs to be pre‐determined in the methods. You may see non‐inferiority testing for efficacy with superiority for safety events. Because both may be going on in the same study, or both on the same data, check methods carefully.

Statistics are set up by defining an alpha, beta, and delta value. Expect the usual values for alpha and beta (0.05 and 0.1‐0.2 respectively). The delta value is the non‐inferiority margin, or also called largest clinically acceptable difference.

To set up the comparisons, a confidence interval is calculated—to conclude they are non‐inferior, the CI must exclude the “lower end” of the delta (lower end—value “below” which Drug A would be considered inferior.) Note: Watch direction of data—depending how they are set up, the lower end may be an “upper” end. For instance, the direction will be different for cure rate vs adverse event rate—with cure rate the higher number would be better and with ADRs the lower number is better. Which side of the delta value indicates Drug A is worse? Does the confidence interval include that value? If so, then the drugs are not noninferior—Drug A is worse than Drug B.

You may see a one sided test—You are only interested in one direction, though you can use a two sided test. Typically, you could see a 97.5% confidence interval for 1‐side test with p value less than 0.025.

P values are the same, but different  If P less than 0.05 (or the alpha value selected), you reject the null hypothesis (just like hypothesis testing in superiority trials). You conclude that Drug A is non‐inferior to Drug B (or in plain English, Drug A is no worse than Drug B)  If P is 0.05 or greater, you accept the null hypothesis and conclude: Drug A is not non‐inferior to Drug B (eliminating the double negatives, in plain speak, would be: Drug A is inferior to Drug B)

Check for common pitfalls:  Is the delta reasonable? Researchers may chose a larger than necessary delta making it easier to show they are non‐inferior.  Is dosing equipotent? If use lower doses of standard drug (Drug B), then easier to show they are non‐inferior.  What data did they include in their analysis? You are looking for per protocol. Intention to treat (ITT) analysis easier to show non‐inferiority. Ideally, you want to see both per protocol and ITT.

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 7 Assessing Risk Prevalence vs. Incidence Prevalence = [Number of cases at a given point in time] / [number of persons in group at that time] Incidence = [Number of new cases occurring during a given time] / [number of persons in group during same time interval] Set up a 2x 2 table to represent data. Disease (Outcome) Present Absent Risk factor Present (exposed group) A B A+B Absent (unexposed group) C D C+D Odds Ratio = OR = AD / BC. Use OR for case control studies and cross‐sectional studies. The OR represents an estimation of the risk based on two assumptions:  The control group is a representative sample of the general population in terms of the risk factors  Assumes that the values A and C are relatively uncommon. Odds ratio is based on prevalence. Relative risk cannot be used since the numbers from a case control trial do not represent the incidence in a population (there is no denominator or sample population; there is no rate of development of outcome over time). Relative risk = RR = [A / A+B] / [C/C+D] Use RR for follow‐up studies and experimental studies. In these studies, risk is presented as relative risk, RR. You don’t have to estimate the risk because you have a starting sample population and you can calculate the risk. You also don’t have to make assumptions. It is calculated by setting up the 2 x 2 table in the samek way. Ris is a proportion of the incidences of the outcome in the two risk groups. You also know the time frame in which the outcome developed. If OR was used instead of a RR (or vice versa), you can usually calculate it and see if the mistake makes a big difference. Typically, the OR is further away from 1 than the RR; so the RR will be closer to 1 and thus more conservative. OR and RR approach being the same value as A and C become are relatively small. (You can figure this out from the equations because A and C will have little to contribute to the denominator.) RR evaluates the proportion of the incidences—as such it doesn’t tell you how frequently the event occurs, only that it occurs so many times more or less often than in the other group. Interpreting OR and RR  Numbers greater than one indicate that the first line on the 2x2 table is the factor at greater risk compared to the second line, for being associated with the outcome. For numbers less than 1, take the inverse and apply the same associations. This means that the first line in the 2x2 table has a decreased risk of causing the outcome, compared to the other factor.  1 = no difference  2 – 5 = mild association  5 – 10 = moderate association  10 = strong association Relative Risk Reduction =RRR= {[C/C+D] – [A/A+B]}/[C/C+D] Also = 1‐RR If relative risk =0.6, then the relative risk reduction is 0.4, but this is often reported as 40% reduction. Attributable Risk or Absolute Risk Reduction = ARR = [A/A+B] ‐ [C/C+D] Represents the difference in incidences between the two groups vs the RR which is the two proportions divided. Used for follow‐up and experimental studies since these are the only study designs that capture incidence. Considers how frequently an event occurs (incidence) and the difference in this frequency. Distinguishes between 1 in a million, and 1 in ten. The other measures above (OR, RR, RRR) are looking at the difference in the proportions! Number Needed to Treat = NNT = 1/ARR Tells you how many patients you would need to treat with the intervention in question in order to prevent an event which would otherwise occur. Some also use the concept as number needed to harm (NNH) if they are evaluating an adverse outcome: How many patients would you need to treat before observing the adverse outcome. NNT also considers frequency but in terms of how many patients you would need to see to observe the benefit or harm being evaluated.

These parameters represent the vocabulary used to describe diagnostic tests, and compare them with other tests. Set up a table: “Gold Standard” test (Reference Test) Disease present Disease absent “New Test” (Positive) (Negative) Disease present (Positive) A B A+B Disease absent (Negative) C D C+D A+C B+D A+B+C+D

Parameter Equation to calculate Sensitivity A / A+C Ability to identify those who have the disease (Addresses “false negatives” = 100% ‐ sensitivity ) Specificity D / B+D Ability to correctly identify those who don’t have the disease (Addresses “false positives” = 100% ‐ specificity ) Positive Predictive Value A / A+B (of the people who tested positive, how many had the disease) Negative Predictive Value D / C+D (of the people who tested negative, how many did not have the disease) Efficiency A+D / A+B+C+D (Ability to correctly classify patients; in this study this was called “observed accuracy”)

Memory aids: Each of these is the “correct” classification (both tests assess as positive or both assess as negative), divided by the denominator for the applicable column, row, or diagonal.  Sensitivity and specificity deals with the columns and are in alphabetical order from left to right.  Predictive value deals with the rows.  Efficiency evaluates everything so is the “diagonal” divided by the total.

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 9 Internet statistical resources: **(All links may be accessed via RxWebLinks: http://pharmacyservices.utah.edu/rxweblinks/ under statistics section)** RxWebLinks is the University of Utah Drug Information Service meta‐website for most useful websites.

General stats info: Wikipedia: http://en.wikipedia.org/wiki/Statistics plus search any stats term you are looking up. Under usual circumstances I would not recommend Wikipedia as a resource because of the open source nature of the reference. However, I have found it may be a good place to start for descriptions for many statistical terms and it provides good links to other resources. This link is specific to the statistics section which gives a brief primer; the resources at the end are good too. Please be sure to check other references as well.

StatPages.net http://statpages.org Great general reference with lots of interactive pages. Links to online statistical pages. Worth browsing the intro page to see all the available resources.

Reading statistics and Research: http://www.readingstats.com/ Companion website for the book: Huck SW. Reading Statistics and Research 6th edition. Boston: Pearsons, 2012: The website has lots of quizzes and resources to work through. It also lists some other articles that help illustrate key points (checkout e‐articles)

Rice University Virtual Lab in Statistics: http://onlinestatbook.com/rvls.html Includes five sections— hyperstat: statistics “book” with links to other statistics sites; Online Statistics: Interactive Multimedia Course of Study; Simulations and demonstrations: lots of visuals to explain concepts; Case studies: includes some medical examples to explain concepts, and Analysis Lab: demonstrates how to work with data.

Simple Interactive Statistical Analysis: http://www.quantitativeskills.com/sisa/ Allows you to conduct statistical analysis directly in the program. Good to give you a feel for how the various statistical tests works.

Internet resources especially good for sample size issues: See StatPages.Net above has good, relatively easy to use power calculators. See also: JavaStat: Post‐hoc power analysis. http://statpages.org/postpowr.html Good to use as a reader of the literature if authors fail to address power. Good description of controversy.

PS: Power and Sample Size calculations (Vanderbilt); http://biostat.mc.vanderbilt.edu/twiki/bin/view/Main/PowerSampleSize Software that can be downloaded to calculate sample size. Addresses several hard to find issues in sample size calculations such as for paired data.

Creative Research Systems: Sample size calculator: www.surveysystem.com/sscalc.htm Sample size calculator for survey research.

Effect Size: Curriculum, Evaluation, and Management Center, Durham University http://www.cemcentre.org/evidence‐based‐education/introduction is the main site. The url for the effect size description is : http://www.cem.org/effect‐size‐resources This focuses on education research however is one of the clearest explanations of effect size and its application. ©2015, Linda S. Tyler, Pharm.D., Salt Lake City, UT. Used with permission.

All links at http://pharmacyservices.utah.edu/rxweblinks/ under Evidence Based Medicine. (RxWebLinks lists the favorite websites of the Drug Information Service at University of Utah Health Care)

Tools for evaluating meta‐analyses 1. CASP: Critical Appraisal Skills Program: http://www.caspinternational.org/?o=1012 Tools developed by the Public Health Resource Unit of the NHS in Great Britain. Offers worksheets for evaluating systematic reviews, RCT, cohort, case‐control, and economic studies plus diagnostic test studies and clinical prediction rule. 2. Bandolier, independent group based out of Oxford to promote evidence based medicine principles, has a guide for meta‐analysis to help assess http://www.medicine.ox.ac.uk/bandolier/

Evaluating Guidelines 1. AGREE Instrument: http://www.agreetrust.org/ Assesses: Scope and purpose, stakeholder involvement, rigor of development, clarity and presentation, applicability, editorial independence, and overall assessment. 2. Bandolier— http://www.medicine.ox.ac.uk/bandolier/  Essay on what makes a good guideline with two page checklist Home>Extended Essays> Scroll down to What is series> Good Clinical Guideline http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/WhatareClinGuide.pdf  Evidence Based Medicine (EBM) glossary—defines and explains lots of terms, including statistical terms, used in evidence based medicine

National Guidelines Clearinghouse http://www.guidelines.gov/Agency for Healthcare Research and Quality (AHRQ) 1. Excellent tool for comparing guidelines: Search: select guidelines to compare >Add to my collection>Compare guidelines 2. Ability to view major recommendations 3. View full text guideline

Sources of clinical guidelines 1. PubMed 2. AHRQ Guidelines.gov 3. Cochrane Collaboration http://www.cochrane.org  Meta‐analysis on topics  Only abstracts are available without a subscription; may have a subscription through your library 4. National Institute for Clinical Excellence http://www.nice.org.uk  British National Health Service website  Contains national guidelines related to various disease states and health issues.

 Define, evaluate and interpret as appropriate

 1‐tail  exclusion criteria  number needed to  2‐tail  experimental treat  absolute risk  explanatory  objective reduction  factorial design  observation bias  absolute benefit  Fisher's exact  odds ratio increase  fixed effect model  ordinal data  adaptive trial design (meta‐analysis)  outcomes study  allocation bias  follow‐up  p value  alpha  forest plot  paired data  alpha error  funnel plot  parallel design  ascendancy search  Hawthorne effect  placebo  attrition bias  hazard ratio  placebo effect  baseline  heterogeneity (meta‐  population  beta analysis)  power  beta error  histogram  predictive value  bias  historical controls  prevalence  blind  I2 (I squared value)  probability  block randomization  incidence  publication bias  case‐control  inclusion criteria  quality of life  categorical data  inferential statistics instruments  Chi‐squared  information bias  random effects model  clinically significant  informed consent (meta‐analysis)  Cochran Q  interval data  randomize  cohort  intention to treat  randomized control  compliance bias  language bias (meta‐ trial  composite endpoints analysis)  range  confidence interval  Mann Whitney U  regression analysis  confounding bias  marginal difference  relative benefit  continuous data (non‐inferiority) increase  controls  matching (case‐  relative risk  correlation analysis control)  relative risk reduction  cross‐sectional  mean  reliability  crossover  median  retrospective cohort  decision analysis  meta‐analysis study  delta value  mode  risk  descendency search  multivariate analysis  risk ratio  descriptive study  nested case‐control  randomized design  nominal data  sample  descriptive statistics  non‐inferiority  selection bias  double‐blind  normal distribution  sensitivity  double‐dummy  null hypothesis  sensitivity analysis  drop‐out  number needed to  sequential design  editorial harm  single‐blind  effect size  specificity

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 12  standard error of the  stratification  Type II error mean  subgroups  Type I error  standard deviation  subjective  validity, internal  statistically significant  t‐test  validity, external

 Compare and contrast

 internal and external validity  inferential and descriptive statistics  observational and experimental study  intention to treat and per protocol designs analysis  experimental and follow‐up trials  intention to treat vs modified intention  parallel and cross‐over studies to treat  case‐control and follow‐up trials  mean, median and mode  case‐control and nested case‐control  nominal, ordinal and continuous data trials  parametric and non‐parametric data  follow‐up and cross‐over trials  Chi squared and Fisher's exact tests  superiority and noninferiority trials  t‐test and Mann Whitney U test  inclusion and exclusion criteria  t‐test and ANOVA  subjective and objective endpoints  t‐test vs paired t‐test  single and double blind  Type I and Type II errors  blinded and unblinded studies  1 and 2 tail tests  review article and meta‐analysis  beta and power  Searches of ascendency and decendency  p value and confidence interval  fixed and random effects model  confidence interval and standard  selection and allocation bias deviation  information and recall bias  forest and funnel plots  surveillance bias and Hawthorne effect  sensitivity and specificity  selection bias in randomized trial and  sensitivity and sensitivity analysis selection bias in a meta‐analysis  positive and negative predictive value  incidence and prevalence  efficiency vs positive predictive value  relative risk and odds ratio  Cochran Q and I2 tests  relative risk and relative risk reduction.  statistical and clinical significance  relative risk and hazard ratio  standard deviation and standard error of the mean

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 13  Given an article from the primary literature:  Describe the purpose of an article.  Identify the type of study design used  Assess the strengths and limitation of the study design in general and as it applies to this study.  Identify flaws and sources of bias in the study design.  Assess sources of potential bias in a clinical study. Assess the potential impact of bias on the study results.  Define the statistical terms used in the study and assess the appropriateness of the statistical tests used.  State the null hypothesis when appropriate.  Assess if the conclusions of the study are appropriate based on the information presented in the study.  Assess the internal and external validity of the study.  Apply the study information to a clinical setting and develop appropriate conclusions pertinent to a given clinical situation.

Concepts (not easily searched) Where possible use MeSH heading Clinical Pharmacist Pharmacist Trees: Concept falls under both Persons and Health care Category Clinical pharmacy services Pharmacy Services, Hospital Trees: Concepts falls under Health Care Category Concept of critical care could be in several Critical Care places Use trees: Intensive Care Units Use Trees: Concepts fall under both: Health Burn Units Care Category and Analytical, Diagnostic and Coronary Care Units Therapeutic Techniques and Equipment Intensive Care Units, Pediatric Category Intensive Care Units, Neonatal Respiratory Care Units "Twigs" under some categories may be more useful. (see Intensive Care Units) Critical Illness Subheadings Mortality Therapy Codes Resuscitation Concepts fall under: Analytical, Diagnostic Subheadings: and Therapeutic Techniques and Equipment Methods Category Therapy No Mesh headings for lactated ringers, Isotonic Solutions normal saline Albumins Fluid Therapy Can search supplementary terms: these are Rehydration Solutions the terms you can search that they don't tell you about. many drug names and Crystalloid solutions [supplementary concept] compounds fall in this category. (maps to isotonic solutions)

Isotonic solutions fall under Chemicals and Subheadings: Therapeutic Use Drugs Category methods Pediatric refers to the practice of medicine Use age "check tags", under filters (need to expand devoted to children, not the age of the filters). patient. ©2015, Linda S. Tyler, Pharm.D., Salt Lake City, UT. Used with permission.

Feature Location and Uses Use MeSH to build Select MeSH instead of PubMed at opening page or under the heading "More search Resources" at opening page. Allows to build search in MeSH. MeSH Definition On MeSH entry—clarifies how term indexed and how long the term has been in use MeSH Subheadings Use MeSH to add subheadings to terms—often only way to search some concepts well (eg surgical wound infections/pc (pc for prevention and control) Restrict to MeSH After subheadings on MeSH pages. One of choices allows you to only search heading that term and nothing below it (won't search the "twiggies") MeSH Entry terms MeSH page. Provides synonyms for finding term MeSH Trees (indexing Lower on MeSH page—helps to clarify how the term is used and to identify hierarchy) more specific terms Advanced PubMed Search page right below search bar. Provides opportunity to search specific to field. Also provides search history: you can go back to any step in your search if you want. You can compare number of results for each step in the search. Filters Column on left of search page. Shows artical types, text availability, publication dates, species, search fields. Show additional filters PubMed search page: on left column at bottom. Can pick other categories such as languages, sex, ages, and search fields (an alternative way to search in specific fields—see also advanced search). Display settings: Below search bar, top of white area. Different formats for citations. Can change display as far as items per page and how the search is displayed (date first, author first, journal first etc). This feature is also on the citation page. Send to: Below search bar, top of white area. Can send to your clipboard, citation manager, or email citation. This feature is also on the citation page. Clipboard: Below search bar, top of white area if you have sent articles to clipboard. It is a temporary area to park citations. From the Clipboard you can send all your articles (same send to locations as above). Search details Describes how PubMed interpreted your search. For instance, if you enter surgical wound infection (not through MeSH) just directly into the search box, the search details tells you it searched it as surgical wound infection as a MeSH heading, then it tried to be helpful and added any articles with the term surgical in all fields (so in the title, abstract etc). As you can see, especially helpful if you enter multiple word searches. Publication Types, Citation page below the citation. Provides MeSH headings used to index MeSH Terms article. If you find the perfect article, and having difficult time finding other articles similar, see how the perfect article was indexed. ©2015, Linda S. Tyler, Pharm.D., Salt Lake City, UT. Used with permission.

1. In your practice, is albumin or crystalloid therapy preferred for septic shock patients needing fluid therapy? A. Albumin 20% B. Albumin 5% C. Crystalloid D. Albumin and crystalloid

2. “Table 1”, the table that compares patient characteristics, helps to best assess which bias? A. Allocation B. Classification C. Measurement D. Selection

3. Patients with head injuries were excluded from the study. By excluding these patients, the researchers were most likely trying to prevent which bias? A. Compliance B. Confounding C. Observer D. Selection

4. What type of data does “pre‐existing condition” represent? A. Nominal B. Ordinal C. Continuous D. Not enough data to assess

The ALBIOS study investigators identified a significant difference between albumin and crystalloid in the 90 day mortality rate in patients who presented with septic shock at the time of enrollment into the study. The mortality in the albumin group was 43.6% and in the crystalloid 49.9% (RR = 0.87, 95%CI=0.77‐0.99, P=0.03).

5. What is the absolute risk reduction between these two groups? A. 0.77% B. 3.1% C. 6.3% D. 9.9%

6. What is the number needed to harm? A. 6.3 B. 15.9 C. 32.3 D. 129

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 17 7. The ALBIOS study investigators reported that in patients who presented with no septic shock (n=660) on enrollment, there was no difference in death at 90 days between the albumin and crystalloid (RR= 1.13, 95%CI=0.92‐1.39, P=0.25). In concluding there is no difference between the groups, the researchers are most like to make which hypothesis testing error? A. Type I B. Type II C. Type III D. Type IV

8. Which of the following can best prevent a beta error? A. Decrease the delta values B. Increase the alpha value C. Study more patients D. Use more sensitive measures

A study assessed if surgical or medical ICU patients with respiratory failure or shock developed long term severe cognitive impairment. The first part of the study, goal 1, was to determine the prevalence of long‐term cognitive impairment. The second part, goal 2, was to determine if patients with short vs long duration delirium, in the ICU developed long‐term cognitive impairment (N Engl J Med 2013; 369:1306‐ 1316).

9. The authors used the Confusion Assessment Method for ICU to assess delirium. In including this information in the methods section, the authors are helping to prevent which bias? A. Classification B. Compliance C. Confounding D. Selection

10. What is the study design for goal 1 of this study? A. Case Control B. Descriptive C. Experimental D. Follow‐up

11. What is the study design for goal 2 of this study? A. Case Control B. Descriptive C. Experimental D. Follow‐up

12. The authors concluded that patients with a longer duration of delirium had worse long‐term cognitive impairment (at 3 months P=0.001, 12 months P=0.04). Which of the following best describes the P value? A. Provides the probability the results are due to chance alone B. Provides the probability of making a Type II error C. Represents that there is no difference between the groups D. Represents the beta value for this study

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 18 A study identified patients with severe sepsis and septic shock that were admitted to a medical ICU. For analysis patients were grouped based on whether they survived to discharge or had an in‐hospital death. The authors assessed factors related to survival, including fluid intake. The authors concluded that increased fluid administration within the first 3 hours of sepsis onset was associated with reduced mortality. The Survivors median fluid intake =2,085 mL (940‐4,080 mL) compared to non survivors median intake=1,600 mL (290‐1,485 mL). The P value was 0.007. [Chest 2014; 146:908‐915.]

13. What is the study design for this study? A. Case Control B. Cross‐sectional C. Descriptive D. Experimental E. Follow‐up

A randomized trial was conducted to compare dabigatran 110 mg and 150 mg with warfarin in patients with atrial fibrillation to assess a difference in stroke or systemic embolism. Patients received dabigatran in blinded doses, but the warfarin was unblinded. Warfarin patients were titrated to an INR of 2‐3.

14. Warfarin arm was not blinded. Which bias could be introduced? A. Compliance B. Measurement C. Observational D. Selection

15. The event rates in the study were dabigatran 150 mg, 1.11% and warfarin 1.69%. What is absolute risk reduction? A. 0.58% B. 0.66% C. 1.51% D. 3.8%

“Must read”

Gelbach SH. Interpreting the medical literature. 5rd ed. New York: McGraw Hill, Inc. 2006. . Still my favorite text on evaluating the medical literature. Examples are abundant. Very readable (which is pretty remarkable for this topic). Some have said the exam started where this book left off. Use to reinforce the principles, then use other references to develop your sophistication in this area.

Gyatt G, Rennie D, Meade MO et al (ed). Users’ Guides to the Medical Literature: Essentials of Evidence‐ Based Clinical Practice. Chicago: American Medical Association. 3rd edition, 2015. . This is a compilation of the AMA users’ guides published between 1993‐2000. These are updated, plus some additional sections. Lots of great tables and examples. Explanations are clear. This is the handbook pocket‐sized version.

Gyatt G, Rennie D, Meade MO et al (ed). Users’ Guides to the Medical Literature: A Manual for Evidence based practice. Chicago: American Medical Association. 3rd edition 2015. . This is the same information as above with expanded explanations in many areas—bigger manual size. I prefer this larger edition. You probably want to get one or the other.

Also very good . . . .

Huck SW. Reading Statistics and Research 6th edition. Boston: Pearsons, 2012. . Not specific to the biomedical sciences. Has a companion website for more info http://www.readingstats.com/ . Book has had the answer to many of my statistical questions over the years. The website has lots of quizzes and resources to work through.

Hulley SB, Cummings SR, Browner WS et al. Designing Clinical Research. 4th edition Philadelphia: Lippincott Williams and Wilkins. 2013 . Especially good if you are going to do a research project. It walks you through the steps and all the things you need to consider in planning a great project. I really like the tables in this book for planning sample size. It gives you a really good feel for if you vary parameters how it will change your sample size.

Two new free finds:

Two books available for free from James Lind Library in pdf forms: www.jameslindlibrary.org. Organizations dedicated to fair tests in health care and making the information public.(Click on Books to find the 3 books they offer for free—the following two related to statistical issues and study design). Evans I, Thorton H, Chalmers I et al. Testing Treatments: Better Research for Better Healthcare, 2nd edition. London: Pinter & Martin Ltd, 2011. Available in pdf for free: www.testingtretments.org Woloshin S, Schwartz LM, Welch HG. Knowing Your Chances. Berkeley: University of California Press. 2008.

Wyer PCC, Keitz S, Hatala R et al. Tips for learning and teaching evidence‐based medicine: introduction to the series. CMAJ 2004; 171:347‐348.

Barratt A, Wyer PC, Hatala R et al. Tips for learners of evidence‐based medicine: 1. Relative risk reduction, absolute risk reduction and number needed to treat. CMAJ. 2004; 171:353‐358.

Montoria VM, Kleinbart J, Newman TB et al. Tips for learners of evidence‐based medicine: 2. Measures of precision (confidence intervals). CMAJ. 2005; 171:611‐615. Correction. CMAJ. 2005; 172:162.

McGinn T, Wyer PC, Newman TB et al. Tips for learners of evidence‐based medicine: 3. Measures of observer variability (kappa statistic). CMAJ. 2004; 171:1369‐1373.

Hatala R, Keitz S, Wyer P et al. Tips for learners of evidence‐based medicine: 4. Assessing heterogeneity of primary studies in systematic reviews and whether to combine their results. CMAJ. 2005; 172:661‐ 665.

Montori VM, Wyer P, Newman TB et al. Tips for learners of evidence‐based medicine: 5. The effect of spectrum of disease on the performance of diagnostic tests. CMAJ. 2005; 173:385‐390.

Annals of Emergency Medicine Series:

This series is my favorite for summarizing statistical concepts.

Gaddis ML, Gaddis GM. Introduction to biostatistics: Part 1, basic concepts. Ann Emerg Med. 1990; 19:86‐89.

Gaddis GM, Gaddis ML. Introduction to biostatistics: Part 2, descriptive statistics. Ann Emerg Med. 1990; 19:309‐315.

Gaddis GM, Gaddis ML. Introduction to biostatistics: Part 3, sensitivity, specificity, predictive value and hypothesis testing. Ann Emerg Med. 1990; 19:591‐597.

Gaddis GM, Gaddis ML. Introduction to biostatistics: Part 4, statistical inference techniques in hypothesis testing. Ann Emerg Med. 1990; 19:820‐5.

Gaddis GM, Gaddis ML. Introduction to biostatistics: Part 5, statistical inference techniques for hypothesis testing with non parametric data. Ann Emerg Med. 1990; 19:1054‐1059.

Gaddis ML, Gaddis GM. Introduction to biostatistics: Part 6, correlation and regression Ann Emerg Med. 1990; 19:1462‐1468.

(Series starting in 2010 has been good to cover issues of statistics, methods, bias and study design issues.)

Study design

Sedgwick P. Block randomisation. BMJ 2011; 343:d7139. doi:10.1136/bmj.d7139. Sedgwick P. Explanatory trials versus pragmatic trials. BMJ. 2014; 349:g6694 doi:10.1136/bmj.g6694. Sedgwick P. Sample size: how many participants are needed in a cohort study?. BMJ. 2014; 349:g6557doi:10.1136/bmj.g6557. Sedgwick P. Explanatory trials versus pragmatic trials. BMJ. 2014; 349:g6694 doi:10.1136/bmj.g6694. Sedgwick P. What is a factorial study design?. BMJ. 2014; 349:g5455 doi:10.1136/bmj.g5455. Sedgwick P. What is crossover trial?. BMJ. 2014; 349:g3191 doi:10.1136/bmj.g3191. Sedgwick P. Before and after study designs. BMJ. 2014; 349:g5074 doi:10.1136/bmj.g5074. Sedgwick P. Bias in observational study designs: prospective cohort studies. BMJ. 2014; 349:g77311 doi:10.1136/bmj.g7731 Sedgwick P. Randomised controlled trials: Balance in baseline characteristics. BMJ. 2014; 349:g5721 doi:10.1136/bmj.g5721.

Survival analysis

Sedgwick P. Survival (time to event) data I. BMJ. 2010; 341:c3537. doi:10.1136/bmj.c3537. Sedgwick P. Survival (time to event) data II. BMJ. 2010; 341:c3665. doi: 10.1136/bmj.c3665. Sedgwick P. Survival (time to event): Median survival times. BMJ. 2011; 343:d4890 doi:10.1136/bmj.d4890. Sedgwick P. Survival (time to event): Censored observations. BMJ. 2011; 343:d4816 doi:10.1136/bmj.d4816. Sedgwick P. How to read a Kaplan‐Meier survival plot. BMJ.; 2014 349:g5608 doi:10.1136/bmj.g5608.

Interpreting risk

Sedgwick P. Hazard ratios. BMJ. 2011; 343:d5918. doi:10.1136/bmj.d5918. Sedgwick P. Hazards and hazard ratios. BMJ. 2012; 345:e5980 doi: 10.1136/bmj.e5980.

Statistics

Sedgwick P. Understanding statistical hypothesis testing. BMJ.2014; 348:g3557 doi: 10.1136/bmj.g3557. Sedgwick P. Pitfalls of statistical hypothesis testing: Type I and Type II errors. BMJ. 2014; 349:g4287 doi:10.1136/bmj.g4287Sedgwick P. The log rank test. BMJ. 2010; 341:c3773. doi:10.1136/bmj.c3773. Sedgwick P. Multiple significance tests: the Bonferroni correction. BMJ. 2012; 344:3509. doi:10.1136/bmj.e509. Sedgwick P. Multiple hypothesis testing and Bonferroni's correction. BMJ. 2014; 349:g6287 doi:10.1136/bmj.g6287.

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 22 Sedgwick P. Confidence intervals and statistical significance. BMJ. 2012; 344:e2238 doi: 10.1136/bmj.e2238. Sedgwick P. Confidence intervals: predicting uncertainty. BMJ. 2012; 344:e3147 doi: 10.1136/bmj.e3147. Sedgwick P. Confidence intervals and statistical significance: rules of thumb. BMJ. 2012; 345:e4960 doi: 10.1136/bmj.e4960. Sedgwick P. Understanding confidence intervals. BMJ. 2014; 349:g6051 doi:10.1136/bmj.g6051. Sedgwick P .Spearman's rank correlation coefficient. BMJ. 2014; 349:g7327 doi:10.1136/bmj.g7327. Sedgwick P. Randomised controlled trials: tests of interaction. BMJ. 2014; 349:g6820 doi:10.1136/bmj.g6820. Sedgwick P. One way analysis of variance: post hoc testing. BMJ. 2014; 349:g7067 doi:10.1136/bmj.g7067. Sedgwick P. Pitfalls of statistical hypothesis testing: multiple testing. BMJ. 2014; 349:g5310 doi:10.1136/bmj.g5310. Sedgwick P. Understanding why "absence of evidence is not evidence of absence". BMJ. 2014; 349:g4751 doi:10.1136/bmj.g4751. Sedgwick P. Understanding P values. BMJ. 2014; 349:g4550 doi:10.1136/bmj.g4550. Sedgwick P. Randomised controlled trials: missing data. BMJ. 2014; 349:g4656 doi:10.1136/bmj.g4656.

Other

Sedgwick P. Non‐inferiority trials. BMJ. 2011; 342:d3253. doi:10.1136/bmj.d3253. Sedgwick P. How to read a forest plot. BMJ. 2012; 345:e8335 doi: 10.1136/bmj.e8335. Sedgwick P. Meta‐analysis: testing for reporting bias. BMJ. 2014; 350:g7857 doi:10.1136/bmj.g7857.

Clinical guidelines

Jaeschke R, Guyatt GH, Schunemann H. The things you should consider before you believe a clinical practice guideline. Intensive Care Med. Dec 2014. DOI 10.1007/s00134‐014‐3609‐9

Power and sample size

Designing Clinical Research (see "Also very good section")

Websites (see resource guide). Sample size calculators can be daunting, usually because they use different vocabulary between the sites. If you know the key things they are asking for, you can usually figure it out.

Goodman SN, Berlin JA. The use of predicted confidence intervals when planning experiments and the misuse of power when interpreting results. Ann Intern Med. 1994; 121:200‐6.

Lantos JD. Sample size: Profound implications of mundane calculations. Pediatrics 1993; 91:155‐7.

Lesaffre E. Superiority, equivalence, and non‐inferiority trials. Bull NYU Hosp Joint Dis. 2008; 66:150‐ 154.

Kaul S, Diamond GA. Good enough: A primer on the analysis and interpretation of noninferiority trials. Ann Intern Med. 2006; 145:62‐69.

LeHenanff A, Giraudeau B, Baron G et al. Quality of reporting of noninferiority and equivalence randomized trials. JAMA. 2006; 295:1147‐1151.

Piaggio G, Elbourne DR, Altman DG et al. Reporting of noninferiority and equivalence randomized trials: An extension of the CONSORT statement. JAMA. 2006; 295:1152‐1160.

Gotzsche PC. Lessons from and cautions about noninferiority and equivalence randomized trials. JAMA. 2006; 295:1172‐1174.

Fueglistaler P, Adamina M, Guller U. Non‐inferiority trials in surgical oncology. Ann Surg Oncol. 2007; 14:1532‐39.

Mulla SM, Scott IA, Jackevicius CA et al. How to use a noninferiority trial: Users'guides to the medical literature. JAMA. 2012; 308:2605‐2611.

Sub‐group analysis

Wang R, Lagakos SW, Ware JH et al. Statistics in medicine—Reporting of subgroup analyses in clinical trials. N Engl J Med. 2007; 357:2189‐2194.

The accompanying editorials to this article include: Proestel S. Subgroup analyses in clinical trials—to the editor. N Engl J Med. 2008; 358:1199.

Kent D, Hayward R. Subgroup analyses in clinical trials—to the editor. N Engl J Med. 2008; 358:1199.

Wang R. Lagakos SW. Subgroup analyses in clinical trials—author reply. N Engl J Med. 2008; 358:1199‐1200.

Pocock SJ, Lubsen. More on Subgroup Analyses in clinical trials—to the editor. N Engl J Med. 2008; 358:2076.

Wang R, Lagakos. More on Subgroup analyses in clinical trials—author reply. N Engl J Med 2008; 358‐2076‐2077.

Lagakos SW. The challenge of subgroup analyses—Reporting without distorting. N Engl J Med. 2006; 354:1667‐1669. Correction: N Engl J Med. 2006; 355:533b.

The accompanying editorials to this article include:

Eisner MD. The challenge of subgroup analyses—to the editor. N Engl J Med. 2006; 355:211.

Lagakos SW. The challenge of subgroup analyses—Dr. Lagakos replies. N Engl J Med. 2006; 355:211‐12

Sun X, Ioannidis JPA, Agoritsas T et al. How to use a subgroup analysis. Users' Guides to the Medical Literature. JAMA. 2014; 311:405‐411.

Meta‐analyses

Mills EJ, Ioannidis JPA, Thorlund K. How to use an article reporting multiple treatment comparisons meta‐analysis. JAMA 2012; 308:1246‐53.

Shojania KG, Sampson M, Ansari M. et al. How quickly do systematic reviews go out of date: A survival analysis. Ann Intern Med. 2007; 147:224‐233.

Other articles of interest

Higgins J, Altman DG, Gotzsche PC et al. The Cochrane Collaboration's tool for assessing risk of bias in randomized trials. BMJ. 2011; 343:d5928 doi: 10.1136/bmj.d5928.

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 25 Research Design, Evidence‐Based Disclosure Medicine and Statistical Analysis • I have nothing to disclose related to the content of this presentation

Linda S. Tyler, Pharm.D., FASHP Chief Pharmacy Officer University of Utah Hospitals and Clinics Salt Lake City, Utah

Learning Objectives Expectations!

• Interpret biomedical literature with regard to • Overview study design, methodology, statistical analysis, • Assume you have had some of this before and significance and applicability of reported • General overview of study designs and stats data and conclusions. • Practical!! • Summarize key points from the most current pharmacy practice literature. • Explain the use of evidence based treatment guidelines and protocols.

Just the beginning . . . . Resources in Syllabus

• Identify those things you know • Questionnaire • Internet resources • Use session to develop a checklist of things • Overview of study • Evidence Based you want to study designs Medicine Resources • Use resources to assist you • Statistical tests • Study checklist • Ask for help! • Noninferiority pearls • Searching hints [email protected] • Assessing risk • References • Sensitivity, specificity

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 26 Validity Study Designs Study design Key features • Internal Descriptive No comparative group, no intervention – Is it a “good” study within the confines of the study? Observational: Careful watching – How is the study structured? Case ‐control Identify patients with outcome; identify – Study design issues; statistics! control; look back in time for risk factors • External – Does the study apply to my situation? Follow‐up Identify a cohort; classify based on risk factor – Is it applicable to the patients I see? Is it practical? Cross‐sectional Identify a cohort; classify base on outcome – Generalizability Experimental: Active intervention • Final judgment is yours! Parallel Randomized to intervention groups – Don’t want to be fooled – Given the limitations, is the article useful? Crossover Randomized to sequence; each patient gets each intervention

Question 1: Case Study In your practice, is albumin or crystalloid therapy preferred for septic shock patients • Caironi P, Tognoni G, Masson S et al. needing fluid therapy? Albumin replacement in patients with severe sepsis or septic shock. N Engl J Med 2014;370:1412‐21. A. Albumin 20%

• ALBIOS Study B. Albumin 5% • Albumin replacement in severe C. Crystalloid sepsis or septic shock. N Engl J Med 2014;371:83‐84. [letter and authors D. Albumin and crystalloid reply]

Getting Started The Skeleton

• What are the bones of the study? In this study . . . • Determine Randomized controlled trial, open label Study Design – Purpose (Experimental‐parallel) – Study design Interventions 20% albumin or crystalloid – Risk factors or interventions – Outcomes Outcomes Death from any cause at 28 days – Who was studied Who was Patients with severe sepsis in ICUs studied

Caironi P et al. N Engl J Med. 2014;370:1412-21.

• Strategy of comparisons • “ . . .conducted a randomized controlled trial • What is being compared to investigate the effects of the administration • Risk factor or intervention of albumin and crystalloids as compared with crystalloids alone, targeting a serum albumin • Primary outcome or measure of success level of 30 g per liter or more in a population • Who is being studied of patients with severe sepsis.”

Caironi P et al. N Engl J Med. 2014;370:1412-21.

Experimental Trials My Purpose Statement Parallel vs Crossover To compare albumin 20% with crystalloids alone in a randomized, open label design, to • Parallel • Crossover assess a difference in mortality at 28 days in – Each patient receives – Each patient receives patients with severe sepsis or septic shock in one therapy one therapy then – Two or more concurrent another an ICU. groups – Randomized to – Interpatient variability sequence – Washout – Position effect

Caironi P et al. N Engl J Med. 2014;370:1412-21.

Experimental ‐ Parallel Experimental –Cross‐over

Treatment A Treatment A Treatment B

Outcome Patient Group 1 Patient Group 1 Patient Group 1 Washout

Recruit & Recruit & Randomize Randomize Treatment A Treatment B Treatment B

Patient Group 2 Outcome Patient Group 2 Patient Group 2

Outcome Outcome

• Control more variables • Types of bias • Better able to prevent bias • Often use flowcharting to follow a patient through the study • Can blind • Ascertain cause and effect • “Cadillac” of study designs

Question 2 : “Table 1”, the table that compares Selection Bias patient characteristics, helps to best • Was bias introduced in how the assess which bias? patients were selected? • Is the study population A. Allocation adequately defined? • Inclusion and exclusion criteria B. Classification • Treatment groups comparable C. Measurement • See “Table 1” Baseline characteristics of patients D. Selection

Image courtesy of dream designs at FreeDigitalPhotos.net

Study Selection Criteria Classification Bias

• Inclusion criteria • Exclusion criteria • Refers to how classifications made: – 18 yo or older – Terminal • Prevent – Clinical criteria for severe – Head injury sepsis within 24 hours – Use structured definitions – CHF – ICU – Conditions that require – Use “reliable,” “complete” sources of information • See supplement albumin administration – Proved or suspected infection at 1 site – Religious objection – Organ dysfunction (SOFA) – Participating in other – Two of: fever, increased studies HR, RR or WBC

Caironi P et al. N Engl J Med. 2014;370:1412-21.

• Was bias introduced when • Severe sepsis (criteria in supplement) patients assigned to their • Severity of Illness: Simplified Acute Physiology groups? • Was it truly random? Score • Evaluate in “Table 1” • Organ function: Sequential Organ Failure • Computer generated Assessment Score (SOFA) • Blinded sequence • Stratified – Center – Time met criteria and randomization

Caironi P et al. N Engl J Med. 2014;370:1412-21. Image courtesy of photoexplorer at FreeDigitalPhotos.net

Interventions Intervention

• Randomized to: Comparable Albumin: 300 mL of 20% solution, administered daily • Blinding to maintain albumin 30g per L – Double‐blind: Neither investigator nor patient or more knows patient allocation Crystalloid – Single‐blind: Either patient or investigator does Crystalloid administered to both not know groups according to early goal • Competing interventions directed therapy Received therapy for 28 days or until discharged from ICU

https://commons.wikimedia.org/wiki/File:Infuuszakjes.jpg

Competing Interventions Observer and Measurement Bias

• “All other treatments at • How are outcomes measured? the discretion of the attending physician.” • Is it appropriate? • Patient or observer influences • Consider other drugs • Sufficient observation • Other supportive care • Is it clinically meaningful?

• Assess mortality at 28 days • Tertiary outcomes: post‐ • Attributing the outcome to • Secondary outcomes hoc analyses a risk factor not related to the outcome – Death from any cause at – Renal replacement • Can control for many 90 days therapy variables in the analysis – Number of patients with – Acute kidney injury • Often difficult to prevent organ dysfunction and – Duration of mechanical degree of dysfunction • Look at exclusion criteria ventilation – Length of stay – – Severity of Systemic illness Time to suspension of • Part of the reason for pressors or inotropes tertiary outcomes was to – Organ function (SOFA) access for confounders

Caironi P et al. N Engl J Med. 2014;370:1412-21.

Last But Not Least in the Methods Section: Assessing Confounding Bias Preventing Other Problems • Exclusion criteria • Exclusion criteria • Study enough patients • Characteristics of – Terminal • Discuss with statistical patients such that they – Head injury power don’t resemble general – CHF • Usually discussed when sample size calculations patient population – Conditions that require presented in methods • Competing interventions albumin administration • Plan for statistical analyses – Religious objection – Participating in other studies

Caironi P et al. N Engl J Med. 2014;370:1412-21.

Other Biases Analyzing Results

• Attrition Bias • Add numbers to flow chart • Follow the numbers • Compliance Bias • Attrition • Present results for everything mentioned in methods • Statistics

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 31 Question 3: Patients with head injury were excluded from the study. By excluding these patients, the researchers were most likely trying to prevent which bias?

A. Compliance B. Confounding C. Observer D. Selection

What Data to Include Statistical Analysis

• Intention to treat: All patients randomized • Descriptive – Used in this study – Measure of central – Considered the most conservative analysis tendency • Modified intention to treat—all patients randomized – Spread of the data and received therapy • Inferential • Per protocol: Only those patients who completed – Null hypothesis the study – No difference exists • For many studies useful to have results of both methods presented

Statistical Tests Nominal Data

• Type of data • Categorical – Nominal: yes or no • Response rate (patients responded or not) – Ordinal: ranked • Adverse events or not – Continuous: numerical data • Alive or dead • Number of groups • Pregnant or not • Independent (parallel) or related (crossover) • Race groups

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 32 Nominal Data “Traps” Nominal Data Tests • Percentages • Chi‐Square – Seem like on a continuous scale – N>40 – Think of the data origin – 20‐40 use if expected frequency of cells >5 – Did the patient have a response or not? – Response is yes‐no • Fisher’s exact – Presented as % patients with response • Related samples: McNemar • Multiple groups or categories • 3 or more independent groups: Chi‐Square – Still assess if yes or no they belong to each group – No ranking

Ordinal Data Ordinal Data

• Ranked • Examples from other studies • Likert scales (strongly agree to strongly – Years of HRT (none, 0‐5 years, 5‐10 years, greater disagree) than 10 years) • Hierarchy – Age at diagnosis (less than 50, 50‐55, 55‐65, older) • Responses not mathematically equal

Ordinal Data “Traps” Ordinal Data Tests

• Data often presented numerically • Mann Whitney U test – Likert scale (1‐5, strongly disagree‐strongly agree) • Wilcoxon rank sum test – Calculate means • Related samples – Behaves like continuous data and presented as – Sign test continuous—need to remember still ordinal data!! – Wilcoxon signed rank test – Useful to present median, mode, “top box” • Kruskal‐Wallis ANOVA

• Interval, ratio data • Data presented as % probably not continuous • Time to disease • Are composite scales, etc, really progression continuous?—many times yes! • WBC – Battery of ordinal scales • Platelet count – Total behaves as continuous • Serum creatinine • Age, weight

Continuous Data: “Mutated” Continuous Data: VAS

• Number of patients with at least a 40% • Visual analog scales (VAS) – Scale of 0‐10, 0 being no pain, 10 being the worst pain reduction in pain (Nominal) imaginable – Only anchors the ends • Continuous data changed to mild, moderate – Administered orally or in writing or severe (Ordinal) – Handled as continuous data • Other pain scales: 0=no pain, 1=mild, 2=moderate, 3=severe – Defines all points – Handled as ordinal data

Question 4: Continuous Data Tests What type of data does “pre‐existing • Parametric vs Non‐parametric condition” represent? • Mann‐Whitney U (median) A. Nominal • Student’s t‐test (2 groups) (mean) – Normal distribution, equal variance B. Ordinal • Related Data: paired t‐test C. Continuous • ANOVA (3 or more groups) D. Not enough data to assess

Type of data Demographics Results • Start with null hypothesis

Nominal Sex Death at 28 days • Superiority trial: There is no difference Reason for admission Acute kidney injury Preexisting condition Renal replacement therapy • Equivalence: The groups are not equivalent • Non‐inferiority: The therapy is not non‐ Ordinal inferior to the other therapy Continuous Age SAPS II and SOFA score BMI Length of stay SAPS II and SOFA score Duration of mechanical ventilation

Errors in Hypothesis Testing Power and Sample Size

Truth • Power = 1 ‐  Conclusion Difference No difference • Determined by  and  values desired exists • Estimated response rate Difference Type I or alpha • exists **** error Difference believed to be valuable (alpha, P-value) • Front‐end concept! No difference Type II or beta error **** (beta)

Data for Sample Size Calculations Sample Size Calculations ALBIOS Study

• Nominal data • Continuous • Sample size of 1350 patients –  and  values –  and  values data • 80% power – Estimated response rate – Expected mean • To detect an absolute between‐group difference – Difference expected – Expected difference of 7.5 percentage points in mortality at 28 days between the groups between means • Baseline mortality 45% – Standard deviation of the means • 2‐sided P value of less than 0.05 • Could increase to 1800 pts if recommended by the safety monitoring board or based on interim analysis

Caironi P et al. N Engl J Med. 2014;370:1412-21.

• Chi‐squared for binary outcomes • Mortality at 28 days – Albumin 31.8% • Wilcoxon rank‐sum for continuous – Crystalloid 32% • 2 factor analysis of variance for repeated • RR=1 measures for fluid volumes and physiologic • 95% CI=0.87‐1.14 measures • P value=0.94 • No difference between groups • Survival estimates using Kaplan‐Meier method on any other outcome compared with log‐rank test parameter

Caironi P et al. N Engl J Med. 2014;370:1412-21.

Expressing Risk OR vs RR

• Used for nominal data • Odds ratio Relative risk • Use a 2x2 table—helpful for organizing data in – Based on prevalence – Based on incidence – – study No denominator, make Denominator assumptions – Association between • Expressed as odds ratio, relative risk or hazard – Case control, cross‐ exposure and disease ratio sectional over time – Logistic regression – Follow‐up, experimental analysis

Incidence Relative Risk

• (Number of persons developing dx/ total at • Expression of risk for follow‐up studies and risk) per unit of time experimental trials • Direct estimate of probability or risk • Accounts for denominator information • Calculation: RR = (a/a+b) / (c/c+d) • The proportion (or ratio) between the two!!

Died Survived Total • The most “optimistic” way to present risk Albumin 285 (31.8%) 610 895 A B A+B • Relative risk reduction (RRR): 1‐RR=1‐ 0.99= Crystalloid 288 (32.0%) 612 900 =1% reduction C D C+D • RR and RRR are both based on the proportion of events!! RR=(a/a+b)/(c/c+d)= 31.8%/32.0% = 0.99

Caironi P et al. N Engl J Med. 2014;370:1412-21.

Absolute Risk Reduction Number Needed to Treat (NNT)

• Takes into account the actual frequency of the • Inverse (or reciprocal) of ARR (convert percentages to results rather than just the proportion decimals!!) • Way to make numbers more practical and meaningful • Are we talking events that occur 1 in 10 or 1 in 1000?!! • “Number Needed to Harm” (NNH) for adverse events • Absolute Risk Reduction=ARR=31.8%‐ • NNT=NNH=1/ARR=1/0.002 =500 32.0%=0.2% • For every 500 patients treated with albumin , there will be 1 less death • ARR and NNT based on absolute rate of events

Describing Risk and Benefit: Survival Analysis Nomenclature Risk Benefit • Takes into account the timing of events 1‐RR Relative Risk Reduction Relative Benefit • Weighted relative risk over the entire study Increase • Result is Hazard Ratio (HR) (A/A+B)‐ Absolute Risk Reduction Absolute Benefit (C/C+D) (can be increase) Increase (can be • Data presented in Kaplan—Meier curves decrease) • Cox proportional hazards regression the most 1/ARR or Number needed to treat Number needed to common statistical analysis 1/ABI (also number needed to treat harm) • Not reported in this study, just reported the p value

• Relative risk can easily be calculated from • Probability results are due to chance alone numbers presented in the study • Determine level of significance (alpha value) • Hazard ratio is the same concept but is the prior to conducting the study weighted relative risk over time • By custom, p < 0.05 is considered “statistically – Adjusts for change over time significant” – Adjusts for “repeated measures” – Adjusts for different “slopes” of the line

Statistical Pearls Statistical Pearl

• The size of the P‐value has nothing to do with • Statistics do not determine what is important, the importance of the result statistics determine how certain we are. • Do not confuse statistical significance with clinical significance • Results that are not statistically significant are still important

Confidence Interval Notes on CI

• 95% CI • CI can be applied to any type of data • Provides a “range” for the result – Determine value that represents no difference • Many consider this the most clinically relevant • **When used with OR, RR or HR** of statistics – No difference value is represented by 1 • Calculation based on Standard Error of Mean – If CI doesn’t include 1, then statistically significant (SEM) • Relatively narrow CI have a large sample size while large CI have a small sample size

• If study was repeated 100 times, 95% of the • Were patients studied for sufficient duration? time the result would likely fall in this range • Do the results change over time? • Confident that the true result would fall in this range 95% of the time • Inferences about the population

Risk of Death at 90 Days: Subgroup Analysis Septic Shock at Enrollment • Allocation no longer applies Died Survived Total Albumin 243 (43.6%) • Sample size calculations don’t hold for ABA+B subgroups Crystalloid 281 (49.9%) • As more subgroups evaluated, more CDC+D opportunity for finding a significant result 1121 when one does not exist RR=(a/a+b)/(c/c+d)= 43.6%/49.9% = 0.87 • Results can be overstated and misleading 95%CI=0.77‐0.99, P=0.03

Caironi P et al. N Engl J Med. 2014;370:1412-21. (supplementary material)

Question 5: Question 6: What is the absolute risk reduction What is the number needed to harm? between these two groups?

A. 0.77% A. 6.3 B. 3.1% B. 15.9 C. 6.3% C. 32.3 D. 9.9% D. 129

Caironi P et al. N Engl J Med. 2014;370:1412‐21.(supplementary material) Caironi P et al. N Engl J Med. 2014;370:1412‐21.(supplementary material)

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 39 Question 7: Risk of Death at 90 Days: In concluding there is no difference between the No Septic Shock at Enrollment groups, the researchers are most like to make Died Survived Total which hypothesis testing error? Albumin 122 (37.0%) ABA+B A. Type I Crystalloid 108 (32.7%) CDC+D B. Type II Total 660 C. Type III

RR=(a/a+b)/(c/c+d)= 37.0%/32.7% = 1.13 D. Type IV 95%CI=0.92‐1.39, P=0.25

Caironi P et al. N Engl J Med. 2014;370:1412-21.(supplementary material)

Question 8: Which of the following can best prevent a Bonferroni Effect beta error? • If you look at enough outcomes or groups, A. Decrease the delta values something will be statistically significant by chance alone B. Increase the alpha value • Correction: If looked at 4 outcomes, multiply C. Study more patients the P‐value by 4 (example: P‐value of 0.02, D. Use more sensitive becomes 0.08 and would be interpreted as measures not statistically significant) • This study not adjusted for multiple testing

Key Issues in ALBIOS Study Observational Study Designs

• Is this how albumin is administered? • Case‐control • Are these results generalizable? • Cross‐sectional • Authors claim that albumin offers advantages • Follow‐up of more albumin patients reaching targeted mean arterial pressure within 6 hours, but this was not one of the study outcomes in the methods section • Other albumin advantages in subgroups

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 40 Observational Study Designs: Summary: BRAIN‐ICU Case Study • Assessed if surgical or medical ICU patients • Pandharipande PP, Girard TD, Jackson JC, et al. with respiratory failure or shock developed Long‐Term Cognitive Impairment after Critical long term severe cognitive impairment. Illness. N Engl J Med 2013;369:1306‐16. • Goal 1 determined the prevalence of long‐ term cognitive impairment. • BRAIN‐ICU Study • Goal 2 determined if patients with short vs long duration of delirium in the ICU developed long‐term cognitive impairment

Pandharipande PP et al. N Engl J Med. 2013;369:1306-16.

Question 9: Question 10: The authors used the Confusion Assessment What is the study design for goal 1 of Method for ICU to assess delirium. Which bias does this assess? this study?

A. Case‐control A. Classification B. Descriptive B. Compliance C. Experimental C. Confounding D. Follow‐up D. Selection

Question 11: What is the study design for goal 2 of Identify Observational Study Designs this study? Case – Cross – Follow‐up control sectional A. Case‐control Timeline Look back Slice in Look forward B. Descriptive time Group by No No Yes C. Experimental risk factor D. Follow‐up Group by Yes Yes No outcome Start with No Yes Yes Cohort

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 41 Case‐control Studies Case‐control Risk • Identify cases with the disease of interest Factor + (outcome) Patients with • Identify controls without the outcome - the disease • Look back in time to assess the risk factors + Patients without the disease - Past Present Adapted from Gehlbach SH. Interpreting the Medical Literature. 2006.

Retrospective . . . . Application of Case‐control

• Often confusing terminology • Applied to new diseases or outbreaks • Study design: another name for case‐control • Can study “rare” diseases • Refer to the time frame of the study • Evaluate multiple risk factors • Relatively easy and inexpensive

Weakness of Case‐control Cross‐sectional

• Most susceptible to bias • Identify a study population • Selection bias • Classify based on outcome • Classification bias • Classify based on risk factor • Information bias • Assess prevalence – Recall – Information adequacy • Confounding bias

Outcome + • Chicken and the egg present - • Confounding bias Study • Selection bias population Outcome + • Classification bias absent - Present

Adapted from Gehlbach SH. Interpreting the Medical Literature. 2006.

Follow‐up Studies Follow‐up Study Outcome

• Identify a study population Risk factor + present • Exclude individuals with the outcome of Study population - interest

• Classify based on risk factor Risk factor + • Follow over time Exclude if absent outcome - • Assess outcome present Present Future Adapted from Gehlbach SH. Interpreting the Medical Literature. 2006.

Features of Follow‐up Bias in Follow‐up

• Strongest study design • Hawthorne effect • Strongest causal link • Surveillance bias • Denominator; predict incidence • Change over time • Can usually address information bias • Attrition bias

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 43 Question 12: Patients with a longer duration of delirium had worse In this Study . . . long‐term cognitive impairment (3 mo P=0.001, 12 mo P=0.04). Which describes the P value? • Goal 1. Descriptive A. Provides the probability the – No comparison group results are due to chance alone • B. Provides the probability of Goal 2. Follow‐up making a Type II error – Started with cohort of ICU patients C. Represents that there is no – Followed them forward in time difference between the groups D. Represents the beta value for – Grouped based on risk factors this study

0% 0% 0% 0%

A. B. C. D. Pandharipande PP et al. N Engl J Med. 2013;369:1306-16.

Sensitivity Analysis Missing Data

• Conduct analysis again making different • Exclusion of these patients could have biased results – Patients with missing data may be different assumptions to assess if results would change – Patients differed in education levels, sex distribution, • In this study, conducted sensitivity analysis frailty and activities. • Imputed data with only those patients with complete data. – Used methodology to assign values to missing data for • Results would be the same. purposes of modeling data – Single imputation for inpatient delirium and coma data (3% missing) – Multiple imputations for follow‐up data: 83‐92% of patients had complete data at 3 months (5 cognitive tests used)

Pandharipande PP et al. N Engl J Med. 2013;369:1306-16. Pandharipande PP et al. N Engl J Med. 2013;369:1306-16.

Fluids and Mortality Assess the Study Design

Lee SJ, Ramar K, Park JG et al. Increased fluid • Reviewed medical records administration in the first three hours of sepsis • Identified patients with severe sepsis and resuscitation is associated with reduced mortality: A septic shock admitted to medical ICU retrospective cohort study. Chest 2014; 146:908‐15. • Grouped based on whether survived to discharge or in‐hospital death • Assessed factors related to survival

Lee SJ et al. Chest. 2014;146:908-15.

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 44 Question 13: Results What is the study design? • Concluded increased fluid administration in first 3 hours associated with reduced mortality A. Case‐control – Survivors median fluid=2,085 mL (940‐4,080 mL) B. Cross‐sectional – Non survivors median=1,600 mL (290‐1,485 mL) C. Descriptive – P value = 0.007 D. Experimental E. Follow‐up

Lee SJ et al. Chest. 2014;146:908-15.

Tips that this is Not a Summary of Study Flowchart Cohort or Follow‐up Study 651 patients Hours 0‐3 • In abstract: gave a range of fluid intake 57 excluded‐ Survived to – If a follow‐up study, would classify fluid intake into incomplete data discharge categories (e.g., 500‐1,000 mL group, 1,000‐1,500 Hours 3.1‐6 mL, etc.) 594 patients

• Based on statistical information hard to tell. Hours 0‐3 • Because continuous variable, don’t have OR or In‐hospital RR to help you distinguish. death • Included study flowchart (Figure 2) Hours 3.1‐6

Lee SJ et al. Chest. 2014;146:908‐15. Lee SJ et al. Chest. 2014;146:908-15.

Summary of Study Flowchart Incidence? Time to events Cohort Retrospective Cohort 651 patients Hours 0‐3 • All data are existing (the retrospective part) 57 excluded‐ Survived to incomplete data discharge • All other aspects of the study should flow like a Hours 3.1‐6 follow‐up study 594 patients – Identify a cohort of patients – Exclude patients with the outcome (not necessary in Grouped Hours 0‐3 this study since outcome is mortality) on Outcome In‐hospital death – Classify based on risk factor (in this case fluid intake) – Assess outcome (mortality)

Hours 3.1‐6 Past Timeline • Sometimes called Trohoc studies Reversed Lee SJ et al. Chest. 2014;146:908‐15.

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 45 Non‐inferiority Trials—They’re Why? Different!! • Different hypothesis: The two • Unethical to do a placebo‐controlled trial treatments are not non‐inferior • Treatment expected to be similar in efficacy to to each other standard treatment • When p‐values used, they mean – Therapeutic non‐inferiority to active control different things—if less than 0.05, means they are non‐inferior • Treatment assumed to be better than placebo • Can’t claim superiority with these • Treatment likely to have other advantages (safety, trials but can do a non‐inferiority cost, convenience) analysis, then a superiority analysis.

Differentiate Between Trial Types RE‐LY Trials

• Superiority trials • Connolly SJ, Ezekowitz MD, Yusuf S et al. Dabigatran – Show if better or worse versus Warfarin in Patients with Atrial Fibrillation. N Engl J Med 2009; 361:1139‐51. • Equivalence trials • Gage BF. Can We Rely on RE‐LY? N Eng J Med 2009; – neither better nor any worse 261:1200‐2. [Editorial] • Non‐inferiority • Dabigatran versus Warfarin in Patients with Atrial – not much worse than the active comparator Fibrillation. 2009; 361:2671‐75. [letters sand author – The same or better reply]

Approaches RE‐LY: Purpose

• Uses alternative hypothesis • Authors’: “We performed a large, randomized • Set a “marginal difference” = Delta trial comparing the use of dabigatran, at doses • Set confidence interval threshold (as in this of 110 mg twice daily and 150 mg twice daily, study) with warfarin.” • Compared dabigatran 110 mg and 150 mg with warfarin in patients with atrial fibrillation to assess a difference in stroke or systemic embolism.

Connolly SJ et al. N Engl J Med. 2009;361:1139-51.

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 46 Question 14: RE‐LY: Study Design Features Warfarin arm was not blinded. Which bias could be introduced? • Experimental • Blinded dabigatran dose A. Compliance • Unblinded warfarin dose adjusted to INR 2‐3 B. Measurement • Low dose ASA or platelet inhibitor use C. Observational permitted D. Selection • Quinidine permitted first 2 years then not allowed due to drug interactions

RE‐LY: Selection RE‐LY: Outcomes

• Multi‐center: 951 centers, 44 countries • Primary: stroke or systemic embolism [composite] • Primary safety: major hemorrhage • Atrial fibrillation • Secondary – Screening or within 6 months – Stroke – Previous stroke, TIA, – Systemic embolism – LVEF <40% – Death • Other outcomes: MI, PE, TIA, hospitalization – NYHA Class II or greater HF within 6 months • Primary net clinical benefit: Stroke, systemic embolism, – 75 yo or older; or 65‐74 yo with DM, PE, MI, death, major hemorrhage [composite] hypertension, or CAD

Connolly SJ et al. N Engl J Med. 2009;361:1139-51.

RE‐LY Statistics RE‐LY: Stats Continued • Primary analysis: Is dabigatran noninferior to • Intention‐to‐treat analysis warfarin? • Cox proportional‐hazards modeling • Noninferior hypothesis: the upper limit of the • After noninferiority established, evaluated for one‐sided 97.5% confidence interval for the superiority using 2‐tailed analysis relative risk needed to fall below 1.46 • Sample size: 15,000 patients • P‐value – 84% power – If higher of P values <0.025 then noninferior – Changed to 18,000 patients during the trial in case – If higher of P values >0.025, then lower needs to of low event rate (without knowing event rates) be <0.0125 to claim noninferior • Protocol change: stratified vitamin K use

Connolly SJ et al. N Engl J Med. 2009;361:1139-51. Connolly SJ et al. N Engl J Med. 2009;361:1139-51.

• Dabigatran 110 mg, BID, n=6015 • Stroke or systemic embolism – Dabigatran 110 mg, BID, 1.53% • Dabigatran 150 mg, BID, n=6075 – Dabigatran 150 mg, BID,1.11% – Warfarin, 1.69% • Warfarin, n=6022 • Both doses noninferior to warfarin: P<0.001 • Median follow‐up 2 years, follow‐up in 99.9% – Dabigatran 110 mg, BID, (RR=0.91; 95%CI=0.74‐1.11, P<0.001) – Dabigatran 150 mg, BID, (RR=0.66, 95%CI=0.53‐0.82, P<0.001) of patients • Superiority testing – Dabigatran 110 mg, BID, (RR=0.91, 95%CI=0.74‐1.11, P=0.34) – Dabigatran 150 mg , BID, (RR=0.66, 95%CI=0.53‐0.82, P=0.001)

Connolly SJ et al. N Engl J Med. 2009;361:1139-51. Connolly SJ et al. N Engl J Med. 2009;361:1139-51.

Question 15: Rate events: Dabigatran 150 mg, 1.11%, Evaluating Set Point Warfarin 1.69%. What is ARR? • Noninferior hypothesis: the upper limit of the one‐sided 97.5% confidence interval for the A. 0.58% relative risk needed to fall below 1.46 • B. 0.66% Both doses noninferior to warfarin: P<0.001 – Dabigatran 110 mg, BID, (RR=0.91; 95%CI=0.74‐1.11, C. 1.51% P<0.001) – Dabigatran 150 mg, BID, (RR=0.66, 95%CI=0.53‐0.82, D. 3.8% P<0.001) • Note: CI “crossing 1” will not help with determining significance

Connolly SJ et al. N Engl J Med. 2009;361:1139-51.

Different Hypothesis P‐values Interpreted Differently

Null Hypothesis P‐value > 0.05 P‐value < 0.05 Superiority There is no difference between Non‐ Active drug is not non‐ Yes, active drug is non‐inferior to inferiority inferior to control control (the same or better) active drug and control. (inferior) Non‐inferiority Active drug is not non‐inferior to Dabigatran 110 mg, BID, P<0.001 control. Dabigatran 150 mg, BID, P<0.001

Superiority No difference Yes, difference between groups Dabigatran 110 mg, BID, Dabigatran 150 mg, BID, P=0.001 P=0.34

• Null hypothesis “Non‐inferiority–showing a treatment • Statistics based on delta or confidence interval set prior to the trial is good because it is not bad.” • P‐value interpretation ‐‐WC Blackwelder

Meta‐analysis Why do a Meta‐analysis?

• Identifying studies • Address sample size and beta error issues that • Selection bias occur in individual studies • Data mutations • Subgroups • Clear clinical question • Forest plots • Heterogeneity • Fixed vs random effects • Sensitivity analysis • Funnel plots

Identifying Studies Selection of Studies

• Find everything!! • How selected? • Search at least two appropriate databases • Was quality assessed? • Searches of ascendancy and descendancy • Selection bias: May mean how studies are • Unpublished sources selected for inclusion in the meta‐analysis! • Language bias

______©2016 American Society of Health-System Pharmacists, Inc. All rights reserved. 49 Does it Make Sense to Combine Search for Studies Studies? • Patients • PubMed • Interventions • Other databases • Outcomes • Unpublished (trial registries, manufacturer) • Study design • Searches of ascendency and descendancy – References from systematic reviews – Web of Science Citation Index to find citations and articles that quoted the articles • Language

Fixed vs Random Effect Forest Plot

• Fixed • Random • Box (individual studies) – Assumes a range of – Assumes one true value – Point estimate – Considers sample effects variation with studies – Considers variation – Size of box: relative sample size – Use if low heterogeneity within study and • Bar: confidence interval between studies – Use if heterogeneity an • Diamond issue – Vertical points: point estimate – Horizontal points: confidence interval

Heterogeneity Forest Plot: Significant Heterogeneity

• Qualitative – Look at forest plot – Are point estimates highly variable? – Do confidence intervals overlap? • Quantitative – Cochran Q – I2 statistic

Favors treatment 1 Favors control

• Based on Chi square statistic • Variability across studies due to treatment • P‐value of less than 0.1 or 0.05 effect • Significant heterogeneity • 0% variability due to chance alone • Limitations • 0‐25% low heterogeneity – Underpowered for studies with few patients – Overpowered for large sample sizes • 25‐50% moderate heterogeneity • This study: Test for heterogeneity with Chi square • >50% high heterogeneity values and P‐value • Confidence intervals

Subgroups Sensitivity Analysis

• Do they make sense? • Evaluate the impact of different decisions • Confounding bias made in the conduct of study • Fishing expeditions • Examples • Subgroups may not reflect randomization – Different study designs – Quality of studies • Adjust for multiple comparisons • Alpha, beta, and power

Funnel Plots Funnel Plot

• Plot effect size vs sample size • Used to assess publication and selection bias (as in how studies selected)

. Data from: BMC Cancer. 2007; 7:153

Final Thoughts

Efficacy versus Effectiveness Effectiveness

• Efficacy • Does the drug therapy work in the real world? – Does the drug therapy work based on clinical – Environment not controlled; reflects how people trials? really prescribe and use the drug – Limitations of randomized controlled trials – Is the therapy worth it, in terms of outcomes —Controlled environment achieved? —Optimal compliance – Considers efficacy and safety, as well as cost • Cost effectiveness = outcome/cost • Does the therapy has value?

Educating Others More Tips . . .

• Teach in context • Flow chart articles – Use articles to reinforce fundamentals and find • “Bubble” flow chart for biases interesting things – Don’t skip methods!! • Use 2x2 tables and make calculations – Use to develop recommendations for your patients • Compare and contrast 2 articles on same topic • Use checklists – Assess study design, selection criteria, similarity of – See evidence based medicine section and other interventions resources • Read the accompanying published editorials • Use vocabulary and letters

• Consensus based • Tools for evaluating meta‐analysis – Based on expert opinion – CASP: Critical Appraisal Skills Program • Evidence based – Bandolier – Hierarchy of evidence • Evaluating Guidelines – Well done meta‐analysis – AGREE instrument – Randomized controlled trials – Bandolier – Epidemiology studies – Guidelines.gov • RxWebLinks

Effect Size Reporting Data

• Qualitative and quantitative concept • How reported affects significance placed on • Way to standardize the effect data • Used for continuous data with normal • Watch graphs!! distribution • Changing numbers to % • Calculated by dividing the difference of means • Collapsing data in categories by standard deviation • % change from baseline • Use table from website to interpret and make more practical

Searching Hints MeSH

Concepts (not easily searched) Where possible use MeSH headings • Use MeSH if having problems with a search Clinical pharmacist Pharmacists – (database choice at top of heading) Clinical pharmacy services Pharmacy Services, Hospital • Can use subheadings!! Critical Care Critical Care – Surgical prophylaxis=wound infection/pc Intensive Care Units — pc =subheading nomenclature for “prevention and control” Critical Illness Resuscitation • Subheadings vary by category Can’t search lactated Ringer’s or Isotonic solutions • Use Trees normal saline Albumins – Helps to get a sense how they index the term Fluid Therapy Rehydration Solutions – Provides other terms that may be more specific Crystalloid solutions [supplementary] • Can Restrict to Major Topics only Pediatric Use age “check tags”, under filters (need to expand • Can Limit to MeSH term without the “twiggies” filters)

6484 results vs 584 results

PubMed Features

• MeSH • Filters and Manage Filters – Subheadings – Trial type – Trees – Publication dates • Advanced – Species – Search in specific fields • Show additional – History – Language • Display settings – Ages – Medline (shows MeSH) – Search fields – Change items per page • Send to: – Sort by • MeSH headings

• Common study designs • Can’t always have a randomized trial • Sources of bias • Selection criteria • Statistical tests • Duration of therapy • Non‐inferiority trials • Meta‐analyses • Duration of evaluation • Treatment guidelines • Dosing equivalents • How results are presented • Balancing cost of therapy and clinical outcomes

www.nasa.gov/multimedia/imagegallery/image_feature_87.html

Summary

• Just the beginning . . . • One article a week, that’s all I ask!! • Use skills frequently • Use resources and reference list • Internet statistical references