Understanding

Errors in interpretation

Statistical significance does not prove clinical importance. The size of the effect determines the importance, not the presence of significance.

Type I (α) error – concluding that a difference is significant when it is due to a sampling error: avoided by ensuring P<0.05, which means statistically some (less than 5%) outcomes are (or may be) the result of chance. Type II (β) error – concluding from a non-significant finding that an intervention has no effect, when there is. Small studies may report non- significance even when important real effects. Avoided by 80%+ power. Assuming External Validity – findings in a study truly refer only to the patients within that study – assessments of patient characteristics, setting and conduct of the trial determine if findings are applicable. Bias - deviation of study results from the true results (affecting validity and applicability and clinical significance), due to conduct, design or analysis of the study. Biases include selection, performance, exclusion, detection, ascertainment, attrition, publication – and more.

Clinical Significance Calculator: http://www.spph.ubc.ca/calc/clinsig.html

Bandolier Learning Zone: http://www.medicine.ox.ac.uk/bandolier/learnzone.html

NNT Calculator: http://www.ebem.org/nntcalculator.html

Stats Calculator: http://ktclearinghouse.ca/cebm/toolbox/statscalc

Diagnostic Test Calculator: http://araw.mede.uic.edu/cgi-alansz/testcalc.pl

Online Data Analysis: http://www.openepi.com/

The Library Tel: 01227 866877 Mon – Fri Education Centre x74851 9.00 – 5.00 Kent and Canterbury Hospital, Canterbury CT1 3NG Direct: 01227 864354 email: [email protected] Fax: 01227 864154

Clinical Studies Library Tel: 01843 225544 Mon – Fri Queen Elizabeth the Queen Mother Hospital x63829 9.00 – 5.00 Margate CT9 4AN email: [email protected] Fax: 01843 234373

Education Centre Library Tel: 01233 633331 Mon – Fri William Harvey Hospital, Ashford TN24 0LZ x88413 9.00 – 5.00 email: [email protected] Fax: 01233 616789

Understanding Statistics

Statistics – Measures of Effect The statistics in a research article can inform clinical decisions, add detail to a presentation or enable you to assess the evidence for the article’s conclusions

The Basics: , and Mode

Mean (Average): The center point of a set of measurements: sum of all measurements divided by total no. of measurements. Median: For measurements listed in numeric order, the median is the number half-way down the list, simply the mid-way point. Mode: The highest point in a distribution graph. But there can be more than one mode, and is not as good as the mean for mathematical treatment. NOTE: In a symmetrical (normal) distribution, Mean, Median and Mode are the same. In an asymmetrical (skewed) distribution, a simple formula can be used to estimate one, if you know the other two: Mean - Mode = 3 x (Mean - Median)

Odds, Odds Ratios, Risk and

Odds: The odds of an event = the number of events divided by the number of non-events (i.e. odds of a baby boy in 100 births = 51 (boys) ÷ 49 (girls) = 1.04). [Odds > 1 = likely to happen; Odds < 1 = not likely to happen].

Odds Ratio (OR): Odds of an event in treated group divided by odds of event in control group, usually expressed as a decimal proportion. - Epidemiological studies look for events that cause harm – odds ratios > 1 - Clinical trials look for favourable outcomes – odds ratios < one.

Risk (probability): number of events divided by total events (boys = 51/100)

Relative Risk (RR): Risk of an event in treated group divided by risk of event in control group. Easier to interpret than OR, usually expressed as a decimal.

Relative Risk Reduction (RRR): Proportion of risk removed by treatment: ARR divided by initial risk in control group, usually expressed as a percentage

Absolute Risk Reduction (ARR): the risk of an event in the control group minus the risk of an event in the treated group, usually expressed as a percentage

Understanding Statistics

Forest Plot (or ‘blobbogram’):  Shows study outcome on vertical axis vs. on horizontal axis.  Used in systematic reviews to show comparisons between trials.  Shows relative risks (risk ratio or ) for two specific outcomes.

Width of line is 95% CIs for individual studies.

When the 95% CI crosses the zero-effect line, the study outcome is not statistically significant.

Overall relative risk for each outcome and 95% CI.

The size of each square or box is The meta-analysis diamond shows the overall net outcome. Its proportional to the area is proportional to the total number of studies represented, study sample size. and the width proportional to the overall CI. The diamond touching the zero-effect line would show an inconclusive result.

Statistical Significance & P Value A relationship is seen as statistically significant if the probability (P) of obtaining that result by chance is less than 5%. The P value is defined as the probability, under the assumption of no effect or difference (the null hypothesis), of obtaining a result equal to or more extreme than what was actually observed. P values range from 0 (impossible to happen by chance) to 1 (event will certainly happen).

Confidence Intervals (CI) The range within which the true treatment effect is likely to lie. In statistics, 95% confidence something only happens one time in 20 or less. A statistical link at 95% confidence is not proof of a connection. CI are preferable to P Values as they show the range of possible effects compatible with the data. On a Forest plot, a wide CI line shows a small study, so estimates of effect size may be imprecise; a narrow line shows a larger study and more precise estimates of true effect.

Understanding Statistics

Summary of Effect Measures No effect Total success ARR – Absolute Risk Reduction ARR=0% ARR=initial risk RRR – Relative Risk Reduction RRR=0% RRR=100% RR – Risk Ratio / Relative Risk RR=1 or RR=100% RR=0 OR – Odds Ratio OR=1 OR=0 (or ∞) NNT – NNT=∞ NNT=1/initial risk

Quantifying Benefit/Harm

Adverse Outcome/Event Totals Present Absent Intervention Group (exposed to treatment) a . b a+b Control Group c . d c+d (not exposed to treatment) Totals a+c . b+d a+b+c+d

RRR = CER – EER = [a/a+b – c/c+d] CER [ a/a+b]

ARR = CER – EER = a/a+b – c/c+d

Number Needed to Treat (NNT): 1 divided by the proportion of NNT = _1_ treatment group who ‘improve’ following treatment, minus the ARR proportion of control population who ‘improve’ with placebo. 1 = perfect treatment, 20-100 or higher may be ok for prophylactic.

RR = EER = a/a+b CER c/c+d

OR = odds of event in intervention group = a/b = ad odds of event in control group c/d cb

In an RCT or : relative risk (RR) = [a/(a+b)]/[c/(c+d)]

A proposed risk factor acts as a PEER: Patient Expected Event Rate significant risk to disease if the odds (= baseline risk) ratio is greater than 1 and the lower EER: Experimental Event rate bound of the does CER: Control Event rate not go below 1. NNH: NNT: Number Needed to Treat RCT: Randomised Controlled Trial