ANOVA ANalysis Of VAriance Sources of Variance & ANOVA Variance means “variation” • BG & WG ANOVA • Sum of Squares (SS) is the most common variation index – Partitioning Variation • SS stands for, “Sum of squared deviations between each of a –“making”F set of values and the mean of those values” – “making” effect sizes SS = ∑ (value – mean)2 • Things that “influence” F – Confounding So, Analysis Of Variance translates to “partitioning of SS” – Inflated within-condition variability • Integrating “stats” & “methods” In order to understand something about “how ANOVA works” we need to understand how BG and WG ANOVAs partition the SS differently and how F is constructed by each. Variance partitioning for a BG design Called “error” Tx C because we can’t account for why 20 30 the folks in a 10 30 condition -- who were all treated 10 20 the same – have different scores. 20 20 Mean 15 25 Variation among all the participants – Variation among represents variation Variation between the participants within due to “treatment conditions – represents each condition – effects” and “individual variation due to represents “individual differences” “treatment effects” differences” SSTotal =SSEffect +SSError An Example … Constructing BG F & r SSTotal = SSEffect + SSError Descriptives ANOVA SCORE SCORE Mean Square is the SS converted to a “mean” dividing it Sum of N Mean Std. Deviation Squares df Mean Square F Sig. 1.00 10 28.9344 8.00000 Between Groups 605.574 1 605.574 9.462 .007 by “the number of things” 2.00 10 17.9292 8.00000 Within Groups 1152.000 18 64.000 Total 20 23.4318 9.61790 Total 1757.574 19 •MS = SS / df Effect Effect IV dfEffect= k - 1 represents design size SS = SS + SS •MSerror = SSerror / dferror dferror = ∑n – k represents sample size total effect error 1757.574 = 605.574 + 1152.000 F is the ratio of “effect variation” (mean MSIV difference) * “individual variation” F= (within condition differences) r2 = SS / ( SS + SS ) MSerror effect effect error = 605.574 / ( 605.574 + 1152.000 ) = .34 r2 = Effect / (Effect + error) conceptual formula = SSEffect / ( SSEffect + SSerror ) definitional formula 2 r = F / (F + dferror) = F / (F + df ) computational forumla error = 9.462 / ( 9.462 + 18) = .34 Variance partitioning for a WG design Called “error” because we can’t Tx C Sum Dif account for why folks who were in the same 20 30 50 10 two conditions -- who were all treated the 10 30 40 20 same two ways – have different 10 20 30 10 difference scores. 20 20 40 0 Mean 15 25 Variation among Variation among participant’s difference participants – estimable scores – represents because “S” is a “individual differences” composite score (sum) SSTotal = SSEffect + SSSubj + SSError An Example … Descriptive Statistics Don’t ever do this with Constructing WG F & r SSTotal = SSEffect + SSSubj + SSError Mean Std. Deviation N SCORE 28.9344 8.00000 10 real data !!!!!! SCORE2 17.9292 8.00000 10 Mean Square is the SS converted to a “mean” dividing it Tests of Within-Subjects Effects Measure: MEASURE_1 by “the number of things” Type III Sum Source of Squares df Mean Square F Sig. factor1 Sphericity Assumed 605.574 1 605.574 19.349 .002 •MS = SS / df Error(factor1) Sphericity Assumed 281.676 9 31.297 effect effect effect dfeffect = k - 1 represents design size •MS = SS / df Tests of Between-Subjects Effects error error error dferror = (k-1)(s-1) represents sample size Measure: MEASURE_1 Transformed Variable: Average Type III Sum F is the ratio of “effect variation” (mean Source of Squares df Mean Square F Sig. MS Intercept 10980.981 1 10980.981 113.554 .000 difference) * “individual variation” effect Error 870.325 9 96.703 (within condition differences). “Subject F= variation” is neither effect nor error, and MS SStotal = SSeffect + SSsubj + SSerror is left out of the F calculation. error 1757.574 = 605.574 + 281.676 + 870.325 2 r2 = effect / (effect + error) conceptual formula Professional r = SSeffect / ( SSeffect + SSerror ) statistician on a = 605.574 / ( 605.574 + 281.676 ) = .68 = SSeffect / ( SSeffect + SSerror ) definitional formula closed course. 2 = F / (F + dferror) computational forumla Do not try at r = F / (F + dferror) home! = 19.349 / ( 19.349 + 9) = .68 What happened????? Same data. Same means & Std. Same total variance. Different F ??? BG ANOVA SSTotal = SSEffect + SSError WG ANOVA SSTotal = SSEffect + SSSubj + SSError The variation that is called “error” for the BG ANOVA is divided between “subject” and “error” variation in the WG ANOVA. Thus, the WG F is based on a smaller error term than the BG F and so, the WG F is generally larger than the BG F. It is important to note that the dferror also changes… BG dfTotal = dfEffect + dfError WG dfTotal = dfEffect + dfSubj + dfError So, whether F-BG or F-WG is larger depends upon how much variation is due to subject variability Control, variance partitioning & F… What happened????? Same data. Same means & Std. Same total variance. Different r ??? ANOVA was designed to analyze data from studies with… • Samples that represent the target populations r2 = Effect / (Effect + error) conceptual formula • True Experimental designs • proper RA = SSEffect / ( SSEffect + SSerror ) definitional formula • well-controlled IV manipulation = F / (F + dferror) computational forumla • Good procedural standardization • No confounds ANOVA is a very simple statistical model that assumes there The variation that is called “error” for the BG ANOVA is divided are few sources of variability in the data between “subject” and “error” variation in the WG ANOVA. BG SS = SS + SS SSEffect / dfEffect Total Effect Error F = WG SS = SS +SS +SS Total Effect Subj Error SS / df Thus, the WG r is based on a smaller error term than the BG r error error and so, the WG r is generally larger than the BG r. However, as we’ve discussed, most data we’re asked to analyze are not from experimental designs. 2 other sources of variation we need to consider whenever we are working with quasi- or non-experiments are… Between-condition procedural variation -- confounds • any source of between-condition differences other than the IV • subject variable confounds (initial equiv) • procedural variable confounds (ongoing equiv.) • influence the numerator of F Within-condition procedural variation – (sorry, no agreed-upon label) • any source of within-condition differences other than “naturally occurring population individual differences” • subject variable diffs not representing the population • procedural variable influences on within cond variation • influence the denominator of F These considerations lead us to a somewhat more realistic Imagine an educational study that compares the effects of two model of the variation in the DV… types of math instruction (IV) upon performance (% - DV) Participants were randomly assigned to conditons, treated, then allowed to practice (Prac) as many problems as they wanted to before taking the DV-producing test SSTotal =SSIV +SSconfound +SSindif +SSwcvar Confounding due to Practice • mean prac dif between cond Control Grp Exper. Grp Sources of variability… Prac DV Prac DV WC variability due to Practice SS IV • wg corrrelation or prac & DV IV S1 5 75 S2 10 82 SS initial & ongoing equivalence problems confound S3 5 74 S4 10 84 IV SS population individual differences compare Ss 5&2 - 7&4 indif S5 10 78 S6 15 88 SS non-population individual differences wcvar S7 10 79 S8 15 89 Individual differences • compare Ss 1&3, 5&7, 2&4, or 6&8 SSIV +SSconfound / dfIV SSEffect / dfEffect F = ≠ SSindif +SSWCVAR / dferror SSerror / dferror r The problem is that the F-formula will … = F / (F + dferror) • Ignore the confounding caused by differential practice between the groups and attribute all BG variation to the type of instruction (IV) overestimating the SSeffect • Ignore the changes in within-condition variation caused by differential practice within the groups and attribute all WG variation to individual differences overestimating SSerror • As a result, the F & r values won’t properly reflect the relationship between type of math instruction and performance we will make statistical conclusion errors ! • We have 2 strategies: • Identify and procedurally control “inappropriate” sources • Include those sources in our statistical analyses Both strategies require that we have a “richer” model for SS Both strategies require that we have a “richer” model for variance sources – and that we can apply that model to our research! sources – and that we can apply that model to our research! The “better” SS model we’ll use … The “better” SS model we’ll use … SSTotal =SSIV + SSsubcon+ SSproccon + SSIndif+ SSwcsubinf +SSwcprocinf SSTotal =SSIV +SSsubcon+ SSproccon + SSIndif+ SSwcsubinf +SSwcprocinf Sources of variability… In order to apply this model, we have to understand: SSIV IV 1. How these variance sources relate to aspects of … • IV manipulation & Population selection SSsubcon subject variable confounds (initial eq problems) • Initial equivalence & ongoing equivalence SS procedural variable confounds (ongoing eq pbms) proccon • Sampling & procedural standardization SS population individual differences 2. How these variance sources befoul SSEffect, SSerror & F of indif the simple ANOVA model SS within-condition subject variable influences wcsubinf •SSIV numerator SSwcprocinf within-condition procedural variable influences • SS denominator Sources of variability their influence on SSEffect, SSerror & F… SSIV IV “Bigger manipulations” produce larger mean differences • larger SSeffect larger numerator