ANOVA ANalysis Of VAriance Sources of Variance & ANOVA Variance means “variation”
• BG & WG ANOVA • Sum of Squares (SS) is the most common variation index – Partitioning Variation • SS stands for, “Sum of squared deviations between each of a –“making”F set of values and the mean of those values” – “making” effect sizes SS = ∑ (value – mean)2 • Things that “influence” F – Confounding So, Analysis Of Variance translates to “partitioning of SS” – Inflated within-condition variability • Integrating “stats” & “methods” In order to understand something about “how ANOVA works” we need to understand how BG and WG ANOVAs partition the SS differently and how F is constructed by each.
Variance partitioning for a BG design Called “error” Tx C because we can’t account for why 20 30 the folks in a 10 30 condition -- who were all treated 10 20 the same – have different scores. 20 20
Mean 15 25
Variation among all the participants – Variation among represents variation Variation between the participants within due to “treatment conditions – represents each condition – effects” and “individual variation due to represents “individual differences” “treatment effects” differences”
SSTotal =SSEffect +SSError An Example … Constructing BG F & r SSTotal = SSEffect + SSError Descriptives ANOVA SCORE SCORE Mean Square is the SS converted to a “mean” dividing it Sum of N Mean Std. Deviation Squares df Mean Square F Sig. 1.00 10 28.9344 8.00000 Between Groups 605.574 1 605.574 9.462 .007 by “the number of things” 2.00 10 17.9292 8.00000 Within Groups 1152.000 18 64.000 Total 20 23.4318 9.61790 Total 1757.574 19 •MS = SS / df Effect Effect IV dfEffect= k - 1 represents design size SS = SS + SS •MSerror = SSerror / dferror dferror = ∑n – k represents sample size total effect error 1757.574 = 605.574 + 1152.000
F is the ratio of “effect variation” (mean MSIV difference) * “individual variation” F= (within condition differences) r2 = SS / ( SS + SS ) MSerror effect effect error = 605.574 / ( 605.574 + 1152.000 ) = .34 r2 = Effect / (Effect + error) conceptual formula
= SSEffect / ( SSEffect + SSerror ) definitional formula 2 r = F / (F + dferror) = F / (F + df ) computational forumla error = 9.462 / ( 9.462 + 18) = .34
Variance partitioning for a WG design Called “error” because we can’t Tx C Sum Dif account for why folks who were in the same 20 30 50 10 two conditions -- who were all treated the 10 30 40 20 same two ways – have different 10 20 30 10 difference scores. 20 20 40 0
Mean 15 25
Variation among Variation among participant’s difference participants – estimable scores – represents because “S” is a “individual differences” composite score (sum)
SSTotal = SSEffect + SSSubj + SSError An Example … Descriptive Statistics Don’t ever do this with Constructing WG F & r SSTotal = SSEffect + SSSubj + SSError Mean Std. Deviation N SCORE 28.9344 8.00000 10 real data !!!!!! SCORE2 17.9292 8.00000 10
Mean Square is the SS converted to a “mean” dividing it Tests of Within-Subjects Effects
Measure: MEASURE_1 by “the number of things” Type III Sum Source of Squares df Mean Square F Sig. factor1 Sphericity Assumed 605.574 1 605.574 19.349 .002 •MS = SS / df Error(factor1) Sphericity Assumed 281.676 9 31.297 effect effect effect dfeffect = k - 1 represents design size •MS = SS / df Tests of Between-Subjects Effects error error error dferror = (k-1)(s-1) represents sample size Measure: MEASURE_1 Transformed Variable: Average Type III Sum F is the ratio of “effect variation” (mean Source of Squares df Mean Square F Sig. MS Intercept 10980.981 1 10980.981 113.554 .000 difference) * “individual variation” effect Error 870.325 9 96.703 (within condition differences). “Subject F= variation” is neither effect nor error, and MS SStotal = SSeffect + SSsubj + SSerror is left out of the F calculation. error 1757.574 = 605.574 + 281.676 + 870.325
2 r2 = effect / (effect + error) conceptual formula Professional r = SSeffect / ( SSeffect + SSerror ) statistician on a = 605.574 / ( 605.574 + 281.676 ) = .68 = SSeffect / ( SSeffect + SSerror ) definitional formula closed course. 2 = F / (F + dferror) computational forumla Do not try at r = F / (F + dferror) home! = 19.349 / ( 19.349 + 9) = .68
What happened????? Same data. Same means & Std. Same total variance. Different F ???
BG ANOVA SSTotal = SSEffect + SSError
WG ANOVA SSTotal = SSEffect + SSSubj + SSError
The variation that is called “error” for the BG ANOVA is divided between “subject” and “error” variation in the WG ANOVA.
Thus, the WG F is based on a smaller error term than the BG F and so, the WG F is generally larger than the BG F.
It is important to note that the dferror also changes…
BG dfTotal = dfEffect + dfError
WG dfTotal = dfEffect + dfSubj + dfError So, whether F-BG or F-WG is larger depends upon how much variation is due to subject variability Control, variance partitioning & F… What happened????? Same data. Same means & Std. Same total variance. Different r ??? ANOVA was designed to analyze data from studies with… • Samples that represent the target populations r2 = Effect / (Effect + error) conceptual formula • True Experimental designs • proper RA = SSEffect / ( SSEffect + SSerror ) definitional formula • well-controlled IV manipulation
= F / (F + dferror) computational forumla • Good procedural standardization • No confounds ANOVA is a very simple statistical model that assumes there The variation that is called “error” for the BG ANOVA is divided are few sources of variability in the data between “subject” and “error” variation in the WG ANOVA. BG SS = SS + SS SSEffect / dfEffect Total Effect Error F = WG SS = SS +SS +SS Total Effect Subj Error SS / df Thus, the WG r is based on a smaller error term than the BG r error error and so, the WG r is generally larger than the BG r. However, as we’ve discussed, most data we’re asked to analyze are not from experimental designs.
2 other sources of variation we need to consider whenever we are working with quasi- or non-experiments are…
Between-condition procedural variation -- confounds • any source of between-condition differences other than the IV • subject variable confounds (initial equiv) • procedural variable confounds (ongoing equiv.) • influence the numerator of F
Within-condition procedural variation – (sorry, no agreed-upon label) • any source of within-condition differences other than “naturally occurring population individual differences” • subject variable diffs not representing the population • procedural variable influences on within cond variation • influence the denominator of F These considerations lead us to a somewhat more realistic Imagine an educational study that compares the effects of two model of the variation in the DV… types of math instruction (IV) upon performance (% - DV) Participants were randomly assigned to conditons, treated, then allowed to practice (Prac) as many problems as they wanted to before taking the DV-producing test SSTotal =SSIV +SSconfound +SSindif +SSwcvar Confounding due to Practice • mean prac dif between cond Control Grp Exper. Grp Sources of variability… Prac DV Prac DV WC variability due to Practice SS IV • wg corrrelation or prac & DV IV S1 5 75 S2 10 82 SS initial & ongoing equivalence problems confound S3 5 74 S4 10 84 IV SS population individual differences compare Ss 5&2 - 7&4 indif S5 10 78 S6 15 88 SS non-population individual differences wcvar S7 10 79 S8 15 89 Individual differences • compare Ss 1&3, 5&7, 2&4, or 6&8
SSIV +SSconfound / dfIV SSEffect / dfEffect F = ≠ SSindif +SSWCVAR / dferror SSerror / dferror
r The problem is that the F-formula will … = F / (F + dferror) • Ignore the confounding caused by differential practice between the groups and attribute all BG variation to the type of instruction (IV) overestimating the SSeffect • Ignore the changes in within-condition variation caused by differential practice within the groups and attribute all WG variation to individual differences overestimating SSerror • As a result, the F & r values won’t properly reflect the relationship between type of math instruction and performance we will make statistical conclusion errors ! • We have 2 strategies: • Identify and procedurally control “inappropriate” sources • Include those sources in our statistical analyses Both strategies require that we have a “richer” model for SS Both strategies require that we have a “richer” model for variance sources – and that we can apply that model to our research! sources – and that we can apply that model to our research! The “better” SS model we’ll use … The “better” SS model we’ll use …
SSTotal =SSIV + SSsubcon+ SSproccon + SSIndif+ SSwcsubinf +SSwcprocinf SSTotal =SSIV +SSsubcon+ SSproccon + SSIndif+ SSwcsubinf +SSwcprocinf
Sources of variability… In order to apply this model, we have to understand:
SSIV IV 1. How these variance sources relate to aspects of … • IV manipulation & Population selection SSsubcon subject variable confounds (initial eq problems) • Initial equivalence & ongoing equivalence SS procedural variable confounds (ongoing eq pbms) proccon • Sampling & procedural standardization
SS population individual differences 2. How these variance sources befoul SSEffect, SSerror & F of indif the simple ANOVA model SS within-condition subject variable influences wcsubinf •SSIV numerator
SSwcprocinf within-condition procedural variable influences • SS denominator
Sources of variability their influence on SSEffect, SSerror & F…
SSIV IV
“Bigger manipulations” produce larger mean differences
• larger SSeffect larger numerator larger F • eg 0 v 50 practices instead of 0 v 25 practices • eg those receiving therapy condition get therapy twice a week instead of once a week
“Smaller manipulations” produce smaller mean differences
• smaller SSeffect smaller numerator smaller F • eg 0 v 25mg antidepressant instead of 0 v 75mg • eg Big brother sessions monthly instead of each week Sources of variability their influence on SSEffect, SSerror & F… Sources of variability their influence on SSEffect, SSerror & F… The influence a confound has on SSIV & F depends upon the “direction of the confounding” relative to “the direction of the IV” SSindif Individual differences • if the confound “augments” the IV SSIV & F will be inflated • if the confound “offsets” the IV SS & F will be deflated More homogeneous populations have smaller within-condition IV differences SSsubcon subject variable confounds (initial eq problems) • smaller SSerror smaller denominator larger F • eg studying one gender instead of both Augmenting confounds • eg studying “4th graders” instead of “grade schoolers” •inflates SSEffect inflates numerator inflates F •eg4th graders in Tx group & 2nd graders is Cx group More heterogeneous populations have larger within-condition • eg run Tx early in semester and Cx at end of semester differences Offsetting confounds • larger SS larger denominator smaller F error • deflates SS deflates denominator deflates F • eg studying “young adults” instead of “college students” Effect •eg2nd graders in Tx group & 4th graders is Cx group • eg studying “self-reported” instead of “vetted” groups • eg “better neighborhoods” in Cx than in Tx
Sources of variability their influence on SSEffect, SSerror & F…
The influence a confound has on SSIV & F depends upon the “direction of the confounding” relative to “the direction of the IV”
• if the confound “augments” the IV SSIV & F will be inflated • if the confound “offsets” the IV SSIV & F will be deflated
SSproccon procedural variable confounds (ongoing eq problems) Augmenting confounds
•inflates SSEffect inflates numerator inflates F • eg Tx condition is more interesting/fun for assistants to run • eg Run the Tx on the newer, nicer equipment
Offsetting confounds
• deflates SSEffect deflates denominator deflates F • eg less effort put into instructions for Cx than for Tx • eg extra practice for touch condition than for visual cond Sources of variability their influence on SSEffect, SSerror & F… Sources of variability their influence on SSEffect, SSerror & F… The influence this has on SS & F depends upon the whether The influence this has on SSError & F depends upon the whether Error the sample is more or less variable than the target pop the procedure leads to more or less within condition variance in the sample data than would be in the population if “more variable” SSError is inflated & F will be deflated • if “more variable” SSError is inflated & F will be deflated if “less variable” SSError is deflated & F will be inflated • if “less variable” SSError is deflated & F will be inflated SS within-condition subject variable influences wcsubinf SSwcsubinf within-condition procedural variable influences More variation in the sample than in the population More variation in the sample data than in the population •inflates SSError inflates denominator deflates F • eg target pop is “3rd graders” - sample is 2nd 3rd & 4th graders •inflates SSError inflates denominator deflates F • eg target pop is “clinically anxious” – sample is “anxious” • eg letting participants practice as much as they want • eg using multiple research stations Less variation in the sample than in the population Less variation in the sample data than in the population • deflates SS deflates denominator inflates F Error • deflates SSError deflates denominator inflates F • eg target pop is “grade schoolers” – sample is 4th graders • eg DV measures that have floor or ceiling effects • eg target pop is “young adults” – sample is college students • eg time allotments that produce floor or ceiling effects
Couple of important things to note !!!
SSTotal =SSIV +SSsubcon+ SSproccon + SSIndif+ SSwcsubinf +SSwcprocinf
Most variables that are confounds also inflate within condition variation!!! • like in the simple example earlier • “Participants were randomly assigned to conditons, treated, then allowed to practice as many problems as they wanted to before taking the DV-producing test” Couple of important things to note !!!
SSTotal =SSIV +SSsubcon+ SSproccon + SSIndif+ SSwcsubinf +SSwcprocinf
There is an important difference between “picking a population that has more/less variation” and “sampling poorly so that your sample has more/less variation than the population”. • is you intend to study 3rd, 4th & 5th graders instead of just 3rd
graders, your SSerror will increase because your SSindif will be larger – but it is a choice, not a mistake!!! • if intend to study 3rd stage cancer patients, but can’t find nd rd th enough and use 2 , 3 & 4 stage patients instead, your SSerror will be inflated because of your SSwcsubinf – and that is a mistake not a choice!!!