Psychometric Evaluation of the Twelve Elements Test and Other Commonly Used Measures of Executive Function
Total Page:16
File Type:pdf, Size:1020Kb
Psychometric evaluation of The Twelve Elements Test and other commonly used measures of Executive Function by Claire Surinder Sira B.Sc., University of Victoria, 1994 M.A., Queen’s University, 1997 A Dissertation Submitted in Partial Fulfillment of the Requirements for the Degree of DOCTOR OF PHILOSOPHY in the Department of Psychology © Claire Sira, 2007 University of Victoria All rights reserved. This thesis may not be reproduced in whole or in part, by photocopy or other means, without the permission of the author. ii SUPERVISORY COMMITTEE Psychometric evaluation of The Twelve Elements Test and other commonly used measures of executive function by Claire Surinder Sira B.Sc., Honours, University of Victoria, 1994 M.A., Queen’s University, 1997 Supervisory Committee Dr. Catherine Mateer, (Department of Psychology) Supervisor Dr. Holly Tuokko, (Department of Psychology) Departmental Member Dr. Kimberly Kerns, (Department of Psychology) Departmental Member Dr. Jillian Roberts, (Department of Educational Psychology and Leadership Studies) Outside Member iii ABSTRACT Supervisory Committee Dr. Catherine Mateer, Department of Psychology Supervisor Dr. Holly Tuokko, Department of Psychology Departmental Member Dr. Kimberly Kerns, Department of Psychology Departmental Member Dr. Jillian Roberts, Department of Educational Psychology and Leadership Studies Outside Member Abstract Objective: The Six Elements Task (SET; Shallice and Burgess, 1991; Burgess et al., 1996) measures examinees’ ability to plan and organize their behaviour, form strategies for novel problem solving, and self-monitor. The task has adequate specificity (Wilson et al., 1996), but questionable sensitivity to mild impairments in executive function (Jelicic, et al., 2001). The SET is vulnerable to practice effects. There is a limited range in possible scores, and ceiling effects are observed. This dissertation sought to evaluate the validity and clinical utility of a modification of the SET by increasing the difficulty of the test, and expanding the range of possible scores in order to make it more suitable for serial assessments. Participants and Methods: The sample included 26 individuals with mixed acquired brain injury, and 26 healthy matched controls (20 – 65 years). Participants completed a battery of neuropsychological tests on two occasions eight weeks apart. To control for confounding variables in executive function test performance, measures of memory, working memory, intelligence, substance abuse, pain, mood and personality were included. Self and informant reports of executive dysfunction were also completed. The two groups’ performances on the various measures were compared, and the external iv validity of the 12ET was examined. In addition, normative data and information for reliable change calculations were tabulated. Results: The ABI group exhibited very mild executive function deficits on established measures. The matched control group attempted more tasks on the 12ET, but the difference was non significant. Neither group tended to break the rule of the task. The 12ET showed convergent validity with significant correlations with measures of cognitive flexibility (Trailmaking B and Ruff Figural Fluency), and a measure of planning (Tower of London). The 12ET and published measures were also significantly correlated with intelligence in the brain-injured group. The 12ET did not show divergent validity with a test of visual scanning speed (Trailmaking A). No demographic variables were found to be significant predictors of 12ET performance at Time 2 over and above performance at Time 1, and both participant groups obtained the same benefit from practice. The 12ET did not suffer from ceiling effects on the second administration, and the test-retest reliability of the 12ET variables ranged from low (r = .22 for Rule Breaks in the brain-injured group) to high (r = .78 for Number of Tasks Attempted in the control group). Conclusions: Despite their (often severe) brain injuries, this sample of brain injured participants did not demonstrate executive impairments on many published tests and their scores were not significantly different from the control group’s scores. Therefore, it was not possible to determine if the 12ET was a more sensitive measure of mild executive deficits than the SET. However, the increase in range did reduce the tendency for participants to perform at ceiling levels. The 12ET showed a number of significant correlations with other executive measures, particularly for the brain-injured group, though these correlations may have been moderated by general intelligence. Two variables of the 12ET, deviation from the optimal amount of time per task and Number of Tasks Completed, showed promise as measures of reliable change in this sample over an 8-week interval. v TABLE OF CONTENTS SUPERVISORY COMMITTEE......................................................................................ii Abstract......................................................................................................................... iii Table of Contents ............................................................................................................v List of Tables ...............................................................................................................viii List of Figures ................................................................................................................ix Acknowledgments...........................................................................................................x Introduction.....................................................................................................................1 Intelligence and Executive Function ............................................................................4 Measurement Error......................................................................................................8 Issues in Test-Retest Reliability.............................................................................16 Methodological Procedures to Control for Practice Effects ....................................17 Measuring Change on Neuropsychological Tests...................................................18 Limitations of Reliable Change Formulae..............................................................22 Standardized Regression Based Change Scores......................................................22 Limitations of SRBs ..............................................................................................24 Rationale of the Current Study...................................................................................25 The Six Elements Task ..............................................................................................25 Limitations of the Six Elements Test .....................................................................32 Personality and the Twelve Elements Test.................................................................34 Assessment of Executive Function Through Questionnaires ......................................36 The Current Study .....................................................................................................37 Part 1. Evaluating the Validity of the Twelve Elements Test..................................37 Part 2. Convergent and Divergent Validity of the Twelve Elements Test ...............37 Convergent Validity ..............................................................................................37 The 12ET and Intelligence .................................................................................37 The 12ET and a Measure of Time Estimation ....................................................38 The 12ET and Measures of Cognitive Flexibility ...............................................38 The 12ET and a Measure of Planning ................................................................39 The 12ET and a Measure of Self Monitoring .....................................................40 The 12ET and Self and Informant Reports .........................................................40 The 12ET and Personality..................................................................................41 Divergent Validity .................................................................................................41 The 12ET and Visual Scanning Speed................................................................41 Part 3 Reliability and Normative Data of the 12ET ....................................................41 Method..........................................................................................................................43 Participants................................................................................................................43 Measures ...................................................................................................................47 Procedure ..................................................................................................................52 Results ..........................................................................................................................64 Data Cleaning............................................................................................................64 Part 1 – Evaluating the Validity of the 12 Elements Test............................................66 vi Group Differences on Published Executive Function Tests. ...................................71 Part Two – Convergent