THE EFFECTIVENESS OF APPLIED BEHAVIOUR ANALYSIS INTERVENTIONS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER - AN UPDATE OF SECONDARY LITERATURE

Report to the New Zealand Ministry of Health 30 September 2009

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 2

PRINCIPAL AUTHOR

Marita Broadstock

ACKNOWLEDGEMENTS

Acknowledgements

This systematic review was undertaken by staff of the New Zealand Guidelines Group, and completed in September 2009. The scope of the review was agreed with the Living Guidelines Group project’s sponsor, Joanna Curzon (Ministry of Health) and was funded by the Ministry of Health to contribute to the work of the Living Guideline Group. Marita Broadstock (Senior Researcher) conducted the review and prepared the report and Evidence Tables. Margaret Paterson (NZGG Information Specialist) conducted the search strategy and managed document retrieval and referencing. A list of excluded publications is available upon request. Anne Lethaby (Acting Manager, Research Services) provided methodological input and peer review.

Please cite this report as:

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 3

New Zealand Guidelines Group. The effectiveness of applied behaviour analysis interventions for people with autism spectrum disorder – an update of secondary literature. Wellington; 2009.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 4

SYNTHESIS OF RECENTLY PUBLISHED SECONDARY LITERATURE ON APPLIED BEHAVIOUR ANALYSIS

BACKGROUND AND SCOPE

In 2008, the New Zealand Ministries of Health, and Education, commissioned two technical reviews on interventions and strategies for people with Autism Spectrum Disorder (ASD) grounded in the principles of applied behaviour analysis (ABA) (Ministries of Health and Education, 2007). These two reviews were completed in parallel by Uniservices and New Zealand Guidelines Group (NZGG) respectively and based on research published to 31 December 2007. These reports represent the prime evidence for the Living Guideline Group (LGG) to consider in determining whether any revision of Recommendations relevant to ABA is needed in the New Zealand Autism Spectrum Disorder Guideline (2008). However, as this field has been the subject of significant research interest, it was considered valuable to supplement the New Zealand commissioned research with a synthesis of other high level evidence that has emerged since the reviews’ search cut-off period. In a hierarchy of evidence for studies of intervention effectiveness, the highest level evidence (Level I) consists of systematic reviews and meta analyses which include (but are not limited to) randomised controlled trials (RCTs) (level II evidence) (National Health and Medical Research Council, 2008). This current update is limited to level I evidence. Such secondary evidence offers the opportunity to consider a range of approaches to reviewing and synthesising the evidence relating to ABA and compare conclusions with those of the two comprehensive parallel technical reviews.

OBJECTIVES AND RESEARCH QUESTION

The objective of this update is to critically appraise secondary research evidence published since December 2007 relating to the effectiveness of interventions and strategies grounded in the principles of applied behaviour analysis for people with autism spectrum disorder. ABA-based interventions can be defined as ‘those in which the principles of learning theory are applied in a systematic and measurable manner to increase, reduce, maintain and/or generalise target behaviours’ (Ministries of Health and Education, 2007). Well-established principles and techniques of ABA include (a) reinforcement, (b) shaping, (c) chaining, (d) fading, (e) response and stimulus prompting, (f) discrimination training, (g) programming, and (h) functional assessment (Broadstock & Lethaby, 2008). The clinical question employed for this update of secondary literature was that used for the parallel technical reviews (Ministries of Health and Education, 2007): To what extent are interventions and strategies based on the principles of applied behaviour analysis effective in leading to the following outcomes for people with autism spectrum disorders:

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 5

. social development and relating to others . development of cognitive (thinking) skills . development of functional and spontaneous communication which is used in natural environments . engagement and flexibility in developmentally appropriate tasks and play and later engagement in vocational activities . development of fine and gross motor skills . prevention of challenging behaviours and substitution with more appropriate and conventional behaviours . development of independent organisational skills and other behaviours . generalisation of abilities across multiple natural environments outside the treatment setting . maintenance of effects after conclusion of intervention . improvement in behaviours considered non-core ASD behaviours, such as sleep disturbance, self mutilation, aggression, attention and concentration problems.

REVIEW METHODS

A systematic method of literature searching, selection and appraisal was employed in the preparation of this report, consistent with New Zealand Guidelines Group (NZGG) review processes (Broadstock & Lethaby, 2008). The search was limited to articles published in the English language between January 1 2008 and August 6 2009 (when the search was conducted). Sources included general bibliographic databases (Medline, PsychINFO, EMBASE, CINAHL, ERIC, Cochrane Library) as well as various health technology assessment/guideline databases, and cross-checking of references from retrieved references. Selection criteria were based on those used for the NZGG technical review (Broadstock & Lethaby, 2008), modified to solely identify on secondary studies. Studies were included if they: . were secondary research (systematic reviews and meta-analyses) . were published on or after January 1st 2008 . had a clear review question . used at least two searching sources and reported on studies: . of eligible interventions (studies which evaluated interventions which were predominantly based on the principles of applied behaviour analysis) . considering comparators including usual care, another intervention or application of interventions . considering individuals with a diagnosis of autism spectrum disorder or where results are reported separately for this group. Research papers were excluded if they:

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 6

. were non-systematic reviews, correspondence, editorials, expert opinion articles, comments, case reports, book chapters, articles published only in abstract form, conference proceedings, news items, unpublished work . were primary studies (except where reported in included systematic reviews . did not provide separate analyses/syntheses of results for eligible interventions and eligible population . were not deemed appropriate to the research question or nature of review, including those reporting on outcomes solely relating to safety; the acceptability of, or ethical, economic or legal considerations relating to ABA; or the impact on persons other than those diagnosed with ASD.

STUDY SELECTION AND APPRAISAL

Selection criteria were applied by a single reviewer to abstracts/titles identified by the search strategy, and again to those retrieved as full text, to identify the final set of included papers for critical appraisal. All included studies represented level I evidence (National Health and Medical Research Council, 2008) as specified in the selection criteria, being systematic reviews or meta analyses of studies that included randomised controlled trials. Included studies were formally critically appraised using NZGG’s quality checklists which are based on the GATE Frame tools designed by Effective Practice, Informatics and Quality improvement (EPIQ), within the School of Population Health at the University of Auckland. Twenty questions addressed methodological quality, including whether the review’s methods were internally valid, precision of results, and the applicability/external validity of the review. An overall quality code was applied based on the appraisal to summarise the quality of each included study. Overall study quality was categorised qualitatively based on summary questions on the GATE checklist and using one of the following descriptors: “very good”, “good”, or “fair”. Details of each study were entered into Evidence Tables, including aspects of methodology, results, authors’ conclusions, reviewer’s additional comments, and the summary study quality descriptor. The included studies are presented alphabetically by first author within two intervention type categories, (1) applied behaviour analysis interventions (defined broadly), and (2) Picture Exchange Communication System (PECS) interventions (see Evidence Tables).

DATA SYNTHESIS

Studies were narratively synthesised to determine the strength of evidence. Strength of evidence is determined by three domains (West, King, & Carey, 2002): - quality (the extent to which bias was minimised); - quantity (magnitude of effect, numbers of studies, sample size or power); - consistency (the extent to which similar findings are reported).

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 7

KEY FINDINGS AND CONCLUSIONS

Overview

Of 461 articles identified by the search strategy, 35 publications were retrieved as full text. Twelve of these met selection criteria and were deemed eligible for critical appraisal and inclusion in this synthesis. Of the 12 included secondary studies, 10 reported on evaluations of interventions broadly based on applied behaviour analysis, and two reported on evaluations of the Picture Exchange Communication System (PECS) intervention specifically (see Table). The 10 reviews reporting on ABA-based interventions (in general) used varying descriptive terms for included interventions, reflecting that there is no universally accepted definition of ABA-based interventions. For example, some studies refered specifically to Early Intensive Behavioural Interventions (EIBI) as being restricted to interventions based on the work of Lovaas (1987), whereas others included as EIBI any intensive behavioural intervention directed at children in pre-school years. In this report the broader definition is used. Of the 10 reviews, four included meta analyses (Sigmund Eldevik et al., 2009; Ospina et al., 2008; Reichow & Wolery, 2009; Spreckley & Boyd, 2009b); all four were rated as being of good (Reichow & Wolery, 2009) or very good quality (Sigmund Eldevik et al., 2009; Ospina et al., 2008; Spreckley & Boyd, 2009b). Four of the 12 reviews (33%) were classified as being of “very good” quality (Blue Cross and Blue Shield Association, 2009; Sigmund Eldevik et al., 2009; Ospina et al., 2008; Spreckley & Boyd, 2009b), another third were of “good” quality (Svein Eikeseth, 2009; Howlin, Magiati, & Charman, 2009; Reichow & Wolery, 2009; Rogers & Vismara, 2008), and the remaining final four systematic reviews were of “fair” quality, including two considering ABA in general (Case-Smith & Arbesman, 2008; Seida et al., 2009), and two reporting on PECS (Ostryn, Wolfe, & Rusch, 2008; Sulzer-Azaroff, Hoffman, Horton, Bondy, & Frost, 2009). It should be noted that as these reviews were all published across an 18 month period on the same broad topic area, they tend to report on the same studies, and also overlap with the studies included in the parallel technical reviews of NZGG and Auckland Uniservices. For this reason it is important to be aware that the results and the studies appraised should not be summated as independent sources of evidence as this would misrepresent the “quantity” of studies and give the individual studies undue weight. No additional primary studies published post December 2007 and eligible for inclusion in either of the two technical reviews were identified by the 12 reviews, and so the evidence “catchment area” is broadly the same as that considered by the parallel technical reviews, although some included studies prior to 1998, the earliest date for publications considered in the parallel technical reviews. In considering secondary reviews which consider overlapping but different selection criteria and methodological approaches, attention should be given to the consistency of

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 8 conclusions. More weight/emphasis should be placed on the reviews that are of higher quality. For this reason, the following synthesis will consider the findings and conclusions of the “very good” quality reviews first.

Reviews of “very good” quality

Four reviews were rated as being of “very good” quality using the systematic critical appraisal process outlined above. All considered ABA interventions in general, though the focus was on studies of early intensive behavioural analysis. The systematic review published by Blue Cross and Blue Shield Association’s (2009) Technical Evaluation Centre considered evidence published 1987 - July 2008. The review considered the use of EIBI based on applied behaviour analysis. It excluded single-case experimental studies (SCED), arguing that such designs lack generalisability across individuals, and are of limited value in evaluating EIBI due to methodological constraints. The reviewers included 16 studies, including 2 RCTs. They concluded that, “weaknesses in research design, differences in the treatments and outcomes compared, and inconsistent results, mean that the impact of EIBI versus other treatments on outcomes for children with autism cannot be determined.” They argued that the weakness of the evidence base and variability in findings meant that they could also not reach conclusions about the impact of EIBI on any specific domains/outcomes, or on whether greater intensity of EIBI can lead to better outcomes. The authors called for research to identify what aspects of an intervention and its delivery lead to greatest positive effect, including content, technique, intensity, starting and ending age. A greater emphasis was said to be needed on randomized, controlled trials with substantially larger sample sizes, uniformity of outcomes and instruments, and consistent treatments. The systematic review by Eldevik and Hastings (2009) considered nine controlled group studies of EIBI in their meta analyses. They reported moderate to large effect sizes for adaptive behaviour composite (ABC) and full-scale IQ scores, respectively, compared with controls or eclectic comparators. It is concluded that, “in the absence of other interventions with established efficacy, EIBI should be an “intervention of choice for children with autism.” However the authors cautioned that the improved outcomes for people receiving EIBI may be due to the EIBI group having received, in general, more frequent and more total hours of supervision and training than comparison groups. They noted that this remains a threat to the validity of their conclusions and called for RCTs where the comparison intervention is of similar intensity and where staff receive similar training and supervision. Another SR with meta analysis rated as being very good quality was authored by Ospina et al (2008). It considered literature published to May 2007 and included 101 controlled studies across a range of interventions ranging from behavioural to developmental. Those interventions relating to ABA specifically included those based on Direct Trial Training (DTT) and Lovaas (1987) therapy. With respect to ABA, the

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 9 reviewers concluded from descriptive synthesis that there was “some support” for discrete trial training in terms of motor and functional skills but not for communication skills. From meta analyses of 3 controlled clinical trials the authors reported that Lovaas’ therapy appeared to show benefits when compared to special education on several domains (adaptive behaviour, communication and interaction, comprehensive language, daily living skills, expressive language, overall intellectual functioning, socialisation) with the exception of non-verbal intellectual functioning. Based on a meta-analysis of two RCTs, they found no difference in communication skill outcomes for children receiving EIBI compared with a Developmental Individual Difference Relationship based intervention. Lower level evidence from meta-analysis of two retrospective cohort studies showed greater effects for “high intensity” versus “low intensity” Lovaas. These findings were tempered by the limitations of the evidence base, with the authors noting that, “these findings are based on pooling of a few, methodologically weak studies with few participants and relatively short-term follow-up”. The heterogeneity of the study populations, interventions and outcomes was also noted, raising questions as to the appropriate use of meta analysis to synthesise these results. The overall conclusion from across interventions was that there is no clear answer regarding the most effective therapy to improve symptoms associated with ASD. The authors called for replication in RCTs to substantiate the use of Lovaas (1987) and to assess the effect of treatment intensity on outcomes of children with ASD. They further recommended that clinical management be guided by individual needs and availability of resources. The fourth review of very good quality also included a meta analysis (Spreckley & Boyd, 2009b), and reported on applied behaviour interventions (ABI) for preschool children with ASD. Randomised or quasi-randomised trials were included, and of the 13 studies meeting initial selection criteria, only six met minimum quality criteria and four had adequate data to be included in the formal meta analysis. Results suggested no significant improvement on outcomes for those in ABI programs compared with standard care on cognitive outcomes, expressive language, receptive language, or adaptive behaviour. Low to moderate heterogeneity existed across outcomes which prompts uncertainty about the suitability of meta analytic techniques here, a limitation the authors acknowledged given high variability in participant characteristics and programme content. Other limitations of the evidence base disussed by the authors included difficulty in establishing genuine control groups, lack of strict selection criteria, and limited information on participant retention. The authors concluded that there is inadequate evidence that applied behaviour intervention has better outcomes than standard care for children with autism.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER Summary These high quality reviews offer some agreement and some divergence in their interpretations of the evidence base. Two reviews, which included meta analyses, gave tentative evidence of positive benefits of ABA-based interventions over comparison treatments from a limited evidence base. EIBI was reported as having beneficial effects on adaptive behaviour and IQ compared with control or eclectic comparators (Sigmund Eldevik et al., 2009). Some support was reported for discrete trial training in terms of motor and functional skills but not for communication skills. Lovaas’ (1987) therapy was presented as having benefit over no treatment, and greater effect for higher intensity treatment (Ospina et al., 2008). However these conclusions are qualified by methodological weaknesses of the individual studies and the small number of studies and participants. In particular, the lack of control for the intensity of treatment delivered between intervention and comparator is problematic. In contrast, the two other reviews suggested there was inadequate evidence of effectiveness of ABA-based early interventions. These conclusions were based on a lack of significant results in the meta analysis of four studies by Spreckley and Boyd (2009b). The Blue Cross and Blue Shield Association (2009) report argued that the limitations and inconsistent results of the evidence base precluded making any conclusions about the relative effectiveness of EIBI compared with alternative treatments, about its effectiveness for particular outcome domains, or about the impact of intensity on outcomes. The report also argued that study heterogeneity ruled out the use of meta analysis. All four reviews emphasised the need for additional research addressing methodological weaknesses, and suggested the need for appropriately powered RCTs to systematically investigate what characteristics of treatment lead to the best outcomes. Particularly called for was the use of comparison interventions of similar intensity and quality to those of EIBI.

Reviews of “good” quality

Four reviews were rated as being of “good” quality, having more flaws and limitations than the “very good” reviews discussed above, often in terms of a limited search strategy, lack of systematic quality assessment, and/or lack of detail and precision in reporting study characteristics and synthesising results. However these studies are still of reasonably good quality and of value in contributing to the evidence base. A systematic review of comprehensive psycho-educational research (Eikeseth, 2009) included 20 studies on ABA-based interventions. The reviewers concluded that intensive ABA treatment carried out by trained therapists is effective in enhancing “global functioning” in pre-school children with autism, and those with PDD-NOS. In Howlin et al’s (2009) review of 11 controlled (any comparator) group EIBI studies, in 9 of 10 studies, IQ scores were significantly higher for children at follow-up in the

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 11

EIBI group compared with those in the control/comparator group, with moderate to large effect sizes in the majority, but not all studies. However the authors also noted that children receiving EIBI received significantly more (mean) hours of intervention per week than those in the comparison group (30 c.f. 19 hours, respectively). It was concluded that whilst EIBI is highly effective for some children, there is wide individual variation in response, and some failed to make progress at all. Howlin et al (2009) also noted that chronological age and length of intervention did not appear to be related to outcome, with varying results for initial language level as a predictor. Initial IQ strongly predicted improvement after EIBI treatment in four studies, but was unassociated with outcome in a fifth. A systematic review and meta analysis by Reichow and Wolery (2009) evaluated 13 controlled group studies on EIBI interventions based on the Lovaas (1987) model (notably, Eldevik and Hastings, 2009, criticised this as too restrictive a definition for a review of EIBI’s effectiveness). Meta analyses of 12 studies considering change in IQ within treatment group suggested a moderately large effect size for improvement at followup, however the authors cautioned that this finding was limited by excluding reference to a comparison group. As such, improvements cannot be attributed to EIBI exclusively; for example, they could relate to maturation effects. As there was significant variability between study characteristics, the appropriateness of using meta analytic techniques for synthesising results is questionable. Descriptive and summative syntheses suggested that EIBI can be an effective treatment, on average, for some children with autism compared with control/various comparators. However individual data typically were not presented and the authors noted that the intervention has not worked for all children, and gains in one domain may not be accompanied by gains in another domain within individuals. Moreover, comparison groups were of limited quality and poorly described. The authors concluded that it is not possible to determine whether EIBI is more or less effective than other treatment options without comparisons between EIBI and empirically validated treatment programs. The systematic review by Rogers and Vismara (2008) considered early interventions including psychopharmacological ones for young children with autism, reporting on seven ABA based interventions. From four controlled studies evaluating Lovaas’s (1987) treatment, the authors argued that this intervention meets criteria for a ‘‘well-established’’ psychosocial intervention for improving the intellectual performance (specifically IQ scores) of young children with ASD. “Well established” interventions were defined as those requiring treatment manuals, clearly specified participant groups, and either (a) two independent well-designed group studies showing the treatment to be better than placebo or alternative treatment, or equivalent to an established effective treatment, or (b) nine or more single-subject design studies using strong designs and comparison to an alternative treatment (Chambless et al., 1998). There was less consistency in the data for improvements in behavior, adaptive skills, and language skills. Overall, the review concluded that focused daily early intervention programs of several different kinds are beneficial for young children with autism, however the lack of comparative studies precludes determining which comprehensive treatment approach is best. Further, the authors

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 12 suggested that other well-known interventions may be as or more efficacious as Lovaas’s (1987) model but they have not been rigorously evaluated.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER Summary The review by Eikeseth (2009) concluded that intensive ABA treatment carried out by trained therapists is effective in enhancing global functioning in pre-school children with autism, and those with PDD-NOS. Rogers and Vismara (2008) suggested that Lovaas-based (1987) EIBI met criteria for a ‘‘well-established’’ intervention for improving the IQ of young children with ASD, but that there was less consistency for improvements in behavior, adaptive skills, and language. Howlin et al (2009) and Reichow and Wolery (2009) both concluded that whilst EIBI is highly effective in increasing IQ for some children, some appear to make no progress, suggesting wide individual variation in response. The former reviewers also found that age and length of intervention do not appear to predict response, but that initial language level did in some studies, and initial IQ appears to be related to impact on outcome in most studies evaluated. Despite these broadly positive conclusions about the potential for benefit among (at least some) children with ASD, the reviewers tempered their findings with reference to the methodological limitations of the evidence base. Major concerns centred around the need to disentangle the effects of programme content from those of programme intensity, which has not been well controlled for. To address this, Howlin et al (2009) called for equivalence trials, where high quality interventions are compared witho other high quality interventions of a similar intensity. Other calls for further research echoed those of the “very good” studies reported earlier, including the need to systematically assess characteristics of children for whom EIBI is most effective, optimal duration and intensity of treatment, and age of commencement of treatment, so that better targetted and cost-effective interventions could be developed (Howlin et al., 2009). Another repeated theme concerned the limitations of the comparator or control groups with which ABA based interventions are compared. These included their lack of standardization, being poorly defined, lack of measures of procedural fidelity, and lack of data on any supplemental treatments participants may also receive. The apparent superiority of EIBI may be an artifact of the lack of rigorous empirical evaluation of alternative interventions of good quality. As fully randomised controlled studies in this area were extremely rare, the nonrandom assignment of participants to treatment groups was also said to severely limit conclusions about treatment effectiveness. This is because factors affecting group allocation (such as parental chioce) may have contributed to improved outcomes (such as increased parental motivation and support), thus confounding any effects attributed to the treatment. Only one review (Rogers & Vismara, 2008) made specific reference to cultural factors, observing that treatment programmes for children with autism have been developed for and evaluated primarily with children from European American backgrounds, and generalization of effects across ethnic groups from this research is premature. It was suggested that cultural issues may moderate the effects of autism intervention programs.

Reviews of “fair” quality

Four reviews were rated as being of fair quality, two considering ABA interventions broadly defined, and the only two reviews considering a specific ABA based intervention, both relating the the Picture Exchange Communication System. These intervention categories will be considered separately, as they are in the Evidence Tables.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER Applied Behaviour Analysis interventions in general Two reviews on ABA interventions were classified as being of “fair” quality due to flaws in methodology and reporting such that results are likely to be susceptible to bias. It should be noted that some of these studies had supplementary or alternative goals to providing comprehensive systematic reviews of the literature. Whilst the narrative critiques of the literature in these reviews may be of interest, conclusions about the evidence base from these studies may not be reliable. The review by Case-Smith and Arbesman (2008) reviewed ASD interventions relevant to occupational therapy. The authors concluded that behavioural interventions are successful in training children in basic academic and life skills, but noted that generalisation of skills to natural environments was unproven. They also commented on the lack of studies into the promotion of effective transition to work and independent living. Discussing barriers to widespread application of behavioural interventions, the reviewers pointed to the time and resources required to implement a 30- to 40-hour/week intervention as well as the lack of definition of the most appropriate candidates. Seida et al (2009) conducted an “umbrella review” of systematic reviews of psychosocial interventions for people with ASD. Broadly favourable outcomes from behavioural interventions versus no treatment were reported. Critiquing the literature, the authors noted the absence of data on the relative effectiveness of treatment options, and the lack of research attention to factors in choice of treatment including cost, convenience and family burdens.

Picture Exchange Communication System Two systematic reviews of fair quality specifically considered interventions based on Picture Exchange Communication System (PECS). The review by Ostryn et al (2008) considered 15 PECS studies concluded that whilst PECS is widely implemented it lacked a strong empirical base. Criticisms of the literature included lack of reporting of statistical significance and precision of results, lack of scope and definition of key outcomes, and lack of data on maintenance. The authors recommended that PECS was best used as an initial intervention to teach manding as part of a multimodal system but is not recommended as a long term intervention. The other PECS review by Sulzer-Azaroff et al (2009) concluded from 34 studies that professionals and parents can teach individuals to successfully initiate exchanges of pictures for tangible and non tangible reinforcers.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER REFERENCES

The references for this report and the appended Evidence Tables are presented below. Publications included for critical appraisal in this review are indicated by ** prefacing the citation.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 16

EVIDENCE TABLES

Evidence tables for included Level I secondary studies assessing the effectiveness of applied behaviour analysis (ABA) for people with autism spectrum disorder. 1. Applied Behaviour Analysis interventions in general

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design Blue Cross Evidence level: I Inclusion criteria: 16 studies met selection criteria, including 2 RCTS, 9 nonrandomized, . Clearly defined Blue Shield . reported on the use of comparative studies, and 5 single-arm studies. No studies were found that methodology though (2009) Aim: EIBI compared to another included children only two databases Technology To conduct a systematic treatment strategy; with Asperger’s disorder; 4 studies explicitly included children with PDD or considered Evaluation review of the research . attempted to identify PDD-NOS. . Study quality coded and Centre literature on the use of features of EIBI that had rated, apparently by early intensive the most impact on its . The strongest evidence was provided by two randomized, controlled trials single reviewer. US behavioural interventions effectiveness; or (Smith et al. 2000; Sallows and Graupner 2005) and they compared . Justification given for based on applied . sought to identify children different interventions, had small sample sizes, and came to different excluding SCED studies SR behavioural analysis most likely to benefit from conclusions. including lack of (EIBI) among young EIBI. . 3 of 4 studies examining the impact of pretreatment cognitive functioning generalisability across children with ASD. Asked: found that it significantly predicted outcomes, while one (a randomized, individuals, inability to (1) How effective is EIBI Exclusion criteria: controlled trial) did not. Some studies suggested that younger age at the use blinding, variability in improving the . sample size <10, start of therapy is a predictor of better outcomes (e.g., Howard et al. of the ASD condition functioning of children including single-subject 2005), while others found no difference based on initial age (e.g., Magiati over time, and potential with autism spectrum studies; et al. 2007). for carryover effect disorders, and how does it . interventions were very . The findings on whether more intense treatment leads to better outcomes between intervention compare to other early poorly described; were inconsistent, and no conclusions were drawn. and comparator (given intervention approaches? . interventions were not that interventions are (2) Can patient comprehensive, where a Authors’ conclusions: intended to have long characteristics be number of domains . Unfortunately, only two randomized, controlled trials have been lasting effects) identified that predict affected by ASD were conducted. Weaknesses in research design, differences in the treatments . Comprehensive Tables better outcomes from addressed; and outcomes compared, and inconsistent results mean that the impact of including details of EIBI? . intervention within a EIBI versus other treatments on outcomes for children with autism cannot study characteristics, (3) Does the effect of EIBI treatment group was be determined. outcomes, results, and vary with the intensity of heterogeneous, . The body of evidence overall is too weak to reach conclusions regarding aspects of quality. treatment? combining a variety of any of the domains/outcome areas, and too variable to assess whether . Thorough critical methods; greater intensity of EIBI can lead to better outcomes. discussion of included Search period: . experimental intervention . The heterogeneity of the interventions used in studies on EIBI and the studies and of recent From 1966 (varied, see was not intensive (< 20 significant methodological weaknesses preclude the use of meta-analysis systematic reviews below) – July 2008. hrs/wk); and of reaching any general conclusions about the effectiveness of EIBI . Discussion of the A more narrowly defined . study did not directly among children with ASD. methodological update was performed in measure outcomes . About half of the studies followed children for approximately 2 years or limitations of studies January 2009 using through a direct less, and some for only 1 year. This is not sufficient follow-up time to . Suggestions made for Medline but identified no assessment of the child’s assess the potential impact of an intervention over a lifetime. future research additional eligible studies. achievement; . Research is needed to identify those characteristics of treatment - content,

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 17

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design Note that selection criteria . study was technique, intensity, starting and ending age, etc. - that maximize its Overall quality: Very good then excluded studies published before effectiveness. (Need) a greater emphasis on randomized, controlled trials; published <1987. 1987, when the substantially larger sample sizes; uniformity of outcomes and instruments; seminal article on and consistent treatments that do not vary widely within the experimental Search strategy: EIBI by Lovaas or control group. Medline (from 1966) was released. . The cost of continuing the current course of assuming that EIBI works may PsycINFO (from 1970) not be obvious. EIBI is costly financially for society and requires a large (search terms provided). Study quality was appraised time commitment from children, their families, and their teachers or and graded (as good, fair or therapists. However, these programs may not appear to pose any harm References of retrieved poor) using detailed criteria for the children themselves. Nevertheless, the opportunity costs could be papers were cross and formal grading systems. high, indeed, of providing suboptimal care to these children, simply checked to identify because we as a society do not know what works best. The children may additional papers. be treated with an intervention that is not as effective as the alternatives.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 18

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design Case-Smith Evidence level: I Inclusion criteria: 49 studies met selection criteria. Six categories of research topics were . Search strategy & Arbesman . Provided evidence identified, including intensive behavioral intervention, which is reported on here. restricted to databases, (2008) Aim: for an intervention no search terms given. To identify, evaluate, and approach used with . EIBI using one-on-one discrete trial training is widely applied to children with . No Tables USA synthesize the research children or ASD and has evidence of moderate to strong effects (Lovaas, 1987; Sallows . Narrative discussion of literature on interventions adolescents with & Graupner, 2005; Smith, Groen, & Wynn, 2000). results with minimal SR for ASD of relevance to ASD; . Positive behavioral support has moderate to strong positive effects in methodological critique occupational therapy. . Peer reviewed; reducing problem behaviors in children with ASD. A systematic approach and minimal detail of . Addressed a that includes prevention of problem behaviors by applying consistent quantitative results. Search period: performance area instruction and consequences to behavior, modifying the environment to . Framed selection and 1987 –2007 or intervention promote appropriate behaviors, and collecting data to monitor children’s discussion of studies in approach within the progress appears to be highly effective (Horner et al., 2002). terms of relevance to Search strategy: domain of practice of occupational CINAHL occupational Authors’ conclusions: therapists. Cochrane SRs database therapy; . Although behavioral interventions are successful in training children in ERIC . RCTs, non basic academic and life skills concepts, it is not known how well these skills Overall quality: Fair Medline randomized clinical transfer into the natural environment, and the findings are mixed regarding PsychINFO trials, and before- changes in behavior (Smith et al., 2000). Barriers to widespread application Social Science Abstracts and-after, one are as follows: (1) time and resources required to implement a 30- to 40- Sociological Abstracts group designs. hr/wk intervention and (2) definition of the most appropriate candidates RehabData (Bassett, Green, & Kazanjian, 2000). Latin American and Exclusion criteria: . When children exhibit problem behaviors, functional analysis is essential to Caribbean Health . Case determine the basis for the behavior. Sciences Literature series, single subject . “Level I” studies of interventions to promote adolescents’ and young adults’ EBSCOHost design, case reports and success in work and independent living were virtually absent from the (search terms not expert opinion, narrative research literature. provided). reviews, consensus statements; . Used qualitative methods to the exclusion of quantitative methods; . Had serious design limitations.

Single reviewer extracted data. Used a hierarchy of evidence based on study design to rank studies.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 19

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design Howlin et al. Evidence level: I Inclusion criteria: 11 studies (reported in 13 reports) met selection criteria, including one RCT . Clearly defined (2009) . Case-control comparison (Smith et al, 2000). methodology and Aim: study of EIBI for children comprehensive search UK To examine the findings with autism; Intervention intensity strategy from controlled early . UCLA affiliated . The mean intervention period for the 9 studies which reported . No rating of study SR intensive behavioural (home or clinic intensity clearly was 27.4 months (SD=10.7, range 14-48 mths). quality intervention (EIBI) studies based) or UCLA- . Intensity (number of hours of intervention per week) varied greatly by . Detailed Tables and published in peer modeled (home- individual within study. Estimated that EIBI children received significantly discussion of results reviewed journals. based) EIBI more hours of intervention per week than controls (EIBI Mean=29.8 hrs/wk . Good attempt to draw program; cf Control Mean=19.1 hrs/wk, p=.007). out patterns of results Search period: . At least 10 people and explain findings. 1985 – May 2007 in EIBI group; Impact of intervention on IQ . Discussion of the . age < 6 years at IQ did not vary at baseline between treatment groups, but at final follow up methodological Search strategy: start of treatment; was significantly higher for EIBI group compared to the Control group in 9 of limitations of included CINAHL . intervention at least 11 studies, with estimated effect sizes moderate to large in the majority of studies Cochrane library 12 hrs/wk; studies, but not all studies. . Suggestions made for ERIC . intervention for 12 future research Medline mths or longer Variability in outcome PsychINFO . adequate data on There was considerable variation between individuals, with a minority Overall quality: Good Embase IQ or standard achieving marked improvement and even educational independence, whilst (search terms provided). outcome measures. the majority had less dramatic improvements. Some failed to make progress at all. Searched websites of Exclusion criteria: several major health, . SCED or case Predictors of outcome autism and research series study without . Length of intervention did not appear to be related to outcome, and organisations. References comparison group, most impact on IQ occurred in the first year of the intervention after which of review papers were studies focused on a the effect tended to plateau. cross checked to identify specific behaviour, use of . Initial (baseline) IQ strongly predicted improvement in 4 studies, but was additional papers. pharmacological or non unassociated in another study that investigated this. ABA interventions, . Chronological age did not predict outcome (though age range tended to outcomes for/views of be limited to 3-5.5 yrs) which does not support suggestions that EIBI must therapists or parents. start at 3 years. . Mixed results were found for initial language level which was associated Summaries/analyses based on with outcome in 4 of 7 studies investigating this. comparisons of the published group means. Methodological limitations . Comparisons were made difficult by varying time lag of follow-up after ending of intervention, and use of different instruments and scoring reported (raw, standard, age equivalents). . Few researchers have assessed outcome in terms of behavioural difficulties/severity of autism, or impact on family life. . Few studies reported on quality of alternative therapies in the Control arm, which tended not to be autism specific.

Authors’ conclusions: . There is little question that EIBI is highly effective for some children, however gains are not universal. . Crucial need to systematically assess for which children is EIBI most and

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 20

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design least effective. Clearer evidence concerning the optimal duration of therapy and the age at which is should begin could result in the development of better targeted, more cost effective programmes. . Need for more RCTs, and need for equivalence trials, where high quality interventions are compared with each other, rather than comparing high quality interventions with low quality/low intensity interventions. Failure to control for time in intervention means that for some studies, EIBI may be more effective due to differences in intensity, not quality.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 21

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design Eikeseth Evidence level: I Inclusion criteria: 25 studies met selection criteria, including 20 which evaluated interventions . Comprehensive search (2009) . study was based on Applied Behavior Analysis (ABA) treatment, 3 studies evaluated strategy, but no search Aim: published in a peer- Project TEACCH, and 2 studies evaluated the Colorado Health Sciences terms or dates given to Norway Evaluates comprehensive reviewed journal; Model. Only those relating to ABA are discussed here. allow replication psycho-educational . children had a . Very detailed Tables, SR research on early mean age of 6 Scientific merit describing studies, intervention for children years or less at . 1 ABA study, an RCT, received Level 1 scientific merit (Smith, Groen, & interventions, and with autism. intake; Wynn, 2000) discussion of results . participants . 4 ABA studies received Level 2 rating (Cohen, Amerine-Dickens, & . Lacks detail on Search period: received Smith, 2006; Eikeseth, Smith, Jahr, & Eledevik, 2002, 2007; Howard, participant Not reported comprehensive Sparkman, Cohen, Green, Stanislaw, 2005; Remington et al., 2007) characteristics at psycho-educational . 9 received Level 3 evidence support baseline. Search strategy: interventions; . 6 ABA outcome studies were classified as having insufficient (level 4) . Discussion of the ERIC . studies contained scientific value. methodological Medline outcome data. limitations of included Psyclit Magnitude of treatment effects studies (search terms not Comprehensive psycho- . 4 ABA studies received Level 1 rating demonstrating that children . Suggestions made for provided). educational interventions were future research receiving ABA made significantly more gains than control group children defined as interventions directions on standardized measures of IQ, language and adaptive functioning References of “recent addressing social behaviors, . Insufficient quantitative (Cohen et al., 2006; Eikeseth et al., 2002, 2007; Howard et al., 2005; papers” were cross communication and information on the Sallows & Graupner, 2005). Several studies also included data on checked to identify ritualistic/stereotyped extent of the gains maladaptive behavior, personality, school performance and changes in additional papers. behaviors. experienced, the clinical diagnosis. meaningfulness of these Researchers known in the Exclusion criteria: . 3 ABA studies received Level 2 rating (Eldevik et al., 2006; Lovaas, 1987; gains and whether there field were contacted by None reported. Smith, Groen, & Wynn, 2000), demonstrating that ABA treated children were subgroups of email and asked for made significantly more gains than the comparison group on one children who did not relevant articles recently Outcome studies were graded standardized measures of IQ or Adaptive Functioning. improve. published or “in press”. according to their (i) scientific . 5 ABA studies received Level 3 rating. value and (ii) according to the Overall quality: Good magnitude of results Authors’ conclusions: documented in the studies. Recommended practice parameter . ABA treatment is demonstrated effective in enhancing global functioning Scientific Merit was evaluated in pre-school children with autism when treatment is intensive and carried based on: (a) diagnosis, (b) out by trained therapists. study design, (c) dependent . ABA treatment is demonstrated effective in enhancing global functioning variables and (d) treatment in children with PDD-NOS (one Level 1 study; Smith, Groen, & Wynn, fidelity. Four levels were used. 2000). Guideline practice parameter . ABA can be effective for children who are up to 7 years-of-age at intake (one Level 2 study; Eikeseth et al., 2002, 2007).

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 22

Reference, Evidence level, aim and Inclusion and exclusion criteria Results and authors’ conclusions Comments and quality country, search method score design Eldevik et al Evidence level: I Inclusion criteria: 9 controlled designs met selection criteria, including one RCT . Moderately broad (2009) . intervention generally adhered to (Smith et al, 2000). Howard et al (2005) included both a search strategy Aim: defined common EIBI elements; Control and Comparison group and so 10 effect sizes were . Excluded Sallows and Wales & To conduct a systematic . aged between 2-7 years at intake; calculated. Four studies had a Comparison group, and six had Graupner (2005) as it Norway literature search for . diagnosed with autism, PDD NOS; a Control group. compared two EIBI studies reporting effects . full-scale measure of intelligence groups of somewhat SR/MA of Early Intensive and/or standardized measure of The average effect size was 1.10 for change in full-scale different intensity Behavioral Intervention adaptive behaviour at intake and intelligence (95% CI .87, 1.34) and .66 (95% CI=.41, .90) for varying in programme (EIBI). Also aimed to post intervention; change in adaptive behavior composite. These effect sizes are delivery (clinic- or replicate and extend (with . intervention lasted 12-36 mths; generally considered to be large and moderate, respectively. parent-directed) methodological . included Control (no intervention, As the two sets of comparative data from Howard et al were . Used more inclusive improvements) the MA of considerably less intensive one, or compared to the same intervention group and not independent, definition of EIBI than Reichow and Wolery poorly described intervention) or an analysis excluding the control group from Howard et al Reichow and Wolery (2009). Comparison (non EIBI intervention (2005) was also conducted, with effect sizes not significantly (2009) of similar duration and intensity) altered. . Cited papers of those Search period: group. retrieved which were - March 2008 Homegeneity was tested and the Q statistic was not significant, excluded and noted why Exclusion criteria: indicating that studies could be combined, and a fixed effect . Detailed account of Search strategy: . Primarily using nonverbal intelligence model was used. No statistical or visual evidence of publication outcome measures ERIC measure; bias was observed. considered PsycINFO . Case study or case series; . Approached authors for PubMed . Data from people in another included Authors’ conclusions: individual data (search terms provided). study. . EIBI produces large to moderate effect sizes for changes . Fully detailed Tables, in IQ and ABC scores for children with ASD when describing design, References of retrieved Inter-rater reliability was high for selection and compared with no intervention controls and eclectic allocation to group papers were cross coding of articles. Approached authors for raw provision. The results support the clinical implication that method, interventions, checked to identify individual data which was re-analysed. at present, and in the absence of other interventions with outcome scores additional papers. Completed a meta-analysis yielding a established efficacy, Early Intensive Behavioral . Moderator variables standardized mean difference (SMD) effect Intervention should be an intervention of choice for were not considered. size for two available outcome measures: children with autism. . One author is from change in full-scale intelligence and/or . The difference in outcome between EIBI and the Lovaas institute. adaptive behavior composite. Effect sizes comparison intervention may be due to differences in the were computed using Hedges’s g. amount and frequency of supervision and training. There Overall quality: Very good was insufficient data to control for this in the present study, however the EIBI group in general received more frequent and more total hours of supervision and training. This remains a threat for the validity of conclusions about the superiority of EIBI in relation to comparison intervention. . Randomized controlled trials comparing EIBI to other interventions are still needed. In particular, studies are needed where the comparison intervention is of similar intensity and where staff receive similar training and supervision.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 23

Reference, Evidence level, aim and Inclusion and exclusion criteria Results and authors’ conclusions Comments and quality country, search method score design Ospina et al. Evidence level: I Inclusion criteria: 101 studies met selection criteria, and evaluated 8 broad types of . Clearly defined (2009) . RCTs, non randomized controlled intervention defined on a continuum of behavioural to methodology and Aim: trials, and prospective and developmental interventions, the most relevant reported here being extensive and Canada To identify, appraise and retrospective cohort studies with a the ABA category, which included discrete trial training (DTT) and comprehensive search synthesise the evidence control group; UCLA/Lovaas therapy as sub-categories. There were 31 ABA strategy SR/MA on the effectiveness of . published in English; studies, including 12 trials and 9 cohort/observational studies. . Review reports behavioural and . reporting on the efficacy of any extensive details of developmental behavioural or developmental Descriptive analyses individual studies interventions for intervention; . The reviewers judged studies to be at predominantly high risk including sample improving core symptoms . for individuals with ASD (including of bias and reported inconsistent results across various characteristics, as well of ASD. those with dual diagnoses); interventions. as assessments for . reporting on outcomes relating to . Discrete Trial Training effects were inconsistent, but motor and each study across Search period: the core features of ASD (though functional outcomes were often positive compared to speech- several quality domains. – May 2007 additional outcomes may also be related outcomes which were generally negative. For Lovaas, 7 . Careful synthesis of reported). of 8 non-RCTs reported positive findings, whereas 3 of 4 RCTs results. Search strategy: reported no significant findings. The reviewers note that as . Discussion of the 22 electronic databases, Exclusion criteria: non-RCT evidence can be more prone to bias and overestimate methodological including: . Individuals with Rett’s Disorder, or treatment effects, this observation has serious implications for limitations of review BioSYS Previews Childhood Disintegrative Disorder. interpreting the results from non-RCTs. approach and studies. CINAHL Heterogeneity was moderate for some outcomes (for overall In particular, notes risk Cochrane central register Two independent reviewers made the intellectual functioning, non verbal intellectual functioning, of “expectancy bias” for of controlled trials final study selection, extracted data socialization). studies with non active Embase using pre-tested checklists, and reached controls (eg, wait list) ERIC consensus on study quality. Meta- Meta Analyses where participants (and Medline analyses of the study results were . Meta-analyses of 3 controlled clinical trials showed that Lovaas their families) in control ProQuest Dissertations conducted when two or more trials treatment was superior to special education on measures of group would not expect and theses assessed the same intervention, used adaptive behaviour (WMD = 11.8; 95%CI, 6.94 to 16.67), improvement. PsychINFO similar comparison groups, and had data communication and interaction (WMD = 16.63; 95% CI, 11.25 . Moderate heterogeneity Web of Science for common outcomes of interest. If the to 22.01), comprehensive language (WMD = 12.84; 95% CI, evident for some (search terms provided). same measure was reported, reported 6.38 to 19.30), daily living skills (WMD = 5.61; 95% CI, 0.54 to outcomes. weighted mean differences (WMD); 10.67), expressive language (WMD = 15.05; 95% CI, 6.19 to . Suggestions made for Hand searching journals, otherwise, reported standardised mean 23.90), overall intellectual functioning (SMD = 0.95; 95% CI, future research cross-checking of differences (SMD). SMD of 0.2 indicated 0.44 to 1.46), and socialization (WMD = 9.17; 95% CI, 2.16 to reference lists, databases a small effect, 0,5 a medium, and 0.8 a 16.19). Overall quality: Very good of theses and large effect. . High-intensity Lovaas was superior to low-intensity Lovaas on dissertations, and measures of intellectual functioning in two retrospective cohort contacting experts in the studies (SMD = 0.92; 95% CI, 0.61 to 1.24). field were used to identify . No statistically significant differences were found for: Lovaas additional papers. versus special education for non-verbal intellectual functioning in a meta analysis of three controlled clinical trials (SMD = 7.83; 95% CI, 22.86 to 18.52); or for Lovaas versus Developmental Individual-difference Relationship-based intervention for communication skills based on two RCTs (SMD = 0.73; 95% CI, 20.26 to 1.72).

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 24

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design

Authors' conclusions:  There is no clear answer regarding the most effective therapy to improve symptoms associated with ASD.  The evidence seems to provide some support for discrete trial training in terms of motor and functional skills but not for communication skills.  Lovaas' therapy showed benefits when compared to ''no treatment'' and evidence from meta-analysis of retrospective cohort studies showed greater effects for High versus Low intensity Lovaas. Whilst Lovaas may improve some core symptoms of ASD compared to special education, these findings are based on pooling of a few, methodologically weak studies with few participants and relatively short-term follow-up. Replication in RCTs is needed to substantiate the use of Lovaas and to assess the effect of treatment intensity on outcomes of children with ASD.  There is considerable potential for heterogeneity in the population, intervention, comparator and outcomes of interest, as ASD is a spectrum disorder, therapy is not always reported in detail, comparators are difficult to control for, and outcomes are somewhat subjective.  As no definitive behavioural or developmental intervention improves all symptoms for all individuals with ASD, it is recommended that clinical management be guided by individual needs and availability of resources.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 25

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design Reichow and Evidence level: I Inclusion criteria: 14 studies met selection criteria, although due to patient overlap, only 13 . Limited detail provided Wolery . evaluated EIBI based on publications were appraised, reporting on 14 distinct samples. The studies on search strategy (2009) Aim: the UCLA/Lovaas (1987) included 10 studies allowing between group analyses, including two RCTs restricting ability for To provide a 3-part model; (Smith et al, 2000; Sallows and Grupner, 2005). replication. USA comprehensive synthesis . participants had . Detailed Tables, figures of the early intensive diagnoses of autistic Descriptive analyses . Precision details SR/MA behavioral intervention disorder, ASD, PDD, or . EIBI is an effective intervention for children with autism. On average, (confidence intervals) (EIBI) for young children PDD-NOS; children present fewer or less severe autism symptoms after intervention. given in Figure only with autism based on the . participant samples Effect size differences between treatment groups . Discussion of the Lovaas University of . The between group (standardized mean difference) effect sizes suggest methodological receiving EIBI had a California at Los Angeles children receiving EIBI made more gains than children receiving minimal limitations of included mean chronological age < Young Autism Project behavioral intervention, eclectic treatment, or treatment as usual. studies and of approach 84 months at the method (Lovaas, 1987). . Children receiving EIBI made large gains on multiple domains of of reviewers. beginning of treatment; The synthesis included: behavior, and made better progress than children with autism who . Studies used a range of . mean duration of EIBI (a) descriptive analyses, receive less intense behavioral intervention or other treatments. measures of IQ was 12 months; (b) effect size analyses, Meta analysis for change in IQ within treatment group . Included quasi . at least one child outcome and (c) a meta-analysis. . A meta-analysis was conducted on 12 samples using standardized mean experimental studies measure was reported; change effect size for IQ within the intervention group (because without an equivalent . experimental (e.g., Search period: comparison groups varied across studies). Using a random effects model, control group in the MA. pre-/post-test multiple- Not reported the mean effect size was 0.69. This is statistically significant (p < 0.001) . Heterogeneity present, group design) or quasi- and represents a large effect. This suggests EIBI is, on average, an which raises questions experimental designs (i.e., Search strategy: effective intervention for increasing IQ scores for children with autism, about whether nonequivalent control An “electronic database though the effect is likely to be inflated by publication bias. combining results group, one-group search”, databases not . Tests of homogeneity suggest that there was significant variability through a meta analysis pre-/post-test design); identified. (Search terms between studies. was appropriate. . publication in English in a not provided). . Moderating variables identified a priori suggest the greatest results on IQ . Suggestions made for peer-reviewed journal. change might be seen when supervisory staff were trained using the future research sample adjustment. References from review UCLA model (p<0.01). directions articles and eligible reports were cross Overall quality: Good checked to identify additional papers.

Hand search of selected journals.

Contact with experts.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 26

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design Study characteristics and Authors’ conclusions: effect size data were coded . While these findings were strong, the nonrandom assignment of using a manual. Two participants to groups limit conclusions about the superiority of EIBI to independent coders for 4 of other treatments. Further, the comparison groups often lacked the studies had mean standardization within the group, were poorly defined, had no measures interobserver agreement of of procedural fidelity, and had no data on whether participants received 92%, range 85-93%. Studies supplemental treatments. rated on experimental rigour. . The findings suggest EIBI can be an effective treatment, on average, for Meta analysis conducted for some children with autism. However the intervention has not worked for studies reporting within-group all children, and individual data typically were not presented. It is unclear comparisons using if individuals making change in one domain (e.g., IQ) also made gains in standardized mean change another (e.g., adaptive behavior). effect size with small sample . These results should be taken with caution because of the small number adjustment. of studies, and because the standardized mean change effect size is calculated without reference to a comparison or control group. Thus, the threats to internal validity of history, maturation, lack of procedural fidelity, and instrumentation threats cannot be eliminated. Thus, while the effect sizes were often large, they cannot be attributed to EIBI exclusively. Effect sizes tend to be inflated when using standardised mean change effect sizes (Morris, 2000). Without comparisons between EIBI and empirically validated treatment programs, it is not possible to determine if EIBI is more or less effective than other treatment options.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 27

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design Rogers and Evidence level: I Inclusion criteria: 22 studies met selection criteria. Studies of non ABA interventions, including . Limited search Vismara . study involved psychopharmacological interventions, are not discussed here. strategy, 1 database (2008) Aim: comprehensive . 5 RCTs, included 4 relating to ABA were identified: Jocelyn et al (1998), and cross-checking What is the empirical (addressing core Drew et al (2002), Smith et al (2000), and Sallows and Graupner (2005). . Detailed Tables, US evidence supporting deficits in autism) . Non-randomized controlled group design studies included 2 of Lovaas- describing studies, efficacy of early treatment based interventions: Eikeseth et al (2002), Cohen et al (2006); and one of interventions, and SR intervention or young approaches for an EIBI approach using a mixture of didactic and naturalistic behavioral discussion of results children with autism. children with teaching approaches by Howard et al (2005). . Discussion of autism; methodological Search period: . predominantly ages . Results suggest that young children with autism, as a group, demonstrate limitations of included 1998 - 2006 5 years or younger; accelerated developmental gains in response to focused daily studies and findings. . controlled group interventions of several different kinds. Significant increases in language . Detailed suggestions Search strategy: designs or single- and communication abilities in the treated group occurred in most studies made for future PsycINFO subject multiple and interventions with many targeted hours per week resulted in research directions, (search terms provided). baseline designs increases in IQ at the group level as well. especially around using 3 or more . The best-designed, controlled studies evaluating Lovaas’s treatment met mediators and References of all subjects with criteria for a ‘‘well-established’’ psychosocial intervention for improving moderators of effect reviewed articles were published data. the intellectual performance of young children with autism spectrum . Practice cross checked to identify disorders, based on the significant increase in IQ reported in these four recommendations are additional papers. Exclusion criteria: studies compared to control groups. Although some of these studies also offered for . studies that did not report reported significant improvements in behavior, adaptive skills, and psychologists. as outcomes analyses of language skills, reviewers argue that there is less consistency in the data . Discussed the value of child progress using in these areas. both RCTs and SCED general measures of . No treatment met the ‘‘probably efficacious’’ criteria, although 3 studies in the pathway children’s language or treatments met criteria for being ‘‘possibly efficacious’’. of intervention intellectual development; . Most studies were either Type 2 or 3 in terms of their methodological rigor development. . studies targeting only one based on Nathan and Gorman’s (2002) criteria. domain (eg; unwanted Overall quality: Good behaviours); Authors’ conclusions: . case reports; . The evidence suggests that early intervention programs are beneficial for . studies whose data were children with autism, often improving developmental functioning and published only in book decreasing maladaptive behaviors and symptom severity at the level of chapters; group analysis. . studies included in review . Lack of comparative studies prevents determining which comprehensive by Rogers (1998). treatment approach is best for young children with autism. Other well- known interventions may be as or more efficacious as Lovaas’s model but Studies were graded they have not been rigorously evaluated. according to methodological . Because the majority of studies did not include ethnically diverse criteria based on Chambless participant groups, generalization of effects across groups is premature. et al (1998) as being (i) “well . Given the few randomized controlled treatment trials that have been established” or (ii) probably carried out, the few models that have been tested, and the large efficacious. Studies also differences in interventions that are being published, it is clear that the graded using Nathan and field is still very early in the process of determining (a) what kinds of Gorman (2002) criteria as interventions are most efficacious in early autism, (b) what variables Type 1 through to Type 6, moderate and mediate treatment gains and improved outcomes following ranging from most to least intervention, and (c) the degree of both short-term and long-term rigorous. improvements that can reasonably be expected for a child with autism.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 28

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design Seida et al Evidence level: I Inclusion criteria: 30 studies met selection criteria representing five intervention domains, . Broad search strategy (2008) . Systematic reviews including one of interventions based on behavioural theory (n=9) reported on . Use of two reviewers Aim: (defined as having here. and quality checklist Canada To present an “umbrella defined a search . Only secondary studies review” of the clinical strategy, and All 9 SRs on interventions based on behavioural theory were rated as having considered SR (of SRs) findings of systematic attempted to low methodological quality. . Only sparsely detailed reviews of psychosocial synthesise data of Tables, listing interventions for ASDs. primary studies . The reviews with meta-analyses showed favourable outcomes for the intervention domains quantitatively or behavioural intervention. Reductions in problem behaviour were found in and the outcome areas Search period: qualitatively); 3 reviews and an increase in adaptive, cognitive, and language skills was where positive, – May 2007 . Data reported for observed in 1 review. negative or unclear participants with . In the reviews without meta-analysis, positive findings were reported for results were found. No Search strategy: autistic disorder, intelligence, developmental gains, functional skills, and communication data provided to allow 25 electronic databases Asperger outcomes. The authors of 3 reviews concluded that there is uncertainty estimation of effect or including Medline syndrome, atypical about whether behavioural interventions produce ‘normal functioning’ and precision of findings. (listed with search terms autism, high improvement on various intelligence and developmental measures. The . Suggestions made for in an online supplement). functioning autism, remaining 2 reviews provided no information on efficacy outcomes; future research, PDD-NOS, and/or instead trends in the use of behavioural treatments over time were including on reporting References of articles and suspected autism; described. and conduct of SRs of “personal collections” . Have covered a were cross checked to psychosocial Authors’ conclusions: Overall quality: Fair identify additional papers. intervention aimed . The reviews reported positive outcomes for many of the interventions, at improving the suggesting that some form of treatment is favourable over no treatment. functioning of . Little evidence for the relative effectiveness of these treatment options. individuals in any of . Many of the systematic reviews had methodological weaknesses that the impairments make them vulnerable to bias. characteristic of . Even if differences in the therapeutic effectiveness of the interventions ASD. exist, differences in cost, convenience and family burdens associated with the interventions are likely to be important factors in individual decision- Exclusion criteria: making. . editorials, . Future studies and reviews that break down the characteristics of the correspondence, individuals with autism and the components of the programmes are abstracts and review needed in order to provide more meaningful and stronger conclusions. summaries

Abstracts scanned by single reviewer but eligibility criteria applied on full text independently by two reviewers. Studies were graded for methodological quality by two reviewers using the Overview Quality Assessment Questionnaire. Data extracted by single reviewer, and verified by another.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 29

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design Spreckley & Evidence level: I Inclusion criteria: 13 studies met selection criteria, 6 were trials with adequate methodologic . Broad database search Boyd (2009) . Systematic reviews, quality (PEDro6), and 4 of these had adequate data to be included in the strategy, but no other Aim: randomized or quasi- meta analysis (Smith et al, 2000; Sallows and Graupner, 2005; Eikeseth et al search technique Australia To review the randomized controlled 2002; and Eikeseth et al, 2007). . Applied minimum effectiveness of applied trials (RCT); quality criteria for SR/MA behavior intervention . Preschool children with Meta-analysis of 4 studies concluded that, compared with standard care, ABI inclusion in the MA (ABI) programs for diagnosis of ASD or programs did not significantly improve outcomes of children in the . Tables described preschool children with PDD; experimental group compared with those who received standard care: design, interventions, autism spectrum disorder . Interventions focused on . For cognitive outcomes, a standardized mean difference (SMD) of 0.38 comparators, and (ASD) in their cognitive, ABI approaches to (95%CI �-0.09 to 0.84; P=.1) intensity. adaptive behavior, and behavioural . For expressive language; SMD of 0.37 (95%CI - 0.09 to 0.84; P=.11) . Estimates of precision language development. management; . For receptive language; SMD of 0.29 (95%CI �- 0.17 to 0.74; P=.22) provided. . Interventions delivered to . For adaptive behavior; SMD of 0.30 (95%CI �-0.16 to 0.77; P=.20). . Too few studies Search period: parents/carers and/or included to consider From 1982 but varied directly to the child by Heterogeneity (measured by I2) was low for cognitive and receptive language sub-group or sensitivity (see below) - Nov 2007 special educators, outcomes, and moderate for expressive language and adaptive behaviour analyses or to consider teachers, speech outcomes. moderator variables. Search strategy: pathologists, . Follow-up Cochrane Library psychologists, or allied Authors’ conclusions: correspondence about Medline (from 1996) health professional . Currently there is inadequate evidence that ABI has better outcomes than this article from Smith PsycINFO (from 1985) students; standard care for children with autism. Appropriately powered clinical et al (2009) suggested CINAHL (from 1982) . Interventions occurred trials with broader outcomes are required. that the Sallows and AMED (from 1985) between 18 mths and 6 . Limitations of the meta-analysis and evidence base include: high Graupner (2005) study (search terms provided). years; variability in the studies included (in ages of children involved, programme be excluded from the . Cognitive, language, content), difficulty in establishing control groups all received some form of MA as its comparison and/or adaptive intervention), poor homogeneity, limited information on retention in the group was a lower behaviour outcomes interventons groups, and lack of strict selection criteria for participants. intensity form of ABA. measured. . What is too often forgotten is that the overwhelming majority of children Spreckley & Boyd’s with ASD change over time as part of their development. (2009b) reply argued Exclusion criteria: that they were . Studies with PEDro score evaluating standard <6; (i.e., high intensity) ABA . RCTs with no useful data with any standard or for meta-analysis; and/or eclectic comparator. . those that did not include . Limitations and discrete trial training heterogeneity of the (DTT) as part of their evidence base raises intervention. questions about whether a meta 2 reviewers independently analysis was applied the Physiotherapy appropriate. Evidence Database (PEDro) Scale of quality assessment to Overall quality: Good critically appraise the studies. Data synthesis used RevMan, reported effect sizes and standardised mean differences (SMD).

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 30

2. PECS

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design Ostryn et al. Evidence level: I Inclusion criteria: 15 studies met selection criteria. Most studies were single subject within . Broad search strategy (2009) . English language; subjects or multiple baseline designs, with one RCT (Yoder and Stone, 2006) . Detailed Tables Aim: . Used experimental and one non-randomised, between-group experimental study (Carr and Felce, describing outcome USA To examine research or quasi- 2007). measures and results, involving the use of PECS experimental but lacking data on SR with individuals with ASD design; . Noted that whilst “positive outcomes” were reported in all but four participant to identify dependent . Included individuals studies, though these were not defined in terms of statistical significance. characteristics and variables and outcomes aged under 18 . Noted that acquisition rates (through phases of PECS 1-6) were reported interventions. reported in the research, years old with as being “fast” by several studies, though “fast” was not defined. . Narrative discussion of and to apply the ASDs; . Commented that outcomes such as spontaneous communication were results framed around communication . evaluated PECS; only defined in two studies in relation to visual prompts. how outcomes competence paradigm to . published in a peer- . Outcomes associated with joint attention were measured in only three measures in PECS the PECS research. reviewed Journal. studies. studies relate to the . Notes that few studies examined maintenance of PECS outcomes. communication Search period: Exclusion criteria: Where they are reported, results are encouraging. competence paradigm. 1985 –2007 Not reported . No formal checklists The authors applied the communication competence paradigm to PECS used to assess or grade Search strategy: Narrative synthesis outcomes and argue that additional strategies to those of manding or tacting quality. ERIC could enhance PECS. The authors suggest that PECS strategies may be too . Minimal synthesis of PsychINFO limited for some individuals and people with ASD should be encouraged to findings, although note PubMed use any communication attempts, including speech, gestures, and vocal that this was not (search terms provided). approximations. specifically the aim of the review. Also searched PECS Authors’ conclusions: . Suggestions made for website. References of . Results of this study reveal that the PECS is widely implemented with future research. retrieved papers were individuals having ASDs but without a strong empirical base. . Suggestions for how to cross checked. . Suggest that PECS is best used as an initial intervention to teach use PECS as part of a manding and the basic elements of communicative exchange, but that multimodal approach other communication systems may be needed to permit communication including other beyond this. communication systems . PECS is not recommended as a long term intervention and is best implemented as part of a multimodal system for when picture Overall quality: Fair communications are more socially appropriate.

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 31

Reference, Evidence level, aim and Inclusion and Results and authors’ conclusions Comments and quality country, search method exclusion criteria score design Sulzer- Evidence level: I Inclusion criteria: 34 studies met selection criteria, including two RCTs (Howlin et al. 2007) . Limited search Azaroff et al. . Articles including (Yoder and Stone 2006b) (Yoder and Stone 2006a) and one non randomised strategy, particularly (2009) Aim: data and protocol experimental study (Carr and Felce 2007). Studies reported on 386 study with respect to search To synthesise key containing the key participants, the “majority” of whom had received diagnoses of ASD. terms and reporting. UK features of the published, features of PECS . Narrow definition of peer-reviewed, data- (as specified by The authors presented a number of summarising statements in the Results PECS. SR based research reports of Frost and Bondy). section asserting positive research findings, however these were not . Lacks detail of outcome Picture Exchange attended by quantitative supporting data, analysis or synthesis of study measures and follow-up Communication System Exclusion criteria: findings. For example, “researchers reported improvement in communication periods of inidivudal (PECS) applications, to . Papers written in among the vast majority of their participants”. Also, “several investigators studies. answer the question “how languages the authors provided evidence that learning to use PECS was associated with some of . Results and Discussion solid is the scientific couldn’t read (n=3) their participants increasing their speaking and social approaching” and “a focussed on narrative, evidence supporting . Studies describing number of investigators cited decreases in disruptive or dangerous imprecise description of PECS effectiveness?” different pictorial behaviours”. studies. interventions to the . Detailed Tables Search period: original PECS. Thus whilst these and similar statements stated that there were examples of . Emphasised the need Not reported positive findings, the authors did not demonstrate the extent to which such for additional reporting The review summarised the findings occurred or the precision of this effect, how effects varied between in future research Search strategy: studies in a narrative way, with studies and individuals, or why. . Note that two of the ERIC little quantification. authors were the PsycINFO The authors urged PECS researchers to report more methodological details, original developers of Science Direct and identified questions to be explored in future research. PECS. Google scholar. Minimal search terms used Authors’ conclusions: Overall quality: Fair including PECS, Bondy, . Analysis of the available research leads to the conclusion that the Frost (who developed majority of participants who lacked functional comunication skills did PECS). acquire extensive functional vocabularies. . Analysis of the studies strongly supports the conclusion that by adhering References of retrieved to the PECS protocol, professionals and parents can teach individuals to papers were cross successfully initiate exchanges of pictures for tangible and non tangible checked to identify reinforcers. additional papers. Key: ABI Applied behaviour intervention ABA Applied behaviour analysis CI Confidence interval DTT Discrete trial training EIBI Early intensive behavioural intervention Hrs/wk Hours per week IQ Intelligence quotient MA Meta analysis PEDro Physiotherapy evidence database PDD NOS Pervasive developmental disorder – not otherwise specified PECS Picture exchange communication system RCT Randomised controlled trial SCED Single case experimental design SD Standard deviation SMD Standardised mean difference SR Systematic review TEACCH Treatment and education of autistic and related communication-handicapped children UCLA University of California, Los Angeles WMD Weighted mean difference

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER 32

UPDATE OF REVIEWS: APPLIED BEHAVIOUR ANALYSIS FOR PEOPLE WITH AUTISM SPECTRUM DISORDER