AN INTERACTIVE SYSTEM to ENHANCE SOCIAL and VERBAL COMMUNICATION SKILLS of CHILDREN with AUTISM SPECTRUM DISORDERS by Maria Jose Minan

AN INTERACTIVE SYSTEM TO ENHANCE SOCIAL AND VERBAL COMMUNICATION SKILLS OF CHILDREN WITH AUTISM SPECTRUM DISORDERS by Maria Jose Minan

A Thesis Submitted to the Faculty of The College of Engineering and Computer Science in Partial Fulﬁllment of the Requirements for the Degree of Master of Science

ACKNOWLEDGEMENTS

I would like to express my sincere gratitude to my thesis advisor, Dr. Shihong Huang for her guidance, constructive criticism, friendly advice, and support during my graduate and undergraduate studies. My deepest gratitude to Dr. Mirjana Pavlovic and Dr. Emmanuelle Tognolli for their insight regarding Autism Spectrum Disorder. Special thanks to Dr. Daniel Raviv and Dr. Maria Petrie for taking the time to review my thesis and provide constructive feedback. My completion of this project could not have been accomplished without the support of my managers and co-workers at Belcan Engineering. I would also like to thank my family for being encouraging and supportive during my academic career. Special thanks to my mother and grandmother for teaching me the importance of education and for their unconditional support. Additional thanks to Becca Huy and Michelle Huang for their professional opinions. Thank you to the Software Engineering Lab at FAU, including Ante Bosnjak, Jose Hurtado, Sheryl Laﬂeur, Pedro Miranda, and James Mulcahy for their support. Finally, thank you to all my friends for always encouraging me to achieve my goals.

iv ABSTRACT

Author: Maria Jose Minan Title: An Interactive System To Enhance Social And Verbal Communication Skills Of Children With Autism Spectrum Disorders Institution: Florida Atlantic University Thesis Advisor: Dr. Shihong Huang Degree: Master of Science Year: 2014

Affecting one in every 68 children, Autism Spectrum Disorder (ASD) is one of the fastest growing developmental disabilities. Scientific research has proven that early behavioral intervention can improve learning, communication, and social skills. Similarly, studies have shown that the usage of off-the-shelf technology boosts motivation in children diagnosed with ASD while increasing their attention span and ability to interact socially. Embracing perspectives from different fields of study can lead to the development of an effective tool to complement traditional treatment of those with ASD. This thesis documents the re-engineering, extension, and evolu- tion of Ying, an existing web application designed to aid in the learning of autistic children. The original methodology of Ying combines expertise from other research areas including developmental psychology, semantic learning, and computer science. In this work, Ying is modified to incorporate aspects of traditional treatment, such as Applied Behavior Analysis. Using cutting-edge software technology in areas like voice recognition and mobile device applications, this project aspires to use software engineering approaches and audio-visual interaction with the learner to enhance so-

v cial behavior and reinforce verbal communication skills in children with ASD, while detecting and storing learning patterns for later study.

vi AN INTERACTIVE SYSTEM TO ENHANCE SOCIAL AND VERBAL COMMUNICATION SKILLS OF CHILDREN WITH AUTISM SPECTRUM DISORDERS

List of Tables ...... xi

List of Figures ...... xii

1 Introduction ...... 1 1.1 Overview of Autism Spectrum Disorder...... 1 1.2 Problem Statement...... 3 1.3 Technology and ASD...... 3 1.4 Approach and Contribution...... 4 1.5 Thesis Organization...... 8

2 Background ...... 9 2.1 Autism Spectrum Disorder...... 9 2.2 Common Symptoms...... 10 2.3 Types of Autism Spectrum Disorder...... 11 2.3.1 Asperger’s Disorder...... 12 2.3.2 Autistic Disorder...... 13 2.3.3 Childhood Disintegrative Disorder...... 13 2.3.4 Rett’s Disorder...... 13 2.3.5 Pervasive Developmental Disorder-Not Otherwise Speciﬁed. 14 2.4 Diagnosis...... 15 2.5 Traditional Treatments...... 16

vii 2.5.1 Applied Behavior Analysis...... 16 2.5.2 Floortime...... 17 2.5.3 Pivotal Response Therapy...... 17 2.5.4 Verbal Behavior Therapy...... 18

3 Research Focus ...... 19 3.1 Verbal Communication...... 19 3.2 Social Communication...... 20 3.3 Adaptation...... 20

4 Literature Review ...... 22 4.1 Technology to Enhance Verbal Communication Skills in ASD Children 23 4.1.1 Interactive Applications...... 23 4.1.2 Static Applications...... 25 4.2 Technology to Enhance Social Communication Skills in ASD Children 28 4.2.1 Interactive Applications...... 28 4.2.2 Static Applications...... 31 4.3 Technology to Enhance Adaptation Skills of ASD Children...... 33 4.3.1 Interactive Applications...... 33 4.3.2 Static Applications...... 37

5 Methodology ...... 39 5.1 Objective...... 39 5.2 Approach...... 40 5.2.1 Learn...... 42 5.2.2 Play...... 44 5.2.3 Schedule...... 48 5.2.4 Rewards...... 48 5.2.5 Progress...... 49

viii 5.2.6 Conﬁguration...... 50

6 Implementation ...... 51 6.1 System Architecture of Ying...... 51 6.1.1 Presentation Layer...... 52 6.1.2 Business Logic Layer...... 52 6.1.3 Data Layer...... 54 6.2 Main Components...... 55 6.2.1 Speech Recognition...... 55 6.2.2 Text to Speech...... 56 6.2.3 Graphing...... 57 6.2.4 The Avatar...... 58 6.3 Modules...... 59 6.3.1 Learn...... 60 6.3.2 Play...... 61 6.3.3 Schedule...... 61 6.3.4 Rewards...... 62 6.3.5 Settings...... 62 6.3.6 Progress...... 63 6.4 Platform...... 64 6.5 Activities...... 64 6.5.1 Virtual Flashcards...... 64 6.5.2 Eye Gaze...... 65 6.5.3 Story Teller...... 66 6.5.4 Say my Name...... 67 6.5.5 Take a Peek...... 68 6.5.6 Message in a Bottle...... 69 6.5.7 Tappy Time...... 69

ix 6.5.8 Fill in the Blank...... 70

7 Conclusion ...... 71 7.1 Research Contribution and Summary...... 71 7.2 Evaluation...... 72 7.3 Future Areas of Development...... 72

Appendices ...... 74 A Evaluation Questionnaire...... 75 A.1 Learning Activity 1: Virtual Flashcards...... 75

Bibliography ...... 91

x LIST OF TABLES

1.1 Prevalence of ASD [1]...... 2

4.1 Existing apps for children with ASD...... 23

xi LIST OF FIGURES

1.1 Welcome Page of Ying [29]...... 5 1.2 High Level Overview of the Original Ying [29]...... 6 1.3 ”Gazy Gazy” - Eye Contact Game [29]...... 7

2.1 Synapses are points of communication between two nerve cells [46]. 10 2.2 Brain Cell of an autistic patient (left) and a non-autistic person(right) [46]...... 10 2.3 Types of Autism Spectrum Disorders...... 12 2.4 Types of ASD and Functioning Levels [18]...... 15 2.5 Therapy Session [62]...... 16

4.1 Classiﬁcation of Software for Children with ASD...... 22 4.2 Vocabulary - Proloquo2go [63]...... 23 4.3 Proloquo2go Features [64, 63]...... 24 4.4 Results of the Evaluation of Proloquo2go [63]...... 25 4.5 Vast Autism 1 Core Main Menu (Left), Modeling Video (Right) [52] 26 4.6 Video Categories [52]...... 27 4.7 Training Faces Interface [53]...... 28 4.8 Training Faces Emotion Recognition Process...... 29 4.9 Tom Cat [37]...... 30 4.10 The Transporters...... 32 4.11 First Then Visual HD Icon...... 33 4.12 First Then Visual HD Interface...... 34 4.13 Schedule and instructions to complete a task [67]...... 34

xii 4.14 AutisMate's User Interface [68]...... 36 4.15 Zac Browser...... 37 4.16 YouTube video accessed through Zac Browser [9]...... 37

5.1 Logo of the Ying system...... 40 5.2 Overview of Ying...... 41 5.3 Virtual Flashcards for Words...... 42 5.4 Virtual Flashcards for Phrases...... 43 5.5 Story Teller...... 44 5.6 Eye Gaze...... 45 5.7 Say My Name...... 45 5.8 Message in a Bottle...... 46 5.9 Take a Peek...... 47 5.10 Tappy Time...... 47 5.11 Fill in the Blank...... 48 5.12 My Schedule Module...... 49 5.13 Rewards Module...... 49 5.14 Progress Module...... 50 5.15 Conﬁguration Module...... 50

6.1 Architecture of Ying...... 51 6.2 Data Contracts in Ying's Service...... 53 6.3 Ying's Database Diagram...... 55 6.4 Speech Recognition Component...... 56 6.5 Text-to-Speech Engine...... 56 6.6 Text-to-Speech Process...... 57 6.7 Graphing Tool...... 58 6.8 Avatar...... 58 6.9 Modules...... 59

xiii 6.10 Learn Module...... 60 6.11 Play Module...... 61 6.12 Schedule Module...... 62 6.13 Rewards Module...... 62 6.14 Conﬁguration Module...... 63 6.15 Progress Module...... 63 6.16 Virtual Flashcards...... 65 6.17 Eye Gaze...... 66 6.18 Story Teller...... 67 6.19 Say My Name...... 67 6.20 Take a Peek...... 68 6.21 Message in a Bottle...... 69 6.22 Tappy Time...... 70 6.23 Fill in the Blank...... 70

xiv CHAPTER 1 INTRODUCTION

“The journey of a thousand miles begins with a single step.” – Lao-Tzu

1.1 OVERVIEW OF AUTISM SPECTRUM DISORDER

Autism Spectrum Disorder(ASD) is a general term for a group of complex disorders of brain development that affect over three million individuals in the United States and tens of millions worldwide. As it can be observed in Table 1.1, this disorder has become more prevalent in the last couple of years, and currently one in 68 children are diagnosed with ASD. ASD is characterized by impaired development in social interaction, communication, and behavior [4]. Some of the social difficulties faced by autistic children are interpreting social cues, maintaining eye contact, and understanding facial expressions. In the same way, communication can be a challenge since some of their tendencies include repeating what others say without understanding the meaning, mixing up the pronouns, and not being able to start a conversation. In terms of behavior, ASD children may rock, spin, and flap hands repetitively [47]. While some children experience delayed development from birth, other children seem to develop normally before they suddenly lose social or language skills. In some cases, loss of language is the major impairment. In others, unusual behaviors seem to be the dominant factors

Currently, ASD is divided into 5 diﬀerent categories including Autism, Asperger's Disorder, Pervasive Developmental Disorder-Not Otherwise Speciﬁed (PDD-NOS),

1 Identiﬁed Prevalence of Autism Spectrum Disorder ADDM Network 2000-2010 Combining Data from All Sites

Surveillance Birth Year Number of Prevalence per This is about 1 Year ADDM Sites 1,000 Children in X children Reporting

2000 1992 6 6.7 1 in 150

2002 1994 14 6.6 1 in 150

2004 1996 8 8.0 1 in 125

2006 1998 11 9.0 1 in 110

2008 2000 14 11.3 1 in 88

2010 2002 11 14.7 1 in 68

Table 1.1: Prevalence of ASD [1]

Childhood Disintegrative Disorder, and Rett's Disorder; the first 3 are the most prevalent. The degree of ASD can vary from mild to severe. Asperger's syndrome is considered the mildest form of ASD since it is mainly characterized by poor social skills. The ways in which ASD is exhibited can differ greatly. Levels of impairment and combination of symptoms vary from patient to patient, which makes every case unique and obstructs the process of finding a treatment that fits all [45].

The causes of ASD are unknown, but it is believed that it is triggered by a combination of factors, and that it is preventable and treatable [48]. Scientists agree that the earlier in life a child receives early intervention services the better the prognosis of the child. All children with autism can beneﬁt from early intervention. There is no medical test that can diagnose autism. Instead, specially trained physicians and psychologists administer autism-speciﬁc behavioral evaluations. A typical diagnos- 2 tic evaluation involves a multi-disciplinary team of doctors including a pediatrician, psychologist, speech and language pathologist and occupational therapist. Genetic testing may be recommended. This comprehensive evaluation helps parents understand as much as possible about their child's strengths and needs [49].

1.2 PROBLEM STATEMENT

Since no two children diagnosed with Autism Spectrum Disorder share the same symptoms, therapy must be tailored to address the specific needs of each patient [55]. One of the methods that has shown to be extremely effective in the education of ASD patients is Applied Behavior Analysis (ABA). Unlike traditional teaching approaches, ABA involves intensive, one-to-one instruction and encompasses a variety of teaching strategies that are more highly-structured and more naturalistic. Due to the significant amount of teacher time and materials needed to implement it effectively, ABA is often prohibitively expensive [12]. The aim of this study is therefore to alleviate the cost of behavioral therapy by designing a customizable, interactive system that provides ASD patients with educational activities that reinforce their social and verbal communication skills at any given moment. This interactive system, named Ying [29], is not meant to replace existing therapies, but complement them instead. Ying incorporates aspects of ABA therapy including rewarding and positive reinforcement while recording learning patterns for later study.

1.3 TECHNOLOGY AND ASD

Children with ASD, due to social, attentional, and motivational deficits, exhibit significant difficulties learning through traditional teaching methods. The cost of intensive behavioral interventions for autistic patients is very high because of the significant amount of teacher time and materials needed to implement them correctly . The use of computer-assisted instruction (CAI) in the education of children with disabilities 3 can make intensive behavioral interventions more accessible by reducing the cost of materials and training. A computer-based approach can help to reduce the number of staff and staff training, which would save families and school districts significant amounts of money. Being capable to automatically collect data on the performance of the patient, computers can provide more accuracy and more comprehensive data than personal instruction. Similarly, computers can store great amounts of information, which means that more concepts can be presented [12]. In the case of children with ASD, who appreciate systematic approaches, computer- based solutions can assist therapy since these are consistent and predictable. In con- trast to humans, computers can also help to build upon past experiences and results over time by incorporating aspects of behavioral therapy, such as reinforcement and repetition without neither frustrating nor intimidating the patient. In the same way, the presence of trained personnel is not necessary while the patient interacts with the system. This factor increases the number of hours of intervention [12]. Finally, since most children with ASD are visual learners, they are able to attend longer, and many skills can be taught with reduced behavior problems and increased learning time [12, 57, 66]. It is important to point out that human interaction and teachers still play an important role and can further improve the results. Scientific research has shown that ASD patients who preferred computers, perform better on the task at hand if a teacher assisted them during their interaction [29].

1.4 APPROACH AND CONTRIBUTION

This approach consists of re-engineering and extending the functionality of an existing web application developed at the Software Engineering Lab at Florida Atlantic University in 2011 [29]. The name of this application is Ying, which is derived from the Chinese word for owls or other birds of prey. Originally, Ying was designed to help identify the factors that make learning more eﬀective for children with ASD. As

4 observed in Figure 1.1, it consisted of four diﬀerent modules, which were assessments, play, progress, and conﬁguration [29].

Figure 1.1: Welcome Page of Ying [29]

The Configuration and Progress Module were implemented to meet the needs of the coach or researcher. The first step consisted of the supervising adult, referred to as a Coach in Figure 1.2, providing an initial configuration for the system. This configuration step involved setting up the various accounts for the children with ASD, creating, editing, and determining the questions and responses, as well as setting up the assessments [29]. The main component of Ying was an avatar. In the Play and Assessment Module, the avatar switched his gaze to one of the surrounding object in his virtual environment, as shown in Figure 1.3. The task of the child was to follow the eyes of the avatar and identify the direction of its gaze. In the Assessment Module the performance of the child was explicitly evaluated. The results of the evaluations were available in the Progress Module [29]. In this thesis, Ying is re-engineered as a customizable, Windows-based mobile 5 Figure 1.2: High Level Overview of the Original Ying [29] application that complements therapy and reinforces verbal and social communication skills in ASD children. Instead of designing a system that fits all, the diversity of the spectrum is taken into consideration. This adaptive approach consists of providing care takers and therapists with tools to address the communication challenges ASD patients face and giving them the flexibility to let the app guide the therapy session. Data will be collected from the interaction between a child and a series of educational, interactive, and fun games. Once completed, the results of these activities will be available for evaluation and therapy improvement. This research incorporates aspects of effective traditional therapy, such as rewarding and positive reinforcement, as well as cutting edge technology, such as voice recognition. Some of the main contributions of this work include:

6 Figure 1.3: ”Gazy Gazy” - Eye Contact Game [29]

• Analysis and review of existing technology targeted to improve communication and adaptation skills in ASD patients.

• Re-engineering of Ying

– Migration to the Windows Mobile Platform

– Architectural re-design

– Database re-design

– Re-design of the avatar component

• Extension of Ying 7 – Addition of two modules, which are Rewards and My Schedule

– Addition of new activities

– Integration of new components including voice recognition and visual-audio cues

– Addition of scoring system

1.5 THESIS ORGANIZATION

The remainder of the thesis is organized as following:

• Chapter 2 presents a more detailed overview of ASD, along with the symptoms and deﬁciencies associated with the various disorders. In addition, diagnosis and traditional treatment are discussed.

• Chapter 3 describes the three areas of focus of this research.

• Chapter 4 introduces some of the latest technology used to improve the life of autistic children that are relevant to the focus of this research.

• Chapter 5 describes the research methodology including objectives, hypothesis, approach, and design of the proposed approach.

• Chapter 6 presents the implementation of a mobile application, named Ying, and all that entails, i.e., the architecture, features, components, and design.

• Chapter 7 summarizes the scope of this research and contributions, discusses some of the areas for future development, and the evaluation of Ying.

8 CHAPTER 2 BACKGROUND

“The more that you read, the more things that you will know. The more that you learn, the more places you‘ll go.” – Theodor Seuss Geisel

2.1 AUTISM SPECTRUM DISORDER

Autism Spectrum Disorder (ASD) refers to a group of closely related developmental brain disorders with a shared of core symptoms. The term spectrum alludes to the wide range of symptoms, skills, and levels of impairment that a person with ASD can have [5] . ASD affects a patient's ability to communicate, respond to surroundings, or form relationship with others [13]. Possible symptoms include sensory problems, emotional difficulties, and uneven cognitive abilities [5].The level of impairment and combination of symptoms varies from person to person. While some children are mildly incapacitated by their symptoms, others are severely affected. Consequently, two children with the same diagnosis may appear very different in terms of abilities and behaviors [13]. Although the causes of ASD have not been identified, scientific research has found many neurological abnormalities that might contribute to the social and cognitive deficits observed in ASD [7]. As shown in Figure 2.1, children affected by autism have a surplus of synapses, connections between brain cells. This excess is attributed to a slowdown in the normal pruning process that takes place during brain development [16, 46].

9 Figure 2.1: Synapses are points of communication between two nerve cells [46]

Figure 2.2: Brain Cell of an autistic patient (left) and a non-autistic person(right) [46]

The overgrowth of the number of brain cell is caused by genes that fail to govern the number of cells that are generated and to sort through these cells to eliminate defective connections. This takes place in prenatal life. As a result, the child produces an abnormal amount of brain cells, which establish connections among them. The connections created are not always good. It takes decades for the brain to eliminate brain cells. Therefore, early treatment makes a diﬀerence in creating healthy connections and preventing bad connections from being made [16, 46].

2.2 COMMON SYMPTOMS

The following symptoms are very common in autistic children [17]:

• No social interaction with others, including parents

– Child displays a lack of interest in physical contact. Avoids making eye

10 contact with others, including parents

– Child fails to develop friends

• Does not communicate well with others

– Language development is delayed or non-existing at all

– Child repeats words or phrases, also known as echolalia

• Repetitive behaviors

– Child demonstrates repetitive motor movements, such as rocking and hand or ﬁnger ﬂapping

• Is preoccupied, usually with lights, moving objects, or parts of objects

• Does not like noise

• Has rituals [17]

– For example, a child may have to touch his favorite toy before going to bed [51]

• Requires routines[17]

2.3 TYPES OF AUTISM SPECTRUM DISORDER

Currently, 5 different types of ASD have been identified. These include Autistic Disor- der, Asperger's Disorder, Pervasive Developmental Disorder-Not Otherwise Specified, Rett's Disorder, and Childhood Disintegrative Disorder [4]. The first three are the most prevalent [5].

11 Figure 2.3: Types of Autism Spectrum Disorders

2.3.1 Asperger’s Disorder

Characterized by poor coordination and social skills, Asperger's Disorder is the mildest form of autism. It is also known as high functioning autism. This type of ASD affects a person's ability to socialize and communicate effectively with others. Children with Asperger's syndrome usually exhibit social awkwardness and an all-absorbing interest in specific topic [6]. Some of the symptoms include [61]:

• Engaging in one-sided, long-winded conversations, without noticing if the lis- tener is listening or attempting to change the subject

• Lack of eye contact, limited facial expressions, or awkward gestures

• Obsessing with one or two narrow subjects

• Lack of sympathy

• Diﬃculties at reading people

• Poor Coordination

• Speaking in a monotonous tone

12 2.3.2 Autistic Disorder

Autistic Disorder is characterized by signiﬁcant language delays, social challenges, and unusual behaviors. Children who meet more rigid criteria for a diagnosis of

autism have autistic disorder. Unlike Asperger's syndrome, they have more severe impairments involving social and language functioning as well as repetitive behaviors. Sometimes patients also have intellectual limitations [19].

2.3.3 Childhood Disintegrative Disorder

Childhood Disintegrative Disorder (CDD), also known as Heller's Syndrome, is a rare form of ASD. Unlike other forms of ASD, children develop normally until at least two years of age, only to subsequently display a severe loss of social, communication and other skills. Treatment for this condition involves a combination of medications, behavior therapy, among other approaches. Some of the symptoms include:

• Severe decline in ability to speak and have a conversation.

• Diﬃculty interacting with others

• Loss of interest in play and activities

• Losing motor skills, such as the ability to walk, climb, etc [20].

2.3.4 Rett’s Disorder

Affecting one in 10 thousand children, Rett's Disorder is a very rare form of ASD that occurs, almost exclusively, in girls. It is discovered during the first two years of life. One of the first symptoms of this condition includes a slowing head growth and loss of muscle tone. This condition is associated with coordination and muscular problems. Similarly, between the ages of 1 and 4, social skills deteriorate. The child stops talking and develops extreme social anxiety. This form of ASD occurs when a

13 mutation in a particular gene on the X chromosome takes place [21]. This disorder consists of 4 stages [22]:

• Stage I: This stage begins between 6 and 18 months of age. The symptoms are vague and hard to notice. Children begin to reduce eye contact and interest in toys. There may be delays in motor skills and decreasing head growth.

• Stage II: At this stage, the child loses the ability to use her hands and spoken language. It usually begins between the ages of 1 and 4.

• Stage III: This stage begins between the ages of 2 and 10. It can last for years. Apraxia, motor problems, and seizures are associated with this stage.

• Stage IV: This is the late motor deterioration stage, and it can last for decades. Scoliosis, muscle weakness, spasticity are related during this phase.

2.3.5 Pervasive Developmental Disorder-Not Otherwise Speciﬁed

Pervasive Developmental Disorder-Not Otherwise Speciﬁed (PDD-NOS) is a relatively new diagnosis, dating back to less than 20 years ago. It is considered a ‘subtreshold’ condition, in which some but not all features of autism are identiﬁed. It is characterized by delays in the development of socialization and communication skills. Some of the symptoms include [23]:

• Delays in using and understanding language

• Diﬃculty relating to people

• Unusual play with toys

• Diﬃculty with changes in routine or surroundings

• Repetitive body movement or behavior patterns

14 2.4 DIAGNOSIS

Although behavioral signs of ASD can emerge as early as six months, presently, less than 15% of children with ASD are diagnosed before age four [24, 26]. Early diagnosis of ASD makes a significant difference. Scientific research has shown that children that received more intensive therapy to combat social-communication impairments, especially at early ages, achieve the best outcomes [25]. There is currently no medical test to diagnose autism. Diagnosis is determined by physicians and psychologists through autism specific behavioral evaluations and personal observation [27]. When

Figure 2.4: Types of ASD and Functioning Levels [18]

ASD is diagnosed, a functioning level is established. Figure 2.4 displays the diﬀerent functioning levels that can be associated with each form of ASD. A patient diagnosed with Mild ASD can live and work independently. These patients usually have normal or above-normal intelligence and struggle with functional communication and social interactions. On the other hand, patients with Moderate ASD can have some level of independence but often require assistance. Moderate ASD patients have normal or below-normal IQs, are sensitive to sounds, and prefer to communicate through signs or technological devices. Finally, patients diagnosed with Severe ASD cannot live independently since they have a degree of mental retardation, lack language, and are socially unreachable. Recent studies show that autism severity can be assessed by

15 measuring how quickly the brain responds to sounds [18, 26].

2.5 TRADITIONAL TREATMENTS

Unfortunately, there is currently no cure for ASD. However, providing a child with special education can signiﬁcantly improve his or her social development and verbal communication skills [13]. Age of child, overall health and medical history, extent of the disorder, and symptoms are some of the factors that can help determine which speciﬁc treatment will work best [17]. Some of the current techniques utilized to treat ASD include Applied Behavior Analysis, Floortime, Pivotal Response Therapy, and Verbal Behavior Therapy.

Figure 2.5: Therapy Session [62]

2.5.1 Applied Behavior Analysis

Applied Behavior Analysis(ABA) focuses on how learning takes place. Its main goal is to bring positive change in behavior. For autism patients, therapists must customize the intervention and adapt to learner's skills needs and interests. Some of the ABA techniques include [40]:

• Variety of behavior analytic procedures

• Positive reinforcement for the learner

• No reinforcement for behaviors that pose harm

16 2.5.2 Floortime

Floortime consists of back-and-forth play interactions among parents, therapists, and children. Parents and therapists follow the lead of the child. Ideally, this takes place in a calm environment. Floortime aims to help children grow emotionally and intellectually by promoting [28]:

• Self-regulation and interest in the world

• Engagement in human relations

• Two-way communication

• Complex communication

• Emotionally based ideas

• Emotionally based thinking

Although this technique does not focus on speech, motor, or cognitive skills, the areas are still targeted through its focus on emotional development. By helping the child stay focused, parents and therapists help them to sharpen interactions and logical thinking [28].

2.5.3 Pivotal Response Therapy

Pivotal Response Therapy (PRT) is one of the best studied and validated behavioral treatments for autism. It is derived from ABA. PRTs goals are development of communication, language and positive social behaviors and relief from disruptive self- stimulatory behaviors. PRT focuses on targeting pivotal areas of the development of a child, including motivation, response to multiple cues, and imitation of social interactions instead of individual behaviors. As a result, this technique produces improvement across the diﬀerent areas of sociability, communication, behavior and academic skill building [39]. 17 2.5.4 Verbal Behavior Therapy

Verbal Behavior Therapy teaches communication using the principles of ABA. Ver- bal Behavior Therapy shows the child how to use language instead of focusing on vocabulary. Language is classiﬁed into four word types [38]:

• Mand: A request. It is considered the most basic type of language

• Tact: Comment used to draw attention

• Intraverbal: A word used to answer a question

• Echoic: A repeated word

Verbal Behavior Therapy helps children understand that communicating produces positive results [38].

18 CHAPTER 3 RESEARCH FOCUS

“Sometimes the questions are complicated and the answers are simple.” – Theodor Seuss Geisel

For ASD children, communication development happens diﬀerently and more slowly [43]. As discussed in chapter II, most ASD patients face diﬃculties using language, forming relationships, and interpreting and responding to the external world around them. This research work focuses on verbal communication, social communication, and adaptation skills.

3.1 VERBAL COMMUNICATION

Verbal communication in ASD children differs from patient to patient. While some are unable to speak, others have rich vocabularies and are able to talk about topics of interest in great depth. One of the major challenges for autistic individuals is using language effectively. Verbal ASD patients often say things that have no content. For example, an autistic child may count from one to five repeatedly. Echolalia, the repetition or echoing of something previously heard, is also common in ASD patients. A child with echolalia may answer a question with the same question. Similarly, a patient with rich vocabulary may be able to talk extensively about a topic of interest, but faces difficulties when participating in an interactive conversation on the same topic. Autistic children may also use stock phrases when starting a conversation. For example, they may start every conversation with “Hello, I am Tom,” even when speaking to people they know. Some ASD patients also have problems using pronouns

19 correctly. For example, when asked, “Are you wearing a red shirt today?” the patient may respond with “You are wearing a red shirt today” [41].

3.2 SOCIAL COMMUNICATION

It is important for children to be able understand the cues of others in order to be successful communicators. Because of the sensory challenges associated with ASD, patients show more interest in environmental sounds which makes them look dis- tracted while interacting with others. As a result, most autistic individuals do not make eye contact and have a poor attention span [41]. Eye contact is a necessary skill for eﬀective and vital communication. It builds trust and allows for connection with others [42]. ASD patients also face diﬃculties using gestures as a primary means of communication, like in sign language, and assisting verbal communication, like pointing to an object they desire [41].

3.3 ADAPTATION

Due to the diﬃculties children with ASD face when communicating and absorbing what they see and hear, understanding what happens around them is challenging. This can make them feel stressed and anxious when something unexpected takes place [51]. Common changes or new situations might include [50]:

• Leaving the house

• Having visitors

• Going somewhere new, such as the dentist

• Switching between toys, activities or tasks

• Doing things in a diﬀerent order from time to time for example, having a bath before dinner 20 • Eating new foods

In addition to needing help to manage changes to their daily routines, children with ASD often have trouble making plans. Having a rigid routine helps comfort them. Obsessions, routines and rituals might help patients cope with their surroundings and reduce their stress and anxiety by allowing them to control their environment and knowing what is going to happen next [51].

21 CHAPTER 4 LITERATURE REVIEW

“Research is what I’m doing when I don’t know what I’m doing.” – Wernher von Braun

There are many software applications dedicated to assist children with ASD and their care takers. The applications presented target the skills mentioned in Chapter 3, which are Verbal Communication, Social Communication, and Adaptation skills as shown on Figure 4.1. Depending on their level of interaction with the user, these

Figure 4.1: Classiﬁcation of Software for Children with ASD

categories can be broken down even further into the following subcategories: static, interactive, and adaptive. Static applications are those that involve constant media, such as ﬂashcards, DVDs, etc. On the other hand, an interactive application provides a user with feedback and allows him or her to interact with it up to some extent. A good example of this type of interaction is a computer application. Finally, adap-

tive applications use a user's past progress in order to adapt the diﬃculty levels of future activities [29]. The applications presented fall under the interactive and static subcategories as shown in Table 4.1 .

22 Existing Technology for Children with Autism Spectrum Disorder

Target Skill: Verbal Social Adaptation

Interactive Proloquo2Go Tom Cat, Training Autismate, First Faces Then Visual HD

Static Vast Autism Core 1 The Transporters Zac Browser

Table 4.1: Existing apps for children with ASD

4.1 TECHNOLOGY TO ENHANCE VERBAL COMMUNICATION SKILLS IN ASD CHILDREN

Twenty ﬁve percent of individuals diagnosed with ASD are non-verbal [32]. The following applications aim to aid children with ASD to express their thoughts and needs.

4.1.1 Interactive Applications

Example 1:Proloquo2go

Figure 4.2: Vocabulary - Proloquo2go [63]

Proloquo2Go is an Augmentative and Alternative Communication (AAC) application that helps children to construct sentences using symbols and pictures as shown on Figure 4.2. About 50% of its users are Autism Spectrum Disorder patients. This app features text-to-speech, word prediction, and a customizable vocabulary and inter- 23 face. For those who cannot read, it facilitates symbol-based communication using the SymbolStix symbols. For literate users it also supports a typing view with advanced self-learning word prediction. Using natural sounding text-to-speech voices, users can use Proloquo2Go to say anything they want. This application runs exclusively on iOS devices [34].

Figure 4.3: Proloquo2go Features [64, 63]

Proloquo2Go has been designed to be extremely flexible, so it can be adapted to different users and their user base consists of a wide variety of diagnoses and ages. As shown on Figure 4.3, it can be used across different applications and has different voices available. Pros of Proloquo2Go

• Prevents accidental selections for users with motor impairments

• The app grows with the user

• Access to pre-recorded expressions and sounds

• Easy to use

• Natural sounding voices available

• It can be used to send emails, messages, tweets, and social media posts

• Multi-user support

• No internet connection is necessary 24 Cons of Proloquo2Go

• Users are limited to images provided by the app

• Only available for iOS devices

Figure 4.4: Results of the Evaluation of Proloquo2go [63]

In the year 2012, this app was used to measure the improvement catalyzed by full featured AAC applications in diﬀerent areas. The results, displayed in Figure 4.4, indicate that this app is eﬀective to improve its users' communication skills, behavior, independence, school performance, among others. Proloquo2Go has been used successfully with individuals diagnosed with autism, cerebral palsy, Down syndrome, developmental disabilities, apraxia, stroke, traumatic brain injury, among others [63].

4.1.2 Static Applications

Example 1:Vast Autism 1 Core

Vast Autism 1 Core aids autism patients with speech diﬃculties by using VAST, an innovative application of video technology to facilitate and improve communication

25 abilities for speech-impaired individuals [52]. By combining the concept of video modeling, which has proven to be very eﬀective, along with written words and auditory cues, this application helps individuals acquire relevant words, phrases and sentences. For patients with strong visual skills, this can be a key to developing speech [35, 53].

Figure 4.5: Vast Autism 1 Core Main Menu (Left), Modeling Video (Right) [52]

Ongoing research indicates that students are highly interested in VAST videos and will almost immediately attempt lip movements or touch their mouths in response to the models. Following close-up video of mouth movements allows users to become able to speak full sentences [52]. After a few short weeks many patients who were essentially nonverbal begin word approximations and word attempts more readily. These videos are organized into a hierarchy of 5 categories [35]:

• Syllable Repetition

• Single Syllable Words

• Multi-Syllabic Words

• Phrases

• Sentences

Each video gives a spoken target utterance that is preceded by the written word, phrase, or sentence. Each word, phrase, and sentence is concrete and has meaning that 26 Figure 4.6: Video Categories [52] can be generalized and practiced. Providing the written word enhances acquisition of spoken language and increases functional communication. VAST Autism has yielded better than expected results. It is a great tool for therapists and has helped many autism and apraxia patients [35, 42]. Some of the pros of this app include:

• The 5 categories allow students from diﬀerent levels to beneﬁt from this app.

• The vocabulary can be used by patients in their daily lives.

• Patients can see how the word is spelled.

• The words are presented with pauses so that the patient can repeat afterwards.

• The app has a mirror feature that allows its users to monitor lip movement while pronuncing a word.

• Vast Autism is available in other languages.

• This tool teaches communication in a traditional and natural manner.

• The app is easy to use and organized.

The cons of this application are:

• Only available for iOS devices.

27 • The vocabulary presented to patients cannot be easily customized. Patients are limited to the default content unless personalized videos are requested at cost.

• The application does not track users' progress.

4.2 TECHNOLOGY TO ENHANCE SOCIAL COMMUNICATION SKILLS IN ASD CHILDREN

As mentioned earlier in Chapter 3, children under the spectrum struggle understanding social cues. They have a hard time connecting with others and knowing how to behave properly in certain settings. The following applications focus on enhancing their emotion recognition and social interaction skills.

4.2.1 Interactive Applications

Example 1: Training Faces

Figure 4.7: Training Faces Interface [53]

Specially designed for Asperger's Disorder patients, this application enforces emotion recognition skills. Training faces is set up as a game in which the user must

28 identify the correct emotion in order to move on to the next round and reach his destination. The game exposes the user to 9 different emotions, which include happy, sad, excited, afraid, angry, sick, tired, confused and silly. To start the game, the player selects an emotion. Each emotion is linked to a train route. Once selected, the train route is displayed, and the player is told where the train is going. Next, an example on why a person would feel that particular emotion is provided. Two train cars with five people each are displayed. This process is shown in Figure 4.8. Then, the user must identify and tap the pictures of the characters that show the selected emotion in order to move to the next station. This application has speedup recognition and gets more difficult after each round [53].

Figure 4.8: Training Faces Emotion Recognition Process

This fun and novel game motivates children with ASD to pay attention to facial expressions and understand the emotions link to them. Something, that in regular occasions, is uninteresting to them. The game promotes emotion recognition by cheering children when identifying emotions correctly. As discussed in Chapter 2, motivation is very eﬀective for treatment. Also, the usage of trains makes it appealing to ASD children. Some of the pros include:

• The game challenges the patient by adding diﬃculty after each round

• It is a fun way to make patients identify emotions and understand the causes behind them 29 • Easy to use

• Intuitive interface

The cons of this app are:

• Limited number of emotions

• Not customizable

Example 2:Talking Tom Cat 2

Figure 4.9: Tom Cat [37]

Talking Tom Cat 2 was not created to aid ASD patients, yet this popular game has gained an enormous amount of positive feedback from the autism community. Talking Tom Cat repeats anything that a user says to him in a funny high-pitched voice. It reacts to gestures and commands. If stroked, it purrs. When its belly is poked, it gasps. It jumps when scared. Talking Tom Cat 2 also includes a new friend

of Tom's, a dog named Ben, who taunts Tom and provokes comical reactions. Any interaction with Tom can be recorded and shared on Facebook and YouTube [65]. Talking Tom Cat 2 is a great application to make social engagement for ASD children less intimidating since Tom Cat is appealing and inoﬀensive to most children. 30 The con of using this app to treat ASD is that Tom only repeats what the patient says. Some of the pros include:

• Easy to use

• Intuitive

• Engaging

• Helps understand social cues since Tom interacts with Ben and with the patient

• Fun

• Available across diﬀerent platforms

• Interactions with Tom can be recorded

4.2.2 Static Applications

Example 1:The Transporters

The DVD ‘The Transporters’ was created especially for ASD children. This show aims to help them recognize the causes of emotion and the facial expressions associated with them. There are 8 toy vehicles with their own personalities and function. The setting of the show is a child‘s bedroom since ASD children prefer predictable environments. In order to ease the transferring of their learning into real life, each character has a real human face. ASD children do not naturally want to look a people‘s unpredictable faces, yet the transporters encourage them to do so by presenting facial expressions in a less intimidating manner. The series has been evaluated by the Autism Research Center for its eﬀectiveness for children aged 4 to 8 with ASD. The results show that [30]:

• Most children caught up their typically developing peers in all tasks tested

31 Figure 4.10: The Transporters

• The results suggest that The Transporters DVD is an eﬀective way to teach emotion recognition to children with ASD and that the learning obtained from this DVD can be used in day to day situations

• Children with ASD who did not watch the DVD remained below typically developing levels

Pros:

• It encourages ASD children to look at faces

• Children can be tested on what they learned since each DVD had quizzes

Cons:

• The human faces on the trains can be intimidating for some children

32 Figure 4.11: First Then Visual HD Icon

4.3 TECHNOLOGY TO ENHANCE ADAPTATION SKILLS OF ASD CHILDREN

4.3.1 Interactive Applications

Example 1:First Then Visual HD

ASD patients require a clear, set schedule to help prevent them from acting out as a result of something unexpected. They have greater difficulties coping with unstructured time than neurotypical people and benefit from increased structure in their lives. Research and anecdotal evidence has shown that visual schedules increase their independence and help them manage their anxiety. Picture schedules can also help patients improve their communication skills by providing a visual image for their feelings, thoughts, wants, or needs. They can help the patient remain calm and reduces inappropriate behaviors When a child uses, or is shown, pictures of routines/schedules, transitions may be easier since the child can see what is happening now and what comes next [66]. The First Then Visual Schedule HD makes it possible for ASD individuals to create a picture based schedule as shown on Figure 4.12. Each event has a timer. This feature allows steps to be further tailored to the unique needs of the individual, and can be set to include the entire event or for each step within a schedule. Users can check off a task as it is completed. Schedules can be viewed in multiple forms, which contributes to a more user-friendly and individualized experience. Schedules 33 Figure 4.12: First Then Visual HD Interface

can be created using a variety of image sources including the web and can be created to show the steps within speciﬁc daily events. This is displayed in Figure 4.13. Also, they can serve as a reminder of events throughout the day [67].

Figure 4.13: Schedule and instructions to complete a task [67]

This application not only helps caretakers to organize the time of their patients but also helps ASD patients develop independence, which increases their self-esteem in the process. 34 The pros of First Then Visual HD include:

• Customizable content

• The formatting of the schedule can be adapted to a patient‘s preference

• Simple, clean and easy to use

• Combines text, pictures, audio and video, which is engaging for children

• Intuitive interface

• Cost eﬀective. With this app, there is no need to print and laminate schedules anymore

The cons of First Then Visual HD include:

• Only available for iOS devices.

Example 2:AustiMate

AustiMate is an AAC application that allows ASD patients to build sentences. Unlike Proloquo2Go, AustiMate allows its users to add their own pictures, videos and voice recordings to create visual scenes that promote social skills and communication in children with ASD. This application helps children with ASD not only to formulate sentences but also to successfully complete their daily routines. Through AutisMate, a care taker can create scenes by taking photos of rooms in their home and use them to construct interactive scenarios for autistic members of their families. These scenes can later be accessed by ASD patients and assist them to complete basic tasks. For example, a photo of the bathroom can point a child with ASD to helpful videos on how to brush his teeth as shown in Figure 4.14. Since autistic patients feel more comfortable dealing with familiar situations, being able to fully customize AutisMate makes it more eﬃcient [68].

35 Figure 4.14: AutisMate's User Interface [68]

This app was the recipient of the About.com 2013 Readers' Choice Award Win- ner in the Favorite Special-Needs App category. It has eased the lives of children with ASD, cognitive impairments, developmental delays, speech impairments, among others. Future updates to this tool will bring features such as in-scene task analysis, visual schedules with rewards for completing all tasks, and an in-app text to speech keyboard for users that can type [68]. Some of the pros of using this app are:

• Helps ASD patients with their schedules while enhancing their communication skills

• Care takers can upload their own pictures to create scenes

• It has over 12,000 included symbols, custom voice recordings, and synthesized voices to fully and easily customize the app for its user’s current and future needs

• The app integrates GPS, so it can show kids with autism relevant scenes based on where they are located

• Available for iOS and Android

• Content can be synced up to 5 devices

36 4.3.2 Static Applications

Example 1:Zac Browser

Figure 4.15: Zac Browser

For autistic children, it can be very hard to focus on a specific task. Dealing with computers, where a multitude of websites and sounds are available, can overwhelm a child with ASD. ZAC is the first web browser developed specifically for children with autism and autism spectrum disorders. It is targeted towards kids with low, medium and high functioning autism [10, 11].

Figure 4.16: YouTube video accessed through Zac Browser [9]

ZAC provides a full-screen interface with access to games, music and other online activities that cater speciﬁcally to kids who display the characteristics of autism 37 spectrum disorders, such as impairments in social interaction, impairments in communication, restricted interests and repetitive behavior. This software takes over the computer and reduces the controls available. By narrowing down the focus, Zac Browser opens a world of possibilities for children with ASD [10, 11]. Some of the pros of using this web application are:

• Filters the internet for children

• Appealing aquarium theme

• Allows children to experience independence on the web

38 CHAPTER 5 METHODOLOGY

“Research is to see what everybody else has seen and to think what nobody else has thought.” – Albert Szent-Gyrgyi

5.1 OBJECTIVE

To reach the goal of creating a system that will supplement therapy and enhance the verbal and social communication skills of those with ASD, the factors that affect these two skills were identified. Scientific literature and recorded therapy sessions provided a better understanding of the needs of ASD patients and contributed to the definition of the core target functions of the system, Ying. The next step was to investigate traditional treatment for children and the reasons they are considered efficient. This step played an important role since it contributed to the integration of some of these techniques into the system. In the same way, reviewing traditional treatment helped to understand the design of existing software for ASD children and determine if their features were based on scientific research. These applications were reviewed and classified in chapter 4. The requirements of Ying were determined after observing the different features and characteristics of the applications mentioned on chapter 4. To find components patients could benefit from, existing ASD technology was reviewed. Some of these included eye tracking and voice recognition. The features from the previous version of Ying were also a topic of discussion. The software implementation will contain

39 the features derived from the supporting scientific literature, existing technology, previous version of Ying, as well as other features discussed. These can be turned on or off depending on the needs of the patient, and caretakers can control the activities to which the patient is exposed. As mentioned in Chapter 3, children with ASD may have difficulty communicating verbally and non-verbally, such as through hand gestures, eye contact, and facial expressions. The goal of this research is to create a system that works hand in hand with therapists in order to assist ASD patients in overcoming these challenges.

5.2 APPROACH

Figure 5.1: Logo of the Ying system

Ying is an application that aspires to reinforce the verbal and social communication skills of ASD patients through a series of educational, interactive, and fun games. Through this application, therapists or care takers have the power to schedule educational activities for a child. As a result, Ying is customizable and could be tailored towards the speciﬁc condition of an individual. Ying also provides care takers with the possibility to assess progress and identify learning patterns. Therapist and care takers have the option to motivate and encourage children's eﬀorts just like in traditional therapy. To achieve the objective of this research, Ying consists of 6 modules:

40 Figure 5.2: Overview of Ying

• Learn: In this module, children acquire skills in which they will be tested later

• Play: This module allows children to apply the skills acquired from the learn module

• My Schedule: Patients can see and access the activities assigned by their caretakers

• My Rewards: This module displays a list of the rewards earned by the user

• Progress: Caretakers can see the progress of a patient over time

• Conﬁguration: Caretakers can customize the application to meet the needs of a child using the conﬁguration module

41 5.2.1 Learn

Learn Activity 1: Virtual Flashcards

The activity Virtual Flashcards, shown in Figure 5.3, aspires to help ASD patients with their verbal communication skills. Communication is the process by which information is exchanged. Communication of language begins with spoken sounds com- bined with gestures, relying on two different types of skills. Children first learn to receive communications by listening to and understanding what they hear along with gestures; next, they experiment with expressing themselves through speaking and gesturing. Speech begins as repetitive syllables, followed by words, phrases, and sentences. Later, children learn to read and write. Photo language cards, flashcards and visual support help to increase vocabulary, promote language development, and strengthen communication skills when teaching children with autism spectrum disorders [72].

Figure 5.3: Virtual Flashcards for Words

In order to help ASD patients with their verbal communication skills, Ying oﬀers the activity Virtual Flashcards, in which a child has the option to hear the pronunciation of the word or see the spelling of it. Therapists control the type of words shown 42 on the ﬂashcards. These can vary from pronouns to objects.

Figure 5.4: Virtual Flashcards for Phrases

In the same way, patients will be able to learn phrases thanks to a similar system that allows them to hear and read. The vocabulary was created for this game. The phrases module was built in order to show children how to articulate the learned words and how to put them together in ways that are acceptable to the people around them. This is an important step in language acquisition [69]. This module also helps them to understand words and use them eﬀectively, which is a challenge faced by ASD patients [73].

Learn Activity 2: Story Teller

The activity Story Teller enhances emotion recognition skills in children with ASD. Through this training, children can learn about the 7 basic emotions, which have been proven to have a facial expression associated with them. These emotions include anger, fear, disgust, contempt, joy, sadness, and surprise [74]. A child's understanding of emotion is embedded in a narrative structure. Therefore,social stories are a good way to help ASD patients understand emotions since they evoke a child's narrative

43 Figure 5.5: Story Teller structure and associate a facial expression to the emotion evoked by the narrative [75].

Learn Activity 3: Eye Gaze

Children with ASD also tend to look at faces in unusual ways, often preferring to look at the mouth rather than the eyes [77]. This may help explain why they miss social cues and have trouble interacting with others. In the training system, the game ‘Eye Gaze’ from the original version of Ying is included in this system [29]. The purpose of this game is for the child to follow the eyes of the avatar and point to the object the avatar looks at, which encourages the child to pay attention to the eyes.

5.2.2 Play

In order to enhance the verbal communication skills of ASD patients, The following games were developed. The vocabulary used by these games is taught in the learning module. Some games work with voice recognition.

44 Figure 5.6: Eye Gaze

Play Activity 1: Say my Name

Figure 5.7: Say My Name

Say my name is a game that teaches words to children. The purpose of the game is for children to say the name of the object in display. Different difficulty levels are available. As difficulty increases, patient has less time to say the name of the object.

45 Play Activity 2:Message in a Bottle

Message in a Bottle allows children to interpret pictures, decode, and say the message out loud. For example, in Figure 5.8, the patient sees an eye, a heart, and two people. In order to win this game, they must say ‘I love my friends.’

Figure 5.8: Message in a Bottle

Play Activity 3:Take a Peek

Take a Peek consists of an image completely covered. The image slowly reveals itself. The purpose of the game is for children to identify the object in the image and say its name before the image is completely uncovered.

Play Activity 4: TappyTime

Tappy Time promotes eye contact in children with ASD. It is very similar to Eye Gaze since it incorporates the avatar component. In this game, the avatar moves its eyes around diﬀerent locations. The objective is to follow the eyes of the avatar and select the same circles in the same order.

46 Figure 5.9: Take a Peek

Figure 5.10: Tappy Time

Play Activity 5: Fill in the Blank

The purpose of Fill in the Blank is to ﬁll in the blank with the most logical event. The child is given two events, the cause and the result. The cause is a situation, while the result is an emotion. The patient must identify the reason behind the outcome.

47 Figure 5.11: Fill in the Blank

5.2.3 Schedule

The Schedule Module consists of a set of activities assigned by a therapist or care taker. Patients must complete a schedule, if available before proceeding to other activities. When a new schedule is assigned, the patient is notified. ASD patients require a clear, set schedule to help prevent them from acting out as a result of something unexpected. Individuals with ASD have greater difficulties coping with unstructured time than neurotypical people and benefit from increased structure in their lives. Research and anecdotal evidence has shown that schedules increase their independence and help them manage their anxiety [66].

5.2.4 Rewards

In the rewards module, care takers can assign a reward to a child for completing a set of activities successfully. As explained on Chapter 2, when a good behavior is followed by some sort of reward, the behavior is more likely to be repeated [40].

48 Figure 5.12: My Schedule Module

Figure 5.13: Rewards Module

5.2.5 Progress

Through the Progress Module, therapists and care takers can keep track of a child's progress. When a child plays a game, he accumulates points. Care takers can see the accumulation of points overtime by activity or by targeted skills.

49 Figure 5.14: Progress Module

Figure 5.15: Conﬁguration Module

5.2.6 Conﬁguration

The Conﬁguration Module allows care takers to add or remove features from the system in order to improve the patient's experience.

50 CHAPTER 6 IMPLEMENTATION

“I hear and I forget. I see and I remember. I do and I understand.” – Confucius

6.1 SYSTEM ARCHITECTURE OF YING

Figure 6.1: Architecture of Ying

Ying is supported by a three-tier architecture and a web server, which is used solely for image hosting. The three tier-architectures includes a presentation layer, a business logic layer, and a data layer. The following sub-sections describe each layer in details.

51 6.1.1 Presentation Layer

The Presentation Layer is the top most level layer in the system. It is the user interface and therefore encompasses all modules and activities available as shown in Figure 6.1. The role of this layer is to translate tasks and results into something users can understand [59]. It functions using both a presentation technology and asynchronous programming. The presentation technology is the user interface. The presentation technology used in this system is Extensible Application Markup Language (XAML). Asynchronous programming refers to all the events that take place when the user interacts with the system. This last component is implemented in C#. All content presented in the Presentation Layer is obtained from the Business Logic Layer.

6.1.2 Business Logic Layer

The main role of the Business Logic Layer is to move and process data between the two surrounding layers. It also processes commands, makes logical decisions, and per- forms calculations [59]. This layer consists of a Windows Communication Foundation (WCF) service. Its operations are implemented in C#, and data is requested to the database via pass-thru SQL queries. The WCF service in this system contains both data and service contracts. As explained in the components section of this chapter, the data contracts describe the data to be exchanged, while service contracts describes the operations available on the service endpoint [56]. The Data Contracts, shown in ﬁgure 6.2, include:

• Flashcard: This contract holds all data related to a ﬂashcard, including label information, url of the image, and category.

• Scene: This contract contains all data related to a scene in a social story, including a description of the scene, url of the image, and its order in the story.

• Story: This contract contains all data related to a social story, including a list 52 Figure 6.2: Data Contracts in Ying's Service

of scenes, name of story, and the category or emotion associated to the story.

• Peek: This contract contains all data used by Take-a-Peek, including image in display, correct answer, and category.

• EyeGaze: This contract contains the url of the avatar and the direction it is looking at.

• Reward: This contract contains all information related to a reward including description, url of image, and a bool to determine if it has been earned or not.

• Conﬁguration: This contract contains a list of the activities available to the patient and a check that determines if speech commands are active or not.

• TappyTime: Like EyeGaze, this contract contains the url of the avatar animation in addition to a list of directions.

• Progress: This contract contains all information to be graphed.

53 • Schedule: The schedule contract holds all data related to a schedule.

• Activities: An activity is composed of name, diﬃculty level, category, and module.

The service contracts of this service include:

• Login: This contract is used for user authentication. It also gets the user role if the user is identiﬁed.

• GetUserSettings: This contract retrieves the settings associated with a user.

• LoadSchedule: This contract retrieves any new schedules available for the user in session.

• RecordActivity: Once an activity from the schedule is completed, this contract sets it as completed in the database.

• GetProgressReport: This contract retrieves all data needed to generate a progress per activity graph.

• GetGaze: This contract retrieves a list of all diﬀerent gazes and the animations related to them.

• GetCardContent: This contract retrieves content for ﬂashcards.

• RecordScore: This contract records points earned in the play module.

• LoadRewards: This contract loads available rewards.

6.1.3 Data Layer

The data layer consists of a relational Microsoft SQL database. It stores all the data used by this app, including the links to access images, settings, and user records. Some of the data recorded includes scores per activity and the date the activity was 54 Figure 6.3: Ying's Database Diagram completed with the purpose of measuring improvement over time. In the same way, the schedules assigned to a user are recorded.

6.2 MAIN COMPONENTS

6.2.1 Speech Recognition

Speech recognition is the translation of spoken words into text. Ying works with the Microsoft Speech Recognition Engine. This particular engine supports Speech Recognition Grammar Speciﬁcation(SRGS) and enables such grammar to be anno- tated with semantic information. This means that a recognition result may contain not only the recognized text but also the semantic interpretation of that text. This makes it easier for applications to consume recognition results. A grammar is created in the presentation layer of Ying. The contents of this grammar come from the database and are retrieved by the business logic layer. Then, the user sends a voice command, which goes to the Speech Recognition Engine. The string returned by the Speech Recognition Engine is then compared against the existing grammar. If there is a match, the word or phrase is accepted. Otherwise, the word is considered

55 Figure 6.4: Speech Recognition Component unknown and the user is prompted to repeat the word or phrase as shown in Figure 6.4.

6.2.2 Text to Speech

Figure 6.5: Text-to-Speech Engine

A speech synthesizer is a computer system that artiﬁcially produces human speech. A text-to-speech (TTS) engine converts normal language text into speech. This engine is composed of a front-end and a back-end. The front-end converts raw text containing symbols and abbreviations into the equivalent of written-out words. This process 56 is often called text normalization, pre-processing, or tokenization. The front-end then assigns phonetic transcriptions to each word, and divides and marks the text into prosodic units, like phrases, clauses, and sentences. Phonetic transcriptions and prosody information together make up the symbolic linguistic representation that is output by the front-end [77].

Figure 6.6: Text-to-Speech Process

Ying works with the Microsoft Text-to-Speech Engine to provide users with visual- audio cues. Thanks to this application program interface(API), the process of gen- erating speech from a string is straight forward. As shown in Figure 6.20, a string is sent to the Text-to-Speech Engine. Then, this engine imposes it as output speech. Currently, all activities in the learn and play modules utilize Speech Synthesis.

6.2.3 Graphing

The graphing tool is used by the Progress Module in order to graph the progress of users in real time. The user can request scores for a certain period of time for a particular game. This request is then sent to the WCF service in the Business Logic Layer. Then, this layer prompts the Database for the data and feeds it to the graph in the presentation layer. One of the challenges faced during the implementation of

57 Figure 6.7: Graphing Tool this component is that Microsofts graphing functionality is not available for Windows Phone 8.0. After doing some research, it was found that this library can be obtained from a regular windows form 8.0 and added to the Windows Phone 8.0 project.

6.2.4 The Avatar

Figure 6.8: Avatar

The avatar, shown in Figure 6.8, is a cartoon version of the character Thomas, from Thomas and Friends. It was slightly altered to facilitate the creation of the animations needed for Eye Gaze and Tappy Time. The avatar from the original version of Ying was powered by a third party application called Haptek, which only runs on Internet Explorer. In this approach, the avatar is a simple gif animation 58 created in Adobe Photoshop CS5. This approach avoids compatibility issues, since gif animations can be played by most platforms. One of the challenges faced during the integration of this component was adding the avatar to the screen. The format of the picture was accepted, but the animation did not play. An image converter and an external image tools library were used to make the animation happen. The gif animations are hosted by a web server.

6.3 MODULES

Figure 6.9: Modules

Ying consists of 6 modules, which are Progress, Rewards, Conﬁguration, Schedule, Learn, and Play. As shown in Figure 6.9 and 6.1 and mentioned in chapter 5, the activities in the Learn and Play modules are divided into two subcategories; verbal and social communication. Each subcategory has games or activities that enhance that particular skill.

59 6.3.1 Learn

Figure 6.10: Learn Module

The purpose of the Learn Module, see Figure 6.10, is to introduce ASD patients to the material they will be tested on the Play Module. The Learn Module consists of 3 different games, which includes Virtual Flashcards, Story Teller, and Eye Gaze. Each game enforces a different skill. Virtual Flashcards exposes children with ASD to different words divided into categories. This activity focuses on reinforcing verbal skills. Through each interactive flashcard, a child can see the spelling and hear the pronunciation of the image displayed as many times as he/she needs. In the same way, Story Teller introduces children with ASD to emotions. Each story in Story Teller teaches patients one of the seven emotions that have a facial expression associated to them. This way, children cannot only learn the facial expression associated with an emotion but also understand the possible causes behind it. Finally, Eye Gaze encourages children to engage in eye contact. In this game, the child must follow the eyes of a friendly avatar and tap on the object the avatar is looking at. These activities will be described in more detail later in this chapter. In this system, therapists have the option to select which modules are available to a patient. They get a list of all activities, and they can assign them to a user.

60 6.3.2 Play

Figure 6.11: Play Module

The purpose of the Play Module is to get the patient to apply the skills obtained from the Learn Module. The Play Menu is composed of diﬀerent games that apply the social and verbal communication skills enforced in the Learn Module. These games include Say my Name, Message in a Bottle, Take a Peek, Tappy Time, and Tell me a Story. Unlike the Learn Module, the activities in this module are rewarded with points based on performance. These points are later used to measure improvement, increase diﬃculty levels, and reward the child.

6.3.3 Schedule

The Schedule Module allows therapists to create a schedule for children to complete on a certain day. This schedule contains activities from both the Learn and Play modules.

61 Figure 6.12: Schedule Module

6.3.4 Rewards

Figure 6.13: Rewards Module

The Rewards Module allows therapists to rewards patients for completing their scheduled activities, as seen in Figure 6.13.

6.3.5 Settings

The Conﬁguration Module gives therapists the option to ﬁlter activities out of the game, as shown in Figure 6.14.

62 Figure 6.14: Conﬁguration Module

Figure 6.15: Progress Module

6.3.6 Progress

The Progress module allows parents and care takers to see how the patient is doing in the diﬀerent activities and put more emphasis on the skills that need to be reinforced. In this image, it is observed that the results of the activities from the learn and play module are retrieved by the progress module. Once the therapist or caretaker has this information, he or she can proceed to modify the diﬃculty and settings of these

63 activities to ﬁt the needs of the child.

6.4 PLATFORM

Ying can be accessed from a Windows 8.0 Phone. Being used by 1.5 billion people every day, Windows is one of the most popular and accessible operating systems in the world [54]. Part of its success is due to its consistent and very easy to use user interface. Although Windows Phones are not the most used mobile devices, this platform was chosen due to its familiarity and consistency. The Windows 8 Phone interface is very similar to its Desktop version. As mentioned in Chapter 3, consistency is a very important factor when it comes to dealing with ASD patients. Also, this platform was chosen because it provides the capability of creating Windows and Mobile Applications simultaneously. In the future, this system can be easily converted to a windows application for desktop and tablets, and more patients would have access to it.

6.5 ACTIVITIES

6.5.1 Virtual Flashcards

Virtual Flashcards focuses on enhancing a patient's verbal communication skills. Once a user selects this module, he can choose between words and phrases. Both options consist of flashcards with a single word on them. The difference is that the Phrases category classifies its content into pronouns, verbs, and objects, while in the Words category, patients can learn about different topics including animals, emotions, colors, etc. Unlike the Words Category, the Phrases Category, classifies its content as Pronouns, Verbs, and Objects. The Pronouns subcategory includes the basic subject pronouns; which are I, You, He, She, It, We, You, and They. The Verbs subcategory consists of images that represent actions. Similarly, the Objects subcategory exposes

64 Figure 6.16: Virtual Flashcards children to a similar vocabulary as the one in the Words category. This learning activity is recommended for kids who are already familiar with the vocabulary. In addition to a visual representation of the word, patients can see the spelling of the word under the image. In the same way, by tapping on the flashcard, patients can hear the pronunciation of the word as many times as needed. The next button takes them to the next flashcard. Virtual Flashcards use the Text-to-Speech component described earlier in the chapter. Every time the user selects a category, the content of the flashcards are loaded from the database through the service contract ‘GetCardContent,’ and it is retrieved in an array of type flashcards. As mentioned in the architecture component, the images on the flashcards are received as urls from the database. These images are hosted in a lamp server.

6.5.2 Eye Gaze

Eye Gaze focuses on enhancing social communication skills in ASD patients. This learning activity consists of an avatar surrounded by up to four objects. The avatar moves its eyes in four diﬀerent directions; up, down, left, right. The objective of the

65 Figure 6.17: Eye Gaze game is to engage children in eye contact and reinforce their understanding of visual cues by making the patient follow the eyes of the avatar and tap on the object the avatar is looking at. The diﬃculty of this activity can vary depending of the child. As the child progresses, more objects can be added. Also, a background can be added as a distraction element. Eye Gaze uses the Text-to-Speech component described earlier in the chapter to prompt the user to follow his gaze. Every time the user selects this game, all possible eye gazes are loaded from the database through the service contract ‘GetGaze,’ and it is retrieved in an array of type EyeGaze, which contains all urls to gif animations and directions the avatar is looking at.

6.5.3 Story Teller

Through Story Teller, ASD children can hear short stories that introduce them the diﬀerent 7 emotions mentioned in Chapter 5. Each story consists of plot line that describes why the main character feels in a certain way. As mentioned in Chapter 5, It can be hard for a child with autism to understand his or her own feelings and the emotional responses of others. Stories about feelings and appropriate emotional

66 Figure 6.18: Story Teller responses can help an autistic child learn how to understand emotions in context [44]. Story Teller works very similarly to Virtual Flashcards. The only diﬀerence is that the data is retrieved as a Story, which is an array of scenes.

6.5.4 Say my Name

Figure 6.19: Say My Name

Say my Name reinforces the skills learned from the Virtual Flashcards activity. 67 Just like in Virtual Flashcards, patients are presented with a visual representation of a word. The objective of the game is to say the name of the object after pressing the ‘Say It’ button. Once they guess the word correctly, the Next button appears and they can move on to the next word. The patient has 3 attempts for each word. Depending on which attempt they get the word right, they earn 3 , 2 or 1 points. The patient can get one hint per flashcard, which would be displaying the label of the flashcard. This game uses the speech recognition component described in the components section of this chapter. To load the flashcards, this activity utilizes the same mechanism as Virtual Flashcards.

6.5.5 Take a Peek

Figure 6.20: Take a Peek

Take a Peek is another verbal communication enhancing activity, and it supports the material learned in the Virtual Flashcards activity. This activity is very similar to Say my Name. The user is presented with an image completely covered. This image then reveals itself in 10 seconds. Once the user identiﬁes it, he must press the Say It button and say the word outloud. Points are given if the word is guessed correctly

68 before time is up. This game uses the speech recognition component described in the components section of this chapter. To load the content, this activity utilizes the same mechanism as Virtual Flashcards.

6.5.6 Message in a Bottle

Figure 6.21: Message in a Bottle

Message in a Bottle focuses on enhancing patients' verbal communication skills. This activity targets the skills learned in the Phrases category of the Virtual Flash- cards activity. The purpose of this game is to help autistic children form sentences with the vocabulary learned. The patient is presented with 3 images that represent a pronoun, a verb, and an object. Once they know what the sentence is, the press the ‘Say It’ button and say the word out loud. Points are given in this module as well. This game operates using the same mechanism as Say my Name.

6.5.7 Tappy Time

Like Eye Gaze, Tappy Time engages children in Eye Contact. The avatar looks at the diﬀerent circles around it, and prompts the user to replicate the pattern. It operates with a similar mechanism to Eye Gaze. 69 Figure 6.22: Tappy Time

6.5.8 Fill in the Blank

Figure 6.23: Fill in the Blank

Fill in the Blank is very similar to Story Teller. The user is exposed to a story that consists of 3 scenes. One of these parts is missing, and the patient must ﬁgure out which one it is. The Story, an array of scenes, is loaded. The Presentation Layer hides the a part of the story, and then asks the user to complete the story.

70 CHAPTER 7 CONCLUSION

“We keep moving forward, opening new doors, and doing new things, because

we're curious and curiosity keeps leading us down new paths.” – Walt Disney

7.1 RESEARCH CONTRIBUTION AND SUMMARY

Due to the prevalence of ASD in society and the high cost of specialized behavioral treatment, it is of vital importance to provide tools that make intensive treatment more accessible. The contribution of this study consisted of converting the legacy web application into a customizable Windows-based mobile application that complements therapy and reinforces verbal and social communication skills in ASD children while recording data for later study. This research project brought Ying up to the next level by incorporating new educational activities for ASD patients. This approach also provide therapists with tools to address the communication challenges ASD patients face and gives them the ﬂexibility to remotely assign tasks to their patients. At the same time, data is collected from the interaction between a child and a series of educational, interactive, and fun games. Once completed, the results of these activities will be available to therapists for evaluation and therapy improvement. The new Ying incorporates aspects of eﬀective traditional therapy, such as rewarding and positive reinforcement, as well as cutting edge technology, such as voice recognition. The architectural foundation of this system and its features can easily be expanded upon and further developed in the future. A tool like Ying cannot only help Autistic

71 children but also others that experience similar diﬃculties with communication.

7.2 EVALUATION

The system was evaluated by professional behavioral and speech therapists. The Evaluation process consisted of two components, which are a video and a questionnaire. Therapists were asked to watch a short video of the application and complete a questionnaire when they were done. The video included the description and purpose of each module followed by a demonstration of each activity. The questionnaire is included in Appendix A, and it consisted of questions regarding the usability, functionality, and eﬃcacy of the system. The results received showed that therapists would use the system in a regular therapy session. Each activity would be used from 2 to 3 times per week according to the surveys. The two activities that would be used more frequently are Take a Peek and Story Teller. Therapists suggested making the activity Story Teller more customizable so that they can write stories that are relevant to their patients' lives. Also, the Rewards system was found very helpful. The evaluators also said that being able to control the content of the activities Say my Name and Virtual Flashcards is key to maintaining children engaged.

7.3 FUTURE AREAS OF DEVELOPMENT

There are many areas of future development including eye tracking. Eye tracking could be integrated into the system to capture and record the eye gaze throughout the interaction between the child and Ying. This feature could be particularly useful to guide the child in activities like Eye Gaze and Tappy Time. This system could also beneﬁt from the addition of new games that target other skills. Additionally, the voice commands from this system could be improved by making them more natural sounding and less robotic. The Story Teller activity could also be improved by giving 72 therapists the capability to create their own stories. Similarly, more options could be added to the avatar component to ensure users do not feel intimidated by it.

73 APPENDICES

74 APPENDIX A EVALUATION QUESTIONNAIRE

A.1 LEARNING ACTIVITY 1: VIRTUAL FLASHCARDS

This activity teaches children vocabulary words, spelling, and pronunciation. Would this game attract the attention of children with ASD?

• Yes

• No

From your experience, do you believe children with ASD could beneﬁt from this game?

• Yes

• No

Would you use this game as part of a verbal communication therapy session for autistic children?

• Yes

• No

If so, how often would you use it?

• Once a day

• Once a week

• Twice a week

• Three times a week

• As often as I can 75 Is this game intuitive enough for ASD children?

• Yes

• No

Do you use ﬂashcards in a regular therapy session?

• Yes

• No

If so, please tell us what do you use and how often? I use:

• Very often

• Rarely

Any other suggestions?

A.1.1 Learning Activity 2: Story Teller

This module teaches children emotions through the usage of social stories. Would this game attract the attention of children with ASD?

• Yes

• No

From your experience, do you believe children with ASD could beneﬁt from this game?

• Yes

• No

Would you use this game as part of a social communication therapy session for autistic children?

76 • Yes

• No

If so, how often would you use it?

• Once a day

• Once a week

• Twice a week

• Three times a week

• As often as I can

Do you use social stories to enhance patients emotion recognition skills?

• Yes

• No

If so, please tell us what do you use and how often? I use:

• Very often

• Rarely

Is this game intuitive enough for ASD children?

• Yes

• No

Any suggestions?

77 A.1.2 Learning Activity 3: Eye Gaze

This activity engages childrens eye contact and promotes social interaction Would this game attract the attention of children with ASD?

• Yes

• No

From your experience, do you believe children with ASD could beneﬁt from this game?

• Yes

• No

Would you use this game as part of a social communication therapy session for autistic children?

• Yes

• No

If so, how often would you use it?

• Once a day

• Once a week

• Twice a week

• Three times a week

• As often as I can

Do you use similar exercises to engage children in eye contact and introduce them to social cues?

• Yes, please specify which: 78 • No

Is this game intuitive enough for ASD children?

• Yes

• No

Any other suggestions?

A.1.3 Game 1: Say my Name

This game uses voice recognition technology to determine if the child says the correct word. Would this game attract the attention of children with ASD?

• Yes

• No

From your experience, do you believe children with ASD could beneﬁt from this game?

• Yes

• No

Would you use this game as part of a verbal communication therapy session for autistic children?

• Yes

• No

If so, how often would you use it?

• Once a day

• Once a week 79 • Twice a week

• Three times a week

• As often as I can

Do you use this exercise or similar exercises to help your autistic patients learn vocabulary?

• Yes. Please specify which:

• No

Is this game intuitive enough for ASD children?

• Yes

• No

Any other suggestions?

A.1.4 Game 3: Message in a Bottle

This game is very similar to Say my Name. The only diﬀerence is that it focuses on sentence structure and construction. Would this game attract the attention of children with ASD?

• Yes

• No

From your experience, do you believe children with ASD could beneﬁt from this game?

• Yes

• No

80 Would you use this game as part of a verbal communication therapy session for autistic children?

• Yes

• No

If so, how often would you use it?

• Once a day

• Once a week

• Twice a week

• Three times a week

• As often as I can

Do you use this exercise or similar exercises to help your autistic patients learn sentence structure and vocabulary?

• Yes. Please specify which:

• No

If so, how often?

• Once a day

• Once a week

• Twice a week

• Three times a week

• As often as I can

81 Is this game intuitive enough for ASD children?

• Yes

• No

Any suggestions?

A.1.5 Game 3: Take a Peek

This module targets verbal communication skills. The user has a couple of seconds to say the correct word before the image is revealed. Would this game attract the attention of children with ASD?

• Yes

• No

From your experience, do you believe children with ASD could beneﬁt from this game?

• Yes

• No

Would you use this game as part of a verbal communication therapy session for autistic children?

• Yes

• No

If so, how often would you use it?

• Once a day

• Once a week

• Twice a week 82 • Three times a week

• As often as I can

Since this game involves recognition and vocabulary skills, would it be overwhelming for ASD patients?

• A little bit

• A lot

• Not much

• Not at all

Do you use this exercise or similar exercises to assess autistic patients in the area of vocabulary?

• Yes

• No

If so, how often?

• Once a day

• Once a week

• Twice a week

• Three times a week

• As often as I can

Is this game intuitive enough for ASD children?

• Yes

• No

Any other suggestions? 83 A.1.6 Game 4: Tappy Time

This module targets eye engagement. Would this game attract the attention of children with ASD?

• Yes

• No

From your experience, do you believe children with ASD could beneﬁt from this game?

• Yes

• No

Would you use this game as part of a social communication therapy session for autistic children?

• Yes

• No

If so, how often would you use it?

• Once a day

• Once a week

• Twice a week

• Three times a week

• As often as I can

Do you use this exercise or similar exercises to assess eye contact and social cues recognition skills?

• Yes 84 • No

If so, how often?

• Once a day

• Once a week

• Twice a week

• Three times a week

• As often as I can

Is this game intuitive enough for ASD children?

• Yes

• No

Any other suggestions?

A.1.7 Game 5: Fill in the Blank

Would this game attract the attention of children with ASD?

• Yes

• No

From your experience, do you believe children with ASD could beneﬁt from this game?

• Yes

• No

Would you use this game as part of a verbal communication therapy session for autistic children? 85 • Yes

• No

If so, how often would you use it?

• Once a day

• Once a week

• Twice a week

• Three times a week

• As often as I can

Since this game involves recognition and vocabulary skills, would it be overwhelming for ASD patients?

• Yes

• No

Do you use this exercise or similar exercises to help your autistic patients learn vocabulary?

• Yes. Please specify which:

• No

If so, how often?

• Once a day

• Once a week

• Twice a week

86 • Three times a week

• As often as I can

Is this game intuitive enough for ASD children?

• Yes

• No

Any other suggestions?

A.1.8 Schedule

This module allows therapists to assign a series of activities from the learn and play modules to a child. From your experience, do you believe children with ASD could beneﬁt from using a schedule to get to their activities?

• Yes

• No

Would you use this module as part of a therapy session for autistic children?

• Yes

• No

If so, how often would you use it?

• Once a day

• Once a week

• Twice a week

• Three times a week 87 • As often as I can

Do you use schedulers with your autistic patients?

• Yes

• No

If so, how often?

• Once a day

• Once a week

• Twice a week

• Three times a week

• As often as I can

Is this module intuitive enough for ASD children?

• Yes

• No

Any other suggestions?

A.1.9 Rewards

This module allows therapists to reward children for activity completion and/or performance. From your experience, do you believe a rewards module would make the experience more enjoyable for ASD patients?

• Yes

• No 88 Would you use this module as part of a therapy session for autistic children?

• Yes

• No

If so, how often would you use it?

• Once a day

• Once a week

• Twice a week

• Three times a week

• As often as I can

Do you use rewards as part of your therapy?

• Yes

• No

Is this module intuitive enough for ASD children?

• Yes

• No

Any other suggestions?

A.1.10 Progress

Would it be useful to you to see how many points a child earns every time they play the previous mentioned games?

• Yes 89 • No

Would it be useful to you to see and compare how two patients with the same diagnosis and same age group perform?

• Yes

• No

Any suggestions?

A.1.11 Overall Experience

Please let us know any comments or suggestions! Thank you.

90 BIBLIOGRAPHY

[1] Center for Disease Control and Prevention. “Autism Spectrum Disorder”. http://sfari.org/news-and-opinion/in-brief/2013/ cognition-and-behavior-children-with-autism-avoid-eyes. Cited 18 January 2014.

[2] Autism Society. “Top Ten Autism Research Advances of 2012”. http://www.autismspeaks.org/science/science-news/ top-ten-autism-research-advances-2012. Cited 18 January 2014.

[3] US News. “Smart Technology May Help Kids With Autism Learn, Commu- nicate” http://health.usnews.com/health-news/news/articles/2013/08/ 02/smart-technology-may-help-kids-with-autism-learn-communicate. Cited 21 January 2014.

[4] National Institute of Mental Health. “Autism Spec- trum Disorder”. http://www.nimh.nih.gov/health/topics/ autism-spectrum-disorders-pervasive-developmental-disorders/index. shtml?utm_source=rss_readers&utm_medium=rss&utm_campaign=rss_full. Cited 24 January 2014.

[5] HelpGuide. “Autism Spectrum Disorders”. http://www.helpguide.org/ mental/autism_spectrum.htm. Cited 24 January 2014.

[6] Mayo Clinic. “Autism Spectrum Disorder”. http://www.mayoclinic.org/ diseases-conditions/autism-spectrum-disorder/basics/symptoms/ con-20021148?_ga=1.197220847.1753449304.1415057408. Cited 24 January 2014.

[7] Genes to Cognition Online. “Brain and Autism” http://www.g2conline.org/ 840. Cited 24 January 2014.

[8] Snapﬁles. “Zac Browser - Zone for Autistic Children”. http://www.snapfiles. com/get/zacbrowser.html. Cited 2 February 2014.

[9] Zac Browser. http://zacbrowser.com/. Cited 2 February 2014.

[10] YouTube. “Zac Browser — Free Internet Browser for Children in the Autism Spectrum”. https://www.youtube.com/watch?v=9jSZ_rx5Bgw. Cited 2 Febru- ary 2014.

91 [11] YouTube. “Researching autistic / Aspergers children’s social skills”. https:// www.youtube.com/watch?v=Ku2dIif7hWA. Cited 4 February 2014. [12] C. Whalen, L. Liden, B. Ingersoll, E. Dallaire, S. Liden. Behavioral Improvements Associated with Computer-Assisted Instruction for Children with Developmental Disabilities. The Journal of Speech and Language Pathology - Applied Behavior Analysis n (2006) 11-26. Cited 4 February 2014.

[13] TGEN. “Autism FAQ”. https://www.tgen.org/media/38119/AutismFAQ. pdf. Cited 4 February 2014. [14] Boston Childrens’ Hospital. “Breaking into the Autistic Brain”. http://www.childrenshospital.org/news-and-events/ research-and-innovation-features/breaking-into-the-autistic-brain. Cited 4 February 2014.

[15] Autism Speaks. “Brain Study Finds Evidence that Autism Involves Too Many Synapses”. http://www.autismspeaks.org/science/science-news/ brain-study-finds-evidence-autism-involves-too-many-synapses. Cited 5 February 2014.

[16] Autism Science Foundation Channel. “Dr. Eric Courchesne explains the underlying brain biology of autism”. https://www.youtube.com/watch?v= YYY2XGEmc1s. Cited 5 February 2014.

[17] Children’s Hospital Of Wisconsin. “Autistic Disorder”. http://www.chw. org/medical-care/child-development-center/developmental-disorders/ pervasive-developmental-disorders-pdd/autistic-disorder/ Cited 8 February 2014.

[18] Love to Know Autism. “Diﬀerent Levels of Autism”. http://autism. lovetoknow.com/Different_Levels_of_Autism. Cited 8 February 2014.

[19] Autism of West Shore. “Deﬁning Autism”. https://www.asws.org/ WhatisAutism.aspx. Cited 8 February 2014.

[20] Mayo Clinic. “Childhood Disintegrative Disorder”. http://www.mayoclinic. org/diseases-conditions/childhood-disintegrative-disorder/basics/ causes/con-20026858. Cited 10 February 2014.

[21] Web MD. “Rett Syndrome Symptoms”. http://www.webmd.com/brain/ autism/rett-syndrome. Cited 15 February 2014. [22] National Institute of Neuorological Disorders and Stroke. “Rett Syndrome Fact Sheet”. http://www.ninds.nih.gov/disorders/rett/detail_rett.htm. Cited 15 February 2014.

[23] Autism Speaks. “PDD-NOS”. http://www.autismspeaks.org/what-autism/ pdd-nos. Cited 15 February 2014. 92 [24] Autism Speaks. “How Early can autism be diagnosed”. http://www.autismspeaks.org/blog/2013/03/15/ how-early-can-autism-be-diagnosed. Cited 15 February 2014.

[25] Science Daily “Children with autism beneﬁt from early, intensive therapy”. http://www.sciencedaily.com/releases/2011/09/110928125418.htm. Cited 15 February 2014.

[26] Healio. “Sensory processing may help with autism diagnosis, classiﬁca- tion”. http://www.healio.com/psychiatry/autism-spectrum-disorders/ news/online/%7Be500dfe5-662a-47da-ab5b-2b796f919bc9%7D/ sensory-processing-may-help-with-autism-diagnosis-classification Cited 1 March 2014.

[27] Autism Speaks. “How is Autism Diagnosed”. http://www.autismspeaks.org/ what-autism/diagnosis. Cited 10 March 2014.

[28] Autism Speaks. “Floortime” http://www.autismspeaks.org/what-autism/ treatment/floortime. Cited 1 April 2014.

[29] J. L. Petersen. Computer Interaction System To Identify Learning Patterns and Improve Performance in Children with Autism Spectrum Disorder Florida At- lantic University n (2011) Cited 3 November 2014.

[30] The Transporters. ”Academic Research Behind The Transporters”. http://www. thetransporters.com/research.html. Cited 10 April 2014.

[31] Simons Foundation Autism Research Initiative. ”Cogni- tion and behavior: Sensory sensitivity tied to autism”. http://sfari.org/news-and-opinion/in-brief/2012/ cognition-and-behavior-sensory-sensitivity-tied-to-autism. Cited 10 April 2014.

[32] Autism Speaks. ”What is Autism?” http://www.autismspeaks.org/ what-autism Cited 10 April 2014.

[33] Super Duper Publications. ”Can Pictures Help Children With Autism?”. http: //www.superduperinc.com/handouts/pdf/189_PicturesHelpAutism.pdf. Cited 10 April 2014.

[34] Origin Instruments. ”Proloquo2Go and Proloquo4Text Combo Packs” http:// www.orin.com/access/Proloquo2Go/. Cited 10 April 2014.

[35] Proactive Speech Therapy. ”VAST Autism 1 Core”. http://www. proactivespeechtherapy.com/vast-core.html. Cited 12 April 2014.

[36] Indiana Resource Center for Autism. ”Using Visual Schedules: A Guide for Parents”. http://www.iidc.indiana.edu/?pageId=394. Cited 13 April 2014.

93 [37] Laptop. ”13 Best Autism Apps for the iPad”. http://blog.laptopmag.com/ best-autism-apps-ipad?slide=3. Cited 13 April 2014.

[38] Autism Speaks. ”Verbal Behavior Therapy”. http://www.autismspeaks.org/ what-autism/treatment/verbal-behavior-therapy. Cited 15 April 2014.

[39] Autism Speaks. ”Pivotal Response Therapy”. http://www.autismspeaks. org/what-autism/treatment/pivotal-response-therapy-prt. Cited 16 May 2014.

[40] Autism Speaks. ”Applied Behavior Analysis”. http://www.autismspeaks.org/ what-autism/treatment/applied-behavior-analysis-aba. Cited 20 Septem- ber 2014.

[41] Come Unity. ”Communication in Autism”. http://www.comeunity.com/ disability/speech/autism.html. Cited 22 October 2014.

[42] Evenesis. “Why Eye Contact is Important Dur- ing Conversation”. http://evenesis.com/blog/ why-eye-contact-is-important-during-conversation/. Cited 8 August 2014.

[43] The Hanen Centre. “How Autism Aﬀects Communication in Young Children” http://www.hanen.org/Helpful-Info/Autism-Corner/ How-Autism-Affects-Communication.aspx. Cited 22 October 2014.

[44] Love To Know. “Teaching Feelings to Autistic Chil- dren”. http://autism.lovetoknow.com/autism-resources/ teaching-feelings-autistic-children. Cited 12 October 2014.

[45] Medicine Net. “Autism Spectrum Disorder (In Children and Adults)”. http: //www.medicinenet.com/autism_and_communication/article.htm. Cited 22 October 2014.

[46] Autism Speaks. “Brain Study Finds Evidence that Autism Involves Too Many Synapses”. http://www.autismspeaks.org/science/science-news/ brain-study-finds-evidence-autism-involves-too-many-synapses. Cited 22 October 2014.

[47] Health Children. “Autism Spectrum Disorder”. http://www.healthychildren. org/English/health-issues/conditions/developmental-disabilities/ Pages/Autism-Spectrum-Disorders.aspx. Cited 22 October 2014.

[48] National Autism Association. “Causes of Autism”. http:// nationalautismassociation.org/about-autism/causes-of-autism/. Cited 22 October 2014.

94 [49] Autism Science Foundation. “Treatment Options”. http://www. autismsciencefoundation.org/what-is-autism/autism-diagnosis/ treatment-options. Cited 10 August 2014.

[50] Raising Children. “Changing routines: children and teenagers with autism spectrum disorder”. http://raisingchildren.net.au/articles/ autism_spectrum_disorder_changing_routines.html. Cited 12 August 2014.

[51] Raising Children. “Obsessive behaviour, routines and rituals: children with autism spectrum disorder”. http://raisingchildren.net.au/articles/ autism_spectrum_disorder_obsessions_routines.html. Cited 12 August 2014.

[52] Speak in Motion. “What is VAST?”. http://www.speakinmotion.com/. Cited 12 August 2014.

[53] Special Needs Apps. “Training Faces”. http://www.friendshipcircle.org/ apps/browse/training-faces-2/. Cited 12 August 2014.

[54] Microsoft. “Microsoft by the Numbers”. http://news.microsoft.com/ bythenumbers/index.html. Cited 22 October 2014.

[55] Autism Speaks. “Treatment” http://www.autismspeaks.org/what-autism/ treatment. Cited 22 October 2014.

[56] Code Project. “Understanding Contracts in WCF” http://www.codeproject. com/Articles/664238/Understanding-Contracts-in-WCF. Cited 22 October 2014.

[57] T. Grandin, R.Panek. The Autistic Brain Thinking Across the Spectrumn (2006) 11-26. Cited 22 October 2014.

[58] Tutorials Point. “WCF Architecture” http://www.tutorialspoint.com/wcf/ wcf_architecture.htm. Cited 22 October 2014.

[59] Wikipedia. “Multitier Architecture” http://en.wikipedia.org/wiki/ Multitier_architecture. Cited 24 October 2014.

[60] Bubbles Make Him Smile. “Five Types of Autism” http://www. bubblesmakehimsmile.com/2010/10/five-types-of-autism.html. Cited 24 October 2014.

[61] Second Start. “About Aspergers” http://www.secondstart.org/about_ aspergers.aspx. Cited 22 October 2014.

[62] Info My Health. “Cause of Autism Still Mysterious” http://infomyhealth. com/health-news/scientists-found-the-gene-chd8-as-forming-autism. html/attachment/cause-of-autism-is-still-mysterious. Cited 22 October 2014. 95 [63] AssistiveWare. “Proloquo2go”. http://www.assistiveware.com/product/ proloquo2go. Cited 22 October 2014.

[64] iTunes. “Proloquo2go”. https://itunes.apple.com/us/app/ proloquo2go-symbol-based-aac/id308368164?mt=8. Cited 22 October 2014.

[65] The Recapp. “Apps for Autism”. http://www.therecapp.com/app_living/ scenario/apps_for_autism/. Cited 22 October 2014.

[66] Indiana Center for Autism. “Using Visual Schedules: A Guide for Parents”. http://www.iidc.indiana.edu/?pageId=394. Cited 26 October 2014.

[67] iTunes. “FTVS HD - First Then Visual Schedule HD”. https://itunes.apple. com/us/app/ftvs-hd-first-then-visual/id624035410?mt=8. Cited 26 Octo- ber 2014.

[68] VentureBeat. “AutisMate: an iPad app for personalized autism therapy”. http: //venturebeat.com/2013/01/29/autismate-ipad-app-therapy/. Cited 26 October 2014.

[69] The Linguist List. “Ask a Linguist FAQ” http://linguistlist.org/ ask-ling/lang-acq.cfm. Cited 28 October 2014.

[70] NIH. “Communication Problems in Children with Autism Spec- trum Disorder”. http://www.nidcd.nih.gov/health/voice/pages/ communication-problems-in-children-with-autism-spectrum-disorder. aspx. Cited 28 October 2014.

[71] Health Children. “Communication Skills” http://www.healthofchildren. com/C/Communication-Skills.html#ixzz3DRNMD8BV. Cited 28 October 2014.

[72] Autism Speaks. “Visual Tools”. http://www.autismspeaks.org/ family-services/resource-library/visual-tools. Cited 29 October 2014.

[73] American Speech-Language-Hearing Association. “Autism Spectrum Disorder”. http://www.asha.org/public/speech/disorders/autism/. Cited 29 October 2014.

[74] Humintell. “The Seven Basic Emotions: Do you know them?”. http://www. humintell.com/2010/06/the-seven-basic-emotions-do-you-know-them/. Cited 29 October 2014.

[75] The British Psychological Society. “Children‘s scripts for social emotions: Causes and consequences are more central than are facial expressions”. https://www2. bc.edu/james-russell/publications/Widen&Russell2010-c&c11.pdf. Cited 29 October 2014.

96 [76] Simon Foundation Autism Research Initiative. “Cogni- tion and behavior: Children with autism avoid eyes” http://sfari.org/news-and-opinion/in-brief/2013/ cognition-and-behavior-children-with-autism-avoid-eyes. Cited 29 October 2014.

[77] Wikipedia. “Speech Synthesis” http://en.wikipedia.org/wiki/Speech_ synthesis. Cited 3 November 2014.

[78] Educate Autism “Social Stories”. http://www.educateautism.com/ social-stories.html. Cited 5 November 2014.