Real-Time 3D Graphic Augmentation of Therapeutic Music Sessions for People on the Autism Spectrum
Total Page:16
File Type:pdf, Size:1020Kb
Real-time 3D Graphic Augmentation of Therapeutic Music Sessions for People on the Autism Spectrum John Joseph McGowan Submitted in partial fulfilment of the requirements of Edinburgh Napier University for the degree of Doctor of Philosophy October 2018 Declaration I, John McGowan, declare that the work contained within this thesis has not been submitted for any other degree or professional qualification. Furthermore, the thesis is the result of the student’s own independent work. Published material associated with the thesis is detailed within the section on Associate Publications. Signed: Date: 12th October 2019 J J McGowan Abstract i Abstract This thesis looks at the requirements analysis, design, development and evaluation of an application, CymaSense, as a means of improving the communicative behaviours of autistic participants through therapeutic music sessions, via the addition of a visual modality. Autism spectrum condition (ASC) is a lifelong neurodevelopmental disorder that can affect people in a number of ways, commonly through difficulties in communication. Interactive audio-visual feedback can be an effective way to enhance music therapy for people on the autism spectrum. A multi-sensory approach encourages musical engagement within clients, increasing levels of communication and social interaction beyond the sessions. Cymatics describes a resultant visualised geometry of vibration through a variety of mediums, typically through salt on a brass plate or via water. The research reported in this thesis focuses on how an interactive audio-visual application, based on Cymatics, might improve communication for people on the autism spectrum. A requirements analysis was conducted through interviews with four therapeutic music practitioners, aimed at identifying working practices with autistic clients. CymaSense was designed for autistic users in exploring effective audio-visual feedback, and to develop meaningful cross- modal mappings of musical practitioner-client communication. CymaSense mappings were tested by 17 high functioning autistic participants, and by 30 neurotypical participants. The application was then trialled as a multimodal intervention for eight participants with autism, over a 12-week series of therapeutic music sessions. The study captured the experiences of the users and identified behavioural changes as a result, including information on how CymaSense could be developed further. This dissertation contributes evidence that multimodal applications can be used within therapeutic music sessions as a tool to increase communicative behaviours for autistic participants. J J McGowan Acknowledgements ii Acknowledgements I would like to thank my supervisory team for their enduring and gracious support throughout this project. My Director of Studies Dr Grégory Leplâtre for his guidance and sensibilities, Dr Iain McGregor for his attention to detail and patience, and Dr Imed Romdhani for his advice and enthusiasm. Thanks to the team at Edinburgh Napier University and the Institute for Informatics and Digital Innovation (IIDI) who provided the opportunity and resources for this research, as well as to my fellow PhD students with whom I have shared this journey. Thanks to David McCluskey, Ewan Sinclair, Bal Cooke, Sensatronic Lab, and the staff at TouchBase Centre for their organisational capabilities, patience and enthusiasm for the project. Love and thanks to my family, in particular to my Dad Joe, who made so many things possible. Finally, all my love and thanks to my partner Jo for supporting me throughout my entire career, for the sacrifices she has made, and the subsequent changes we have made together. J J McGowan Associated Publications iii Associated Publications McGowan, J., Leplâtre, G., and McGregor, I. (2017). CymaSense: A Real-Time 3D Cymatics- Based Sound Visualisation Tool. DIS '17 Companion: Proceedings of the 2017 ACM Conference Companion Publication on Designing Interactive Systems, Edinburgh (DIS 2017). http://doi.org/10.1145/3064857.3079159 McGowan, J., Leplâtre, G., and McGregor, I. (2017). Exploring Musical Creativity through Interactive Cymatics. HCI '17: Proceedings of the 31st British Computer Society Human Computer Interaction Conference, Sunderland (BHCI 2017). http://dx.doi.org/10.14236/ewic/HCI2017.67 McGowan, J., Leplâtre, G., and McGregor, I. (2017). CymaSense: A Novel Audio-Visual Therapeutic Tool for People on the Autism Spectrum. ASSETS '17: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, Baltimore (ASSETS 17). https://doi.org/10.1145/3132525.3132539 J J McGowan Table of Contents iv Table of Contents Abstract ..................................................................................................................... i Acknowledgements .................................................................................................. ii Associated Publications ........................................................................................... iii Table of Contents .................................................................................................... iv Index of Figures ..................................................................................................... viii Index of Tables .................................................................................................... xviii 1 Introduction ............................................................................................................ 1 1.1 Background ............................................................................................................. 1 1.2 Research Aim ........................................................................................................... 4 1.3 Research Questions ................................................................................................. 4 1.4 Dissertation Structure ............................................................................................. 6 1.5 Summary ................................................................................................................. 8 2 Autism .................................................................................................................... 9 2.1 Challenges ............................................................................................................. 10 2.1.1 Communication ............................................................................................................. 11 2.1.2 Social Interaction ........................................................................................................... 13 2.1.3 Agency and Emotional Regulation ................................................................................. 14 2.1.4 Cognitive Processing and Sensory Awareness ............................................................... 15 2.1.5 Investment, Support and Technical Intervention .......................................................... 18 2.2 Current Interventions for Autism .......................................................................... 22 2.2.1 Music Therapy ............................................................................................................... 26 2.2.2 Assistive Technologies ................................................................................................... 35 2.3 Methods of Evaluating Interventions .................................................................... 54 2.3.1 Quantitative ................................................................................................................... 55 2.3.2 Qualitative ..................................................................................................................... 60 2.3.3 Single Case Studies ........................................................................................................ 61 2.4 Summary ............................................................................................................... 64 3 Visual Representation of Sound ............................................................................. 66 3.1 Cross-Modal Correspondences ............................................................................. 67 3.2 Theoretical and Technological Developments of Musical Sound ......................... 84 3.3 Temporal Graphical Representation of Musical Structure ................................... 86 3.3.1 Analysis, Learning and Teaching .................................................................................... 88 J J McGowan Table of Contents v 3.3.2 Performance .................................................................................................................. 96 3.3.3 Composition................................................................................................................. 100 3.4 Temporal Representation of Audio Signals ......................................................... 101 3.4.1 Analysis ........................................................................................................................ 102 3.4.2 Performance ................................................................................................................ 105 3.5 Interaction ........................................................................................................... 106 3.5.1 Applications ................................................................................................................. 108 3.6 Cymatics .............................................................................................................