Making the Invisible Visible: Visualization of Music and Lyrics for Deaf and Hard of Hearing Audiences by David W. Fourney Master of Science, University of Saskatchewan, 2008 Post-Graduate Diploma, University of Saskatchewan, 2004 Bachelor of Science, University of Saskatchewan, 2001 Bachelor of Arts Honours, University of Saskatchewan, 1993 A dissertation presented to Ryerson University in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Program of Mechanical and Industrial Engineering Toronto, Ontario, Canada, 2015 © David W. Fourney 2015 AUTHOR’S DECLARATION FOR ELECTRONIC SUBMISSION OF A DISSERTATION I hereby declare that I am the sole author of this dissertation. This is a true copy of the dissertation, including any required final revisions, as accepted by my examiners. I authorize Ryerson University to lend this dissertation to other institutions or individuals for the purpose of scholarly research. I further authorize Ryerson University to reproduce this dissertation by photocopying or by other means, in total or in part, at the request of other institutions or individuals for the purpose of scholarly research. I understand that my dissertation may be made electronically available to the public. iii Making the Invisible Visible: Visualization of Music and Lyrics for Deaf and Hard of Hearing Audiences Doctor of Philosophy 2015 David W. Fourney Mechanical and Industrial Engineering Ryerson University Abstract Music is a major art form present in all cultures. It is nearly ubiquitous in Western culture. As a shared social experience, music is a shared knowledge space. Some music is so well recognized that people may refer to it by name (e.g., Beethoven’s 5th symphony, the Star Wars theme). Signing Deaf communities have indigenous forms of lyrical music. While these may not be melodic in the Western expectation, they still conform to the idea of music as metrical lyrics accompanied by instrumentation. Music is a system made up of several elements and attributes containing discrete information chunks that can be perceived by a human listener. Music communicates complex emotional signals and, in some cultures, has specific conventions designed to manipulate emotions. However, deaf, deafened, and hard of hearing people are unable to fully access the richness of the aural musical experience. This dissertation explores ways to make information that is presented in an auditory modality ac- cessible to those who cannot fully perceive it due to hearing loss or environmental barriers; making the invisible visible. A multi-disciplinary, multi-method project, this research examines the design of visualizations intended to present different kinds of otherwise invisible music information in a visible, entertaining form. Ninety-four people, 66 women and 28 men, who identified themselves as Hard of Hearing (46), Deaf (38), Cochlear Implantees (8), or Deafened (2), participated in one of four experiments. Eight of the 94 participated in a fifth experiment. Research questions included determining the fundamen- v tal properties of music that visualizations must present, determining what emotional responses users might have to visualizations of music information, and determining whether a music experience similar to that of hearing audiences can be experienced via visualizations of music information. Among several findings, this project learned that many deaf, deafened and hard of hearing people are active consumers of music; lyrics represent a key piece of music information hard of hearing people require of any visualization; different visualizations might be better suited to different musical genres or emotional moods; and the set of music visualizations tested convey some types of emotional infor- mation but do not obviously communicate the emotional component of the music. vi Acknowledgements This Dissertation was a challenging project. It required quickly gaining a deep understanding of multi- ple subjects, such as music theory, before I could tackle the research questions themselves. This work, would not have been possible without the support and caring of several people. Special thanks to my supervisor, Dr. Filippo Salustri, for providing me with an excellent atmo- sphere for doing research. I have been fortunate to have his invaluable guidance, caring, and patience. Dr. Salustri willingly assumed responsibility for the heavy lifting and took the weight off of my shoul- ders during the low-points of my studies. Without Fil’s support, which continued throughout his sab- batical, this work would never have been completed. Thank you to the other members of my Examination Committee: Dr. W. Patrick Neumann, Dr. M. Wahab M. Ismail, and Dr. Kathryn Woodcock of Ryerson University and Dr. Richard Ladner of the University of Washington for agreeing to examine this Dissertation. I am indebted to Dr. Benjamin Bahan of Gallaudet University for a discussion that helped me to shape some of my design ideas. I owe my gratitude to Joe Monkhouse, my Research Assistant, for his time, energy, and attention to detail in transcribing interviews and reviewing focus group data. He also assisted with coding and statistical analyses. This work would have never gotten finished without his help. I would like to acknowledge and thank Dr. Deborah Fels and the students of the Inclusive Media and Design Centre (previously Centre for Learning Technologies), especially Mike P., Jorge, Quoc, and Hong Tai, for making available many resources. Special thanks also to Dr. Frank Russo and the stu- dents of his Science of Music, Auditory Research and Technology (SMART) lab, specifically Arla, for their thoughts on my work. Thanks also to the Canadian Hard of Hearing Association (CHHA) for making available to me some space at their conference venue to conduct interviews. Many thanks to my research participants. Without them there would have been no way to do my project. This research benefitted from funding provided by the Toronto Rehabilitation Institute, Standards Council of Canada (SCC), Natural Sciences and Engineering Research Council (NSERC), and Social Sciences and Humanities Research Council (SSHRC). In particular, without the support of a (partial) Toronto Rehab Scholarship in Rehabilitation-related Research for Students with Disabilities, I would not have been able to move to Toronto to begin my studies leading to this Dissertation. I thank my family for their support and my wife for her love and patience. Colleen’s dedication and support helped me to focus on my work in the good times as well as the most stressful times. Finally, I am forever grateful for the substantial financial support of my late father, the Rev. Dr. Lloyd Fourney, without which the last two years of my studies would have been impossible. vii Dedication in memory of my dad ix Contents Declaration ............................................... iii Abstract .................................................v Acknowledgements ........................................... vii Dedication ................................................ ix List of Tables .............................................. xvii List of Figures ............................................. xxi 1 Introduction 1 1.1 Universal Usability........................................3 1.1.1 Usability and Accessibility................................4 1.2 Research Objectives.......................................8 1.3 Dissertation Organization....................................9 2 Background 11 2.1 The Audience........................................... 11 2.1.1 Disability......................................... 13 2.2 The Nature of Music....................................... 16 2.3 The Nature of Emotion..................................... 23 2.4 Music and Emotion........................................ 23 2.4.1 Emotion Conventions in Music............................. 24 2.4.2 Emotion Conventions in Music in Film......................... 25 2.5 Sensory Modalities........................................ 29 2.5.1 Auditory......................................... 29 2.5.2 Tactile........................................... 29 2.5.3 Olfactory......................................... 31 2.5.4 Visual........................................... 32 2.5.5 Sensory Substitution or Sensory Supplementation................... 33 2.6 Theories of Information Visualization & Sonification..................... 33 2.7 Prior Work in Music Visualization............................... 36 2.7.1 Relevant Existing Visualization Solutions....................... 37 2.7.2 Textual Visualizations.................................. 42 xi 2.7.3 Visualizations and Emotion............................... 48 2.7.4 Why Music Visualizations?............................... 49 2.8 Summary............................................. 49 2.9 Next Steps............................................ 50 3 Methodology 51 3.1 Design Research Methodology.................................. 52 3.1.1 DRM Stage 0: Criteria Definition / Research Clarification.............. 52 3.1.2 DRM Stage 1: Descriptive Study I........................... 52 3.1.3 DRM Stage 2: Prescriptive Study........................... 52 3.1.4 DRM Stage 3: Descriptive Study II.......................... 53 3.2 Grounded Theory......................................... 53 3.2.1 The Basics........................................ 53 3.2.2 Approach......................................... 55 3.2.3 Software Support..................................... 56 3.2.4 When Should Grounded Theory be Used?......................