Bell3d: an Audio-Based Astronomy Education System for Visually-Impaired Students Applications Research and and Research

Bell3D: An Audio-based Astronomy Education System for Visually-impaired Students Applications Research and and Research

Jamie Ferguson Keywords University of Aberdeen and University of Sonification, human-computer interaction, Glasgow, Scotland multimodal, visually impaired, software [email protected]

This article presents a software system that allows astronomical data to be represented and analysed purely through sound. The goal is to use this for communicating astronomy to visually impaired students, as well as providing a new modality of astronomy experience for those who are sighted. The programme — known as Bell3D and developed at the University of Aberdeen — allows various stars to be selected from a database and then maps a variety of their astronomical parameters to audio parameters so that they can be analysed, compared and understood, solely through the medium of sound.

Introduction and background is the “transformation of data relations to use of sonification as a method of pub- perceived relations in an acoustic signal” lic engagement with the Rosetta mis- The ability to experience the surrounding (Kramer et al., 2010). sion (European Space Agency, 2014); Universe is inherent and taken for granted NASA’s Earth+ system1 and the Harvard– by those who are sighted, but for those with Sonification, in many situations, is better Smithsonian Centre for Astrophysics’ Star a visual impairment, the night sky remains suited to analysing large datasets for pat- Songs: From X-rays to Music project2. almost entirely impenetrable. Utilising soni terns and anomalies than visual modalities Furthermore, during the press coverage fication — the use of non-speech audio to (Kramer et al., 2010). For example, a sig- of the LIGO experiment’s recent detection convey information (Kramer et al., 2010) nificantly large dataset such as a genomic of gravitational waves (Abbot et al., 2016), — allows those with a visual impairment sequence, which would traditionally have the audification of the detection was used to engage with data that would previously taken a lot of time and resources to ana- ubiquitously by the media and has become have only been accessible via visual modal- lyse for patterns and anomalies visually, synonymous with the detection3. These ities, and therefore available exclusively to may be sped up and compacted using technologies have only recently begun to sighted people. Sonification allows tradi- sound, allowing it be analysed in the order be implemented as astronomy commu- tionally visually-oriented and data-driven of minutes. nication tools, but computing and audio fields such as astronomy to be explored technologies are now reaching a level and experienced by visually-impaired peo- Both sonification and audification have, of low cost and high performance that ple, some of whom may never have had the until recently, only been applied sporadi- gives sonification the potential to be used opportunity to enjoy and experience these cally throughout the history of astrophys- throughout astronomy communication and fields before. Furthermore, this technol- ical and astronomical research. The first education for the visually impaired. ogy allows a new, complementary modal- use of these technologies in the litera- ity to visual experience for sighted people, ture was during the Voyager 2 mission in with potential applications across data the late 1970s (Scarf, 1982). These tech- How the system works science, science communication, art and nologies have more recently been used to education. analyse radio plasma waves around Saturn The Bell3D system4 was developed after (Gurnet, 2005), as well as examining large realising the potential of sonification A simple example of sonification would be collections of solar wind data (Wicks et and audio-based data displays within a kettle whistling when the water within it al., 2016). Furthermore, recent work into a astronomy communication and educa- reaches around a hundred degrees centi- sonification prototype for analysing satel- tion, not only as an essential resource grade. The data in this example being the lite and radio telescope data (Candey et for visually-impaired people, but also as temperature of the water and the informa- al., 2006; Diaz-Merced, 2013) has shown a new, engaging medium of astronomy tion extracted from analysing the whistling that researchers within the astronomy and experience for those who are sighted. The being that the water has reached its boiling astrophysics communities are hopeful of programme is in its infancy and requires point. Another classic example is a Geiger– the potential of applications of sonification more work, but has so far proven to be an Müller radiation detector, which creates at the leading edge of these sciences. important step towards ubiquitous access an audible click when particles associ- to astronomy experience and education. ated with radioactivity strike the detec- The astronomy and astrophysics commu- tor. However, sonification is not to be con- nication communities have experienced a The software involves the user selecting a fused with audification. Audification is the surge in the use of sonification and audi- constellation and then specific stars within direct playback of data samples (Kramer, fication in the last decade. Recent works that constellation to be sonified — mean- 1993), as opposed to sonification which including the European Space Agency’s ing data relating to the selected stars is

CAPjournal, No. 20, August 2016 35 Bell3D: An Audio-based Astronomy Education System for Visually-impaired Students

transposed into audio signals. When stars are selected, their astronomical parameters such as magnitude, location, size 1.0 and distance from Earth obtained from the SIMBAD database5 (Wenger, 2000) 0.5 are mapped onto audio parameters such as volume and pitch. The audio is spatial- ) –2 1 ised, meaning that the user listening to the 0 0.0 sounds perceives that these sounds are in three-dimensional space, with audio –0.5 coming from particular areas around their Strain (1 head; the star’s equatorial coordinates are –1.0 used for this. This provides an engaging experience and gives the user an idea of where the star(s) reside in relation to Earth.

The sounds produced are simple and easy to analyse. For each star there are 1.0 two sounds: a repeating “ping” sound and a continuous tone. Both can be turned on 0.5 and off independently. The former encodes )

three data points within one sound — a –2 1 star’s distance from Earth, its location with 0 0.0 reference to the Earth and its apparent magnitude — and the latter encodes one –0.5 attribute — a star’s colour index. Strain (1 –1.0 The ping represents distance, location and apparent magnitude data values in the following manner: The star’s distance from Earth is represented sonically by how loud the ping is, with the volume of this sound being inversely proportional to 1.0 the star’s distance from Earth (the closer the star, the louder the sound); the star’s 0.5 location (based on its equatorial coordi- )

nates) is represented by the sound’s per- –2 1

0 0.0 ceived location in the three-dimensional sound space; and, finally, the star’s apparent magnitude or brightness is represented –0.5 by the pitch of the ping sound, the pitch Strain (1 being proportional to the star’s apparent –1.0 magnitude.

The second sound — the continuous tone which represents a star’s colour index — 0.30 0.35 0.40 0.45 is represented sonically by the pitch of the Time (sec) continuous tone. The higher the pitch of the tone, the further towards the red end of the Figure 1. The LIGO gravitational wave detection waveform, which was audified as a successful means of com- spectrum a star is, the lower the pitch, the municating the discovery in the media. Credit: LIGO Scientific Collaboration further toward blue end.

This rather simple mapping of parameters Additional applications of the and audification have been employed allows the user to make basic assumptions system successfully within artistic endeavours about stars based on what they hear. For attempting to explore ideas combining sci- example, one can assume that Betelgeuse The obvious, prevalent use for the system ence and art on many occasions (Mast et is a red star, based on the fact that the pitch lies within astronomy education and com- al., 2015; Polli, 2005) and have been the of its tone sound is high, or that Proxima munication, but uses within the arts have basis of many sound artist’s practice, such Centauri is closer to Earth than Sirius, been discussed as potential further appli- as Robert Alexander6, Mélodie Fenez7, because its “ping” sound is louder. cations of the Bell3D software. Sonification Ryoji Ikeda8 and Rory Viner9. Bell3D has

36 CAPjournal, No. 20, August 2016 CAPjournal, No. 20, August 2016

the potential to be used in such a way in Astronomical Union’s Office of Astronomy severe visual impairment can use the sys- future either as a composition or perfor- for Development (OAD) in Cape Town, tem without a screen via the use of haptic mance tool for artists. South Africa. During this project the soft- technologies and speech recognition. This ware will benefit from input by researchers, further research will focus much more thor- The system was used as an astronomy- educators and outreach specialists at the oughly on investigating user perception teaching resource by Dr Wanda Diaz- South African Astronomical Observatory, and response to sonification within astron- Merced of the Harvard–Smithsonian as well as being used in OAD-led astronomy in both visually-impaired and sighted Centre for Astrophysics during a recent omy lessons with students at the Athlone groups. The prototype described here series of lectures and workshops at South School for the Blind. This will allow for pre- is a useful preliminary investigation into Africa’s national science festival, SciFest liminary evaluation of visually impaired the software technology needed to facil- Africa10. Dr Diaz-Merced used sounds users’ responses to and perceptions of the itate such sonification interaction, how- from the system to explain the basic ideas system’s sonifications. ever it lacks thorough empirical analysis of sonification and how astronomical data on response and perception. can be portrayed via sound. Work on the Bell3D system will be the basis of a PhD within the Multimodal Furthermore, as the University of Glasgow Interaction Group at the University of played a significant role in both the recent Future work Glasgow, Scotland, beginning in late 2016. LIGO discovery of gravitational waves Plans for additions to the system include: (Abbot et al., 2016) and the communication Development of the Bell3D system and, head-tracking to provide a more engag- of it in the media, a collaboration between potentially, further astronomy-related son- ing experience for the user; functionality the multimodal group and the astronomers ification tools will continue during a pro- for planets, galaxies and other astronomi- and astrophysicists involved at Glasgow ject that will commence in August 2016, cal bodies to be sonified; and a completely will be discussed in the coming years. The run in collaboration with the International “visual-free” interface, so that those with a media presence surrounding the LIGO

0 / nT X B –5

–10 10

0 / nT Y B –5

–10 10

0 / nT Z B –5

–10 10

0 / nT Ma g

B –5

–10 UTC 10:00:00 10:10:00 10:20:00 10:30:00 10:40:00 10:50:00 11:00:00 Figure 2. The magnetic field parameters of comet 67P/Churyumov–Gerasimenko, measured by ESA’s Rosetta spacecraft, which were sonified and the resulting audio used as the base for the highly successful blog post titled The Singing Comet13. Credit: Karl-Heinz Glaßmeier, Technische Universität Braunschweig, Germany

Bell3D: An Audio-based Astronomy Education System for Visually-impaired Students Bell3D: An Audio-based Astronomy Education System for Visually-impaired Students

Figure 3. Bell3D prototype running on OSX. Credit: Jamie Ferguson, University of Aberdeen

discovery was a major win for sonification Kilometre Array12, and the ever-increas- 2 The Harvard–Smithsonian Centre for Astro- and audification as a science communi- ing ubiquity and accessibility of comput- physics’ Star Songs: From X-rays to Music cation tool, therefore taking advantage ing devices there have never been more project: https://www.cfa.harvard.edu/sed/ of having both teams being based at the opportunities to communicate astronomy projects/star_songs/ same university will allow us to capitalise to as many people as possible, regardless 3 Audifications of LIGO gravitational wave on this success. of disability. detection: https://losc.ligo.org/events/ GW150914/ Discussions have also taken place with As our telescopes, probes and satellites 4 Further details and sounds from Bell3D: the Glasgow Science Centre Planetarium continue to provide an ever-increasing vol- http://jfergusoncompsci.co.uk/research — one of the UK’s foremost establishments ume of data to be analysed, research into 5 SIMBAD database: http://simbad.u-strasbg.fr/ for science communication — to potentially utilising more senses than the eye for data simbad/ include sounds from the Bell3D system in analysis is becoming increasingly vital. A 6 Article by Michael Byrne, 2013 on Robert exhibitions and presentations within the more multimodal approach to data anal- Alexander Spaced Out: ‘The Space Com- planetarium and science centre. ysis will not only allow vast amounts of poser’ is Making Music With the Sun: astronomical data to be analysed more http://motherboard.vice.com/read/spaced- efficiently, it will also begin to open the out-making-music-with-the-sun Conclusion discoveries made, and the opportunity to 7 Sound artist Mélodie Fenez: make them, to people who mere decades http://www.a-melodie.com/ This system provides the opportunity for ago would have had little opportunity to be 8 Sound artist Ryoji Ikeda: people of all ages with a visual impair- involved with astronomy. http://www.ryojiikeda.com/ ment to access, experience and learn 9 Sound artist Rory Viner: about basic astronomy and the night sky; http://rory-viner.tumblr.com/ an opportunity that should be accessible Notes 10 Video of SciFest lecture: https://www.you- to everyone. With a wealth of new astro- tube.com/watch?v=QilOFTvCTzA nomical instruments becoming operational 1 NASA’s Earth+ system: http://prime.jsc. 11 James Webb Space Telescope: in the coming years, such as the James nasa.gov/earthplus/ http://www.jwst.nasa.gov/ Webb Space Telescope11 and the Square

38 CAPjournal, No. 20, August 2016 CAPjournal, No. 20, August 2016

12 Square Kilometre Array: Diaz-Merced, W. L. 2013, Sound for the Wenger, M. et al. 2000, Astronomy and Astro- https://www.skatelescope.org/ exploration of space physics data, PhD physics Supplement Series, 143, 1, 9 13 European Space Agency, The Singing Thesis, University of Glasgow available Wicks, R. T. et al. 2016, The Astrophysical Comet, 2014, available http://blogs.esa.int/ http://theses.gla.ac.uk/5804/1/2014Diaz- Journal, 819, 1, 6 rosetta/2014/11/11/the-singing-comet/, MercedPHD.pdf, (accessed 15 June 2016) (accessed 9 January 2016). Gurnett, D. A. et al. 2004, The Cassini- Huygens Mission, (Netherlands: Springer) Kramer, G. 1993, Auditory display: Sonification, audification, and auditory interfaces, References (London: Perseus Publishing) Biography Kramer, G. et al. 2010, Faculty Publications, Abbott, B. P. et al. 2016, Physical Review Department of Psychology, University of Letters, 116, 6, arXiv:1602.03837 Jamie Ferguson completed his undergrad- Nebraska, Paper 444, http://digitalcom- uate studies in computing science at the Uni- Candey, R. M., Schertenleib, A. M. & Diaz- mons.unl.edu/psychfacpub/444 versity of Aberdeen earlier in 2016. He is begin- Merced, W. L. 2006, Proceedings of the 12th Mast, S. et al. 2015, Leonardo, 49, 1, 19 ning a PhD at the University of Glasgow later International Conference on Auditory Dis- this year on multimodal computer applications play (ICAD), London, UK Polli, A. 2005, Leonardo, 38, 1, 31 within astronomy and astrophysics. Scarf, F. L. et al. 1982, Science, 215, 4532, 587

4th ESO Astronomy Camp For Secondary School Students Credit: ESO/V. Vicenzi Explore the theme of The Visible 26.12.2016 – 01.01.2017 and Hidden Universe through sev- Saint-Barthélemy, Nus, Aosta Valley, Italy eral astronomical sessions, including lectures, hands-on activities, and night-time observations with the telescopes and instruments at the observatory. Enjoy a mem- orable camp with an international group engaged in social activities, winter sports and excursions.

#AstroCamp http://www.sterrenlab.com/camps/eso-astronomy-camp-2016