Journal Title XX(X):1–12 Three Orthogonal Dimensions for c The Author(s) 2016 Reprints and permission: Psychoacoustic Sonification sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/ToBeAssigned www.sagepub.com/ SAGE Tim Ziemer12 and Holger Schultheis13

Abstract Objective: Three perceptually orthogonal auditory dimensions for multidimensional and multivariate data sonification are identified and experimentally validated. Background: Psychoacoustic investigations have shown that orthogonal acoustical parameters may interfere perceptually. The literature hardly offers any solutions to this problem, and previous auditory display approaches have failed to implement auditory dimensions that are perceived orthogonally by a user. In this study we demonstrate how a location in three-dimensional space can be sonified unambiguously by the implementation of perceptually orthogonal psychoacoustic attributes in monophonic playback. Method: Perceptually orthogonal auditory attributes are identified from literature research and experience in music and psychoacoustic research. We carried out an experiment with 21 participants who identified sonified locations in two-dimensional space. Results: With just 5 minutes of explanation and exploration, naive users can interpret our multidimensional sonification with high accuracy. Conclusion: We identified a set of perceptually orthogonal auditory dimensions suitable for three- dimensional data sonification. Application: Three-dimensional data sonification promises blind navigation, e.g. for unmanned vehicles, and reliable real-time monitoring of multivariate data, e.g., in the patient care sector.

Keywords Auditory Display, Audition, Noise/acoustics, Sound Design, Interpretability

Introduction Noirhomme-Fraiture et al.(2008); Bly(1982); Stewart (2010); Rabenhorst et al.(1990); Hermann(2002). Sonification is a powerful means to complement or replace visual displays, especially in situations in which vision is Here, perceptual orthogonality means that while two quan- limited (e.g., in darkness, fog, smoke, muddy waters, etc. or tities are simultaneously sonified, both can be interpreted. due to occlusion), in which the visual scene is overloaded Moreover, if one quantity changes, the change of sound (e.g. due to too many displays or visual distractors), or in can be attributed to its corresponding quantity, and unam- which spatio-visual processing is the bottleneck of spatial biguously interpreted. This obvious necessity is not eas- cognition Walker and Nees(2011). ily achieved. Due to the complicated, nonlinear processing of the auditory system, all physical sound field quantities There is a need for orthogonal dimensions in sonification can affect practically all perceptual attributes of sound. for multidimensional or multivariate data Neuhoff et al. Despite its importance, the lack of perceptual orthogonality (2002); Yeung(1980); Barrass(1997); Watson and is considered one of the most challenging issues in sonic Sanderson(2004); Worrall(2019)(Worrall 2009, ch. 6). interaction design, auditory interfaces for Human-Computer The most prominent application area for multidimensional Interaction, auditory display, and, especially, sonification sonification is spatial navigation, e.g., in real and virtual design Visell et al.(2013); Brewster(2003); Worrall(2009); environments Ziemer and Schultheis(2018b); Lokki and Hermann(2002); Anderson and Sanderson(2009, 2004); Grohn¨ (2005); Walker and Lindsay(2006), games Degara arXiv:1912.00766v1 [cs.SD] 28 Nov 2019 Neuhoff(2011); Kramer(1994); Grond(2013). et al.(2013a), piloting Towers et al.(2014); Florez(1936), remote vehicle control Vasilijevic et al.(2016), autonomous In this paper we present auditory attributes that can serve drivingGray(2011), image-guided surgical interventions as three orthogonal dimensions. The approach is evaluated in Black et al.(2017); Ziemer and Black(2017a); Ziemer et al. a listening test with naive listeners. (2017) and neuronavigation Willems et al.(2005). Degara et al.(2014) even consider sonification for navigation “one of the most important tasks in auditory display research”. Besides navigation, examples for multidimensional or multivariate data sonification include motion analysis and interactive feedback in sports training and neuromotor 1University of Bremen, Bremen Spatial Cognition Center rehabilitation Huang et al.(2006); Scholz et al.(2014); 2Medical Image Computing Group 3Institute for Artificial Intelligence Reh et al.(2019); Schmitz et al.(2018) and in multivariate Corresponding author: data monitoring, like anesthesia and patient monitoring Tim Ziemer, University of Bremen, Medical Image Computing Group, Sanderson et al.(2005), stock market monitoring Neuhoff Enrique-Schmidt-Str. 5, 28359 Bremen, Germany et al.(2002), data exploration and browsing Yeung(1980); Email: [email protected]

Prepared using sagej.cls [Version: 2017/01/17 v1.20] 2 Journal Title XX(X)

Background when altering several parameters at once, the urgency levels do not add up, but create somewhat nonlinear effects. Orthogonality is a topic that has been treated a lot in the fields of psychoacoustics and auditory display research, and will be Hermann(2002) lists the “lack of perceptual orthogonal- briefly discussed in this section, followed by previous work. ity” as one of the most important difficulties in auditory A lot of previous work either focused on the implementation displays. He agrees with Anderson and Sanderson(2009) of psychoacoustics in sonification design or on orthogonal that auditory perception is too little understood to specify dimensions in sonification. Our work integrates these two multiple orthogonal dimensions. Likewise, Brewster(2003) lines of research by leveraging psychoacoustic knowledge to lists the “lack of orthogonality” one of the problems with sonify perceptually orthogonal dimensions. nonspeech sound. (Worrall 2009, ch. 2) agrees with this observation, too, Orthogonality stating that “(. . . ) parameter mapping requires a working knowledge of how the parameters interact with each other Following the literature on sonification Worrall(2009); perceptually”, because these interaction may obscure data Hermann(2002) and psychoacoustics Schneider(1997, relations and even confuse the listener Worrall(2019). He 2018a) we define dimensions in a Cartesian way as having thinks that attempts to create a perceptually orthogonal both a direction and a distance, also referred to as polarity sonification space have not yet been successful, giving the and magnitude. In that sense, information like the radius, timbre space sonification approach Barrass(1997) as an which is one dimension in polar and cylindrical coordinates, example. At the same time, he expressed the need for better is only a half-dimension; it only informs about a distance, tools. not a direction Parseihian et al.(2016). It is widely accepted that auditory sensations and other psychological attributes are never perfectly orthogonal, as the dimensions may be correlated to some extent Schneider(1997). Hence, we Psychoacoustics in Sonification Design consider dimensions as orthogonal, if a magnitude change of one dimension hardly affects the magnitude of any other The need to consider psychoacoustics in sonification design dimension, often referred to as separable Garner(1974); has been expressed in numerous studies Hellier et al.(1993); Schneider(2018b); Neuhoff(2004). In that sense, they are Hermann(2002); Bovermann et al.(2011); Bly(1982); linearly independent from one another Worrall(2009), i.e., Kramer(1994); Barrass(1994); Smith(1990); Williams they barely exhibit any coupling or perceptual interactions (1994); Ferguson and Brewster(2017); Walker and Kramer (Hermann 2002, ch. 3); Anderson and Sanderson(2004, (2004); Ferguson et al.(2006); Bliss and Spain(2007); Hunt 2009). Furthermore, a dimension must be continuous, i.e., and Hermann(2011); Degara et al.(2013b). on interval scale or ratio scale rather than ordinal or nominal Ferguson and Brewster(2017) evaluate psychoacoustic scale Schneider(1997). parameters for sonification. They argue that pitch is a meaningful dimension, as human listeners have a high Orthogonality in Sonification resolution in pitch perception. They suggest the use of There are plenty examples of sonifications mapping one half- loudness fluctuation and roughness as additional dimensions. dimension to amplitude and another to frequency Neuhoff The authors of Ferguson et al.(2006) suggest mapping of et al.(2002) and it is not surprising that the authors parameters to pitch, loudness, roughness, and brightness. realized in their evaluations that these physically orthogonal Likewise, Parseihian et al.(2016) consider pitch, loudness, dimensions interact perceptually. duration/tempo and timbre as orthogonal and as the main Anderson and Sanderson(2009) tried out several perceptual attributes of sound. Arfib et al.(2002) name pitch, mapping principles for multivariate data in complex loudness, timbre aspects, like brightness, roughness, attack work domains. They do not consider psychoacoustics time, vibrato and formants, spatialization, as well as their in their parameter mapping approach, but map multiple temporal derivatives, as psychoacoustic parameters suitable variables to physical parameters, like amplitude, amplitude for understandable multidimensional sonification. modulations, fundamental frequency, cutoff-frequency, pulse The authors of Parseihian et al.(2016) managed to width, etc. They realize that participants had problems implement and validate sonification designs derived from interpreting multiple variables at once. They criticize psychoacoustic considerations. Here, the distance to a target that psychoacoustic research does not provide sufficient was not just mapped to physical audio parameters, but guidelines for sophisticated, orthogonal sonification design. to psychoacoustic quantities. The distance to a target was Yet, they hope that “...careful sonification design, based mapped to the speed of modulations of either frequency on a complete understanding of the mechanisms causing or amplitude. These modulations create the impression of perceptual interactions, could overcome such problems”. A pitch fluctuations or loudness fluctuations, respectively. Only similar observation has been made by Hellier et al.(1993), at the target location, the pitch, or loudness, respectively, who carried out experiments in which they altered the was steady. They implemented neither a complete one- magnitude of several acoustical quantities to see how it dimensional approach (with both a polarity and a distance) affects perceived urgency. They realized that changing the nor a multi-dimensional approach. But they suggest to magnitude of one parameter, like raising the fundamental map orthogonal dimensions to segregate auditory streams frequency, increasing the amplitude or increasing the Bregman(1990), like one to pitch- and another to tempo- playback speed, increased the perceived urgency. However, fluctuations.

Prepared using sagej.cls Ziemer and Schultheis 3

Orthogonal Psychoacoustic Sonification inexperienced listeners. In a multiple-choice task with 16 A few studies aimed at creating multi-dimensional sonifica- fields on a map, they correctly identified 41% of the sonified tion based on psychoacoustic knowledge. Already in 1980 targets even though the performance of one participant was Yeung(1980) argues that no less than 36 dimensions can near chance level. Over 83% of the quadrants were identified be created from the parameters pitch, loudness, damping, correctly. These results indicate that listeners are able to direction, duration of sound and duration of silence, attack interpret the direction and distance along each dimension time, phase coherence and overtones. However, the study independently, despite the fact that both dimensions are gives no evidence for this claim. The work does not clearly presented at the same time. Motivated by these results we distinguish between physical and perceptual parameters and carried out slight improvements of the sonification and then neglects the interference problems mentioned in the previous conducted an interactive experiment with 18 participants section. Ziemer and Schultheis(2018a,b). Results of this experiment Barrass(1994, 1997) describes a theory to map three underlined that these dimensions are in fact orthogonal, and cylindrical dimensions to a perceptual auditory space. Here, gave additional indication about good linearity and high pitch is the height dimension, brightness is the radius, and resolution of the dimensions and about learnability and timbre in terms of different musical instruments serve as training effects and the way people interact with the sound angles. However, the author identify timbre in terms of in a navigation task. In Ziemer and Schultheis(2019) we musical instruments to be nominal rather than in interval describe a modified signal processing approach to add a third scale. Furthermore, he recognized that this timbre choice dimension to the two-dimensional sonification. However, the does not allow for comprehensible opposite angles, which interpretability and orthogonality have only been explored by would be necessary for interpretable cylindrical coordinates. the authors. Overall, he considered the sounds of his sonification In this contribution we explain how we derived the third approach as difficult to interpret. dimension. We repeat the passive experiment from Ziemer Scholz et al.(2014) map one direction to pitch and another et al.(2017); Ziemer and Black(2017b) to evaluate the one to brightness of a synthesized sound. The sonification orthogonality of our improved two-dimensional sonification informs about the magnitude in each direction, but not about and our newly introduced third dimension as described in the polarity. Hence, we consider these as half-dimensions. Ziemer and Schultheis(2019). In an experiment elderly participants were presented one reference sound. Then, they explored a map with 7 times 7 Psychoacoustic Sonification fields, each playing one sound with a distinct combination This section starts with an overview of perceptual auditory of pitch and brightness. Their task was to select the field qualities that can be found in the literature. We then describe whose sound equaled the reference sound. Their mean error how to derive three orthogonal dimensions, including lay between about 0.3 and 0.7 fields for the pitch direction direction and distance. We distinguish acoustic attributes and between 1 and 1.6 for the brightness direction. A random from auditory attributes, the first describing the physical guess would have led to a mean error of 2.2. Based on these domain, the latter referring to the perceptual domain. results, they consider the two parameters as orthogonal and implement the two, together with loudness as parameter for Perceptual Auditory Qualities the third half-dimension, for motion sonification in Scholz et al.(2016). However, their study does not evaluate the Previous work demonstrated that acoustic attributes may perceptual orthogonality of the third dimension. interfere perceptually, and even individual auditory quali- The authors of Ferguson et al.(2006) come up ties may correlate to some extent. This led to the above with a framework, for psychoacoustic sonification of statements that there is a lack of orthogonal auditory multidimensional or multivariate data. They suggest to map attributesHermann(2002); Worrall(2009); Anderson and one dimension or variable to one psychoacoustic parameter Sanderson(2009). However, in our opinion, orthogonal audi- and another dimension or variable to another psychoacoustic tory dimensions exist. What is missing is a comprehensive parameter. They understand that mapping orthogonal data to treatise of orthogonality in the psychoacoustic literature. As the magnitude of orthogonal auditory qualities is an inverse Neuhoff(2004) states: “. . . perceptual interaction of auditory problem; the desired perceptual outcome is known, but the dimensions (. . . ) have also been studied very little compared the physical audio parameters necessary to create such output with more traditional areas of psychoacoustic research”. need to be found. This problem is ill-posed. Hence, there is However, a heuristic technique to derive perceptually inde- no analytical solution. They suggest to solve the problem pendent attributes for multidimensional sonification is an by massive lookup tables. However, they see the problem accepted and promising approach Worrall(2019). A brief that this solution may cause large jumps of audio parameter discussion is presented in this section. magnitudes by just small changes of the input data. These Literature on auditory sensation and perception describes jumps may cause audible artifacts. several auditory qualities. Some are unidimensional, others In our own previous work we introduced chroma as one are multidimensional. Some are independent from the others, and a combination of beats and roughness as orthogonal whereas some interfere to some extent. Auditory qualities auditory dimensions for two-dimensions sonification Ziemer include: and Black(2017b); Ziemer(2017). The digital signal • Loudness (Zwicker and Fastl 1999, ch. 8); (Ziemer processing for this psychoacoustic sonification approach 2020, ch. 4) is explained in Ziemer et al.(2017, 2018). We validated • Pitch (Zwicker and Fastl 1999, ch. 5); Shepard(1964); the approach in a passive listening experiment with 7 (Ziemer 2020, ch. 4); Neuhoff(2004)

Prepared using sagej.cls 4 Journal Title XX(X)

∗ Height Shepard(1964); (Ziemer 2020, ch. 4); The paper at hand explains how the sonification works Schneider(2018c) and gives an experimental evaluation of the orthogonality ∗ Chroma Shepard(1964); (Ziemer 2020, ch. 4); of these perceptual auditory attributes. Experimental results Schneider(2018c) provide evidence that all dimensions are independent ∗ Strength/Salience (Zwicker and Fastl 1999, ch. from one another. The sonification is perceived as one 5); (Ziemer 2020, ch. 4); Schneider(2018c) sound, i.e., as one auditory stream in terms of auditory • Timbre Schneider(2018b); (Ziemer 2020, ch. 4); scene analysis Ziemer and Black(2017a). This sound Neuhoff(2004) has multiple orthogonal characteristics, i.e., independent ∗ Sound color/tonal color Schneider(2018b); perceptual auditory qualities. Each perceptual auditory (Ziemer 2020, ch. 4) quality represents another direction along the orthogonal ∗ Brightness/sharpness Zwicker and Fastl(1999); dimensions within the Cartesian space. The magnitude of Schneider(2018b) each individual quality indicates the distance along that ∗ Roughness/sensory dissonance (Zwicker and direction. Fastl 1999, ch. 11); Schneider(2018b) ∗ Percussiveness vs. mellowness Schneider Orthogonal Qualities (2018b) From the discussion above, we can derive an orthogonal, ∗ Tonalness/tonality vs. noisiness (Zwicker and three-dimensional sonification. Following the idea of Fastl 1999, ch. 9); Schneider(2018b) Ferguson et al.(2006); Peres et al.(2008), we map ∗ Harmonicity (Ziemer 2020, ch. 4); Schneider three orthogonal dimensions to independent psychoacoustic (2018b) parameters, which are closely related to perceptual auditory ∗ Fullness/volume/sonority Schneider(2018b) qualities. Each quality stands for one direction, its magnitude ∗ Beats / Loudness fluctuation (Ziemer 2020, ch. for the distance along that direction. As most perceptual 4); (Zwicker and Fastl 1999, ch. 4) auditory qualities tend to have a magnitude, but no ∗ Vowel quality/Vocality Schneider(2018b,c) direction, two of the three dimensions are made of two • Sensory pleasantness (Zwicker and Fastl 1999, ch. 9) independent perceptual auditory qualities, one for each • Auditory event location (Zwicker and Fastl 1999, ch. direction along the dimension. The result is a three- 15); (Ziemer 2020, ch. 4); Neuhoff(2004) dimensional sonification made from five psychoacoustic ∗ Azimuth angle (Ziemer 2020, ch. 4) quantities that are summarized in table1. ∗ Median angle (Ziemer 2020, ch. 4) ∗ Distance (Ziemer 2020, ch. 4) Dim. Dir. Psychoac. quantity Dist. left counterclockwise chroma change speed • Perceived source extent (Ziemer 2020, ch. 4) x • Rhythm (Zwicker and Fastl 1999, ch. 13) right clockwise chroma change speed up loudness fluctuation speed ∗ Subjective duration (Zwicker and Fastl 1999, ch. y down roughness degree 12); Schneider(2018b) front fullness degree ∗ Fluctuation strength (Zwicker and Fastl 1999, ch. z back brightness degree 10) Table 1. Psychoacoustic mapping for orthogonal In the psychoacoustic literature orthogonality of only a multidimensional data sonification, including dimension (Dim.), few of the above-listed auditory attribute has been discussed, direction (Dir.) and distance (Dis.). e.g., in Ziemer et al.(2016); Aures(1985); Marozeau and de Cheveigne´(2007); Schneider(2018b, 1997, 2018c); The detailed signal processing for the psychoacoustic Shepard(1964); Grau and Nelson(1988); Neuhoff(2004); sonification is described in Ziemer et al.(2018); Ziemer and Zwicker and Fastl(1999); Terhardt(1981); Lichte(1941). Schultheis(2019). Figure1 illustrates the three-dimensional From our own experience in the recording studio and sonification. The x-axis is the chroma axis. At x = 0, the in psychoacoustic research, and from the above-mentioned pitch is steady in terms of both chroma and height. Targets to literature, we could already combine roughness and the the right are denoted by a clockwise motion of chroma. Most subjective duration of chroma change and beats to two listeners perceive this as a rising pitch Shepard(1964). The dimensions and provide evidence for their orthogonality further to the right, the faster the chroma cycles clockwise. Ziemer et al.(2017); Ziemer(2017); Ziemer and Black In the figure the rising speed of clockwise chroma change (2017b). These two dimensions are briefly described in Sect. is indicated by the blue, clockwise winding whose density Orthogonal Qualities. of turns increases. Targets to the left are denoted by a To extend our previous sonification to three dimensions, counterclockwise motion of chroma. Most listeners perceive the literature suggest the use of sharpness, tonalness, and/or this as a falling pitch Shepard(1964). The further to the left, fullness. Unfortunately, tonalness is not an option, since a the faster the chroma cycles counterclockwise. In the figure low degree of tonalness, i.e., a high degree of noisiness the counterclockwise chroma change is indicated by the blue, eliminates pitch in terms of both height and chroma. This counterclockwise winding. The y-axis is divided in two. means that tonalness is not orthogonal to any aspect of A target above is indicated by cyclic, continuous loudness pitch. This leaves us mainly one choice: the incorporation fluctuation. The distance is denoted by the fluctuation speed. of sharpness and fullness for the two directions of the third The further up, the faster the fluctuation. In the graphic this is dimension. The signal processing to implement this has been indicated by the purple envelope with increasing fluctuation described in Ziemer and Schultheis(2019). density. A target below is denoted by roughness. The further

Prepared using sagej.cls Ziemer and Schultheis 5 down, the higher the degree of roughness. In the graphic this three attributes. Furthermore, we already have a benchmark is indicated by the orange curve that fades from sinusoidal for two- but not for three-dimensional sonification. to random. The z-dimension is also divided in two. Targets We carried out the experiment with N = 21 participants in front are denoted by fullness. The further away, the lower (4 female, age between 20 and 53, median = 26, mean = the degree of fullness. In the graphic this is indicated by the 27.8, σ = 8.5). Most participants were recruited from our rainbow whose spectral bandwidth decreases. Targets in the near environment, i.e., mostly undergraduate and graduate back are denoted by brightness. The distance in this direction computer science students. Participants volunteered to take is denoted by the degree of brightness. In the graphic this is part in the study without monetary compensation. First, indicated by the visual brightness level of the rainbow. the participants filled out a questionnaire, reporting age and sex, confirming that they were not aware of suffering from hearing loss, and rating their previous experience with sonification on a scale from 0 (no experience) to 6 (a lot of experience). Some of the participants had heard previous versions of the sonification, or were familiar with sonification, generally, from their car’s park distance control system, whereas others were completely naive concerning sonification. The participants were arbitrarily assigned to one of the three groups x-y, x-z and z-y, so that each group comprised 7 participants. To each group we first explained the psychoacoustic mapping principle, which took about 5 minutes. First, we explained the sound attributes for the horizontal dimension in colloquial terms and imitated it with our voice. We repeated this for the vertical dimension. Then, in contrast to our earlier study, we let the participants explore the two dimensions themselves with a computer mouse for about 2 minutes. Our hope was that this interaction with the sound would create Figure 1. Psychoacoustic sonification principle. The a better understanding of the sonification, so no participant x-dimension is related to chroma, the y-dimension to beats and would perform at chance level. roughness and the z-dimension to fullness and brightness. In We explained the experimental procedure to the partici- this illustration, the origin of the coordinate system is the pants. We showed them a map with 16 fields as illustrated location of the user (red arrow). in the background of Figs.2 to4. Then, a series of 20 sounds was played to them. Each sound was a sonification Note that the sonification is perceived as one continuous of a location in one of the fields. They could take all the sound, i.e., as one auditory stream in terms of auditory time they needed to decide in what field they assumed the scene analysis (Ziemer 2020, ch. 4); Ziemer and Black sonified target to be and click on it. After each click, the next (2017a). The magnitude of its perceptual auditory qualities target was sonified without a pause in between. Participants inform about the distance along its respective direction. did not receive feedback on their choice. We told them that No reference sound is needed, as the sonification itself the order of sonified targets would be random and that a) one communicates if the target is already reached, and if not, or more target fields might be sonified multiple times and where it is located. b) not necessarily every target field would be sonified. We did this to prevent the participants from drawing conclusions from already experienced trials, like excluding fields that Evaluation they had clicked before. In fact we sonified all 16 target fields In this section we describe our experimental setup to in pseudo-random order and then repeated four randomly evaluate the orthogonality of the proposed dimensions for chosen fields. Participants were allowed to adjust the volume psychoacoustic sonification. Basically, we employed the as they like and even mute the sound occasionally, if it would same experimental setup as in of our previous study with help them to take a break or to concentrate better. passive listeners Ziemer et al.(2017); Ziemer and Black The experimental preparation, i.e., explanation of the (2017b); Ziemer(2017). These existing results serve as a mapping principle, the sonification exploration and the benchmark. description of the experiment process took about 8 minutes. We repeated the experiment for the x-y-plane to ensure that the modified signal processing did not affect the Results and Discussion interpretability of these two dimensions. More importantly, we carried out the same experiment for the x-z and the z- On average, it took participants roughly 7 minutes to y plane to evaluate whether the new z-dimension is readily complete the experiment. The main results are shown in Figs. interpretable and orthogonal to both the x- and the y- 5 to 10. The boxplots give details about user performances in dimension. We decided to stick to two dimensions at a time the three scenarios. They show the score of each individual because this procedure is typical for evaluating orthogonality participant, the range, the 25 and 75 percentile, the median of auditory attributes Neuhoff(2004), since learning two and the arithmetic mean value, and, where available, the attributes is easier for inexperienced listeners than learning results of our previous studies that serve as a benchmark

Prepared using sagej.cls 6 Journal Title XX(X)

36 48 7 15 2

10 9 13 15 1211 14 16

Figure 2. Map and psychoacoustic sonification metaphors for Figure 4. Map and psychoacoustic sonification metaphors for the x-y group. The target fields are numbered from 1 to 16. the z-y group.

90% and 95%. Figs.8 to 10 show how often the x-, y-, or z-direction was identified correctly by the participants. Here, the arithmetic mean lay between 90% and 98%, the median between 95% and 100%. All these measures clearly show that the participants performed similarly well in all groups, and about as good as in our previous experiment, that had already been validated by an interactive experiment Ziemer and Schultheis(2018b).

80

60 — — — 40 ××

20

Figure 3. Map and psychoacoustic sonification metaphors for x-y x-z z-y the x-z group. Figure 5. Boxplot of hits per group, showing minimum and maximum (whiskers), median (white dash) and arithmetic mean Ziemer et al.(2017); Ziemer and Black(2017b); Ziemer (dark dash), the performance of each individual participant (2017). Overall, one can see that the results of all three (dots), and the arithmetic mean (gray x) from our benchmark groups are comparable in magnitude to the results of our study Ziemer et al.(2017); Ziemer and Black(2017b); Ziemer previous study. (2017). Fig.5 shows the hit rates of the three groups, which have a mean value between 51% and 64% and a median Of particular interest was to what extent previous expe- between 40% and 70%. Binomial tests indicated that the rience with sonification had an influence on performance hit rate of every single participant was significantly higher and also to what extent the different axis combinations than expected by chance (all ps≤ 0.001). Accordingly, every were easier/harder to use than others. To investigate these participant was able to interpret the sonification. Fig.6 two questions we proceeded as follows. First, we split shows the number of correct quadrants, having a mean value participants into two groups based on experience: one group between 85% and 91% and a median between 90% and 95%. considered experienced (rating ≥ 4) and the other considered Fig.7 shows how frequently the correct field or its direct inexperienced (rating < 4). We have both cases in each neighbor was identified. Here, as for the quadrants, the mean group, i.e., 4 vs. 3 in x-y, 5 vs. 2 in x-z, and 6 vs. 1 z-y. value lay between 85% and 91% and the median between On average, the participants rated their previous experience

Prepared using sagej.cls Ziemer and Schultheis 7

100 100 — — 90 — — 90 ×× — 80 ×× 80 70 70 60 60 50 50 x-y x-z z-y x-y z-y

Figure 6. Boxplot of correctly identified quadrants per group, Figure 9. Boxplot of correctly identified y-direction per group, showing minimum and maximum (whiskers), median (white showing minimum and maximum (whiskers), median (white dash) and arithmetic mean (dark dash), the performance of dash) and arithmetic mean (dark dash), the performance of each individual participant (dots) and the arithmetic mean (gray each individual participant (dots) and the arithmetic mean (gray x) from our benchmark study Ziemer et al.(2017); Ziemer and x) from our benchmark study Ziemer et al.(2017); Ziemer and Black(2017b); Ziemer(2017). Black(2017b); Ziemer(2017).

100 100 — 90 — — 95 — 80 ×× — 70 90 60 50 85 x-y x-z z-y x-z z-y

Figure 7. Boxplot of either correctly identified field or its’ direct Figure 10. Boxplot of correctly identified z-direction per group, neighbor per group, showing minimum and maximum showing minimum and maximum (whiskers), median (white (whiskers), median (white dash) and arithmetic mean (dark dash) and arithmetic mean (dark dash) and the performance of dash) and the performance of each individual participant (dots). each individual participant (dots).

100 along with a correct left/right and up/down direction, etc. 95 — We therefore carried out a Principal Component Analysis, ×× to summarize the performance of the participants. Here, the 90 — first component explained 81.5% of the variance and the load 85 of all measures on the component ranged between 0.79 and 0.99. Two-way analysis of variance (ANOVA) revealed no 80 significant effect of previous experience and/or group on the 75 results in terms of the first principal components(0.46 < p < 0.69). 70 We have demonstrated already in Ziemer et al.(2017); x-y x-z Ziemer and Schultheis(2018b), that the x- and y-dimensions are orthogonal. Hence, we can conclude from the results Figure 8. Boxplot of correctly identified x-direction per group, of the statistical tests that also the new z-dimension is showing minimum and maximum (whiskers), median (white orthogonal to the previous ones, as the x-z- and z-y-group dash) and arithmetic mean (dark dash), the performance of each individual participant (dots) and the arithmetic mean (gray reveal no significant difference in terms of performance. This x) from our benchmark study Ziemer et al.(2017); Ziemer and finding confirms that the new z dimension is orthogonal to Black(2017b); Ziemer(2017). the x and the y dimension. The new z dimension is equally well-combinable with the x and the y dimension. The fact that some participants have heard previous versions of the with sonification with 2.14 (median = 2) .Our evaluation sonification before did not affect the results. measures — i.e., number of hits, correct quadrants, correctly We observe that the x-y-group in this experiment identified field or neighbor, correct left/right direction and performed better than in our previous study Ziemer et al. correct up/down direction — exhibit relatively high correla- (2017); Ziemer(2017). The main reason for this may tion (0.55 ≤ ρ ≤ 0.95), as, for example a hit naturally comes be that we optimized the mapping based on the results

Prepared using sagej.cls 8 Journal Title XX(X) of the previous study. Another reason may be the slight successfully identified and implemented three orthogonal difference in the signal processing between Ziemer et al. auditory dimensions that are interpretable by inexperienced, (2018) and Ziemer and Schultheis(2019). Furthermore, passive listeners. letting the participants interactively explore the single and the combined audible dimensions may have improved their Conclusion understanding of the perceptual auditory qualities and the psychoaocustic mapping principle. In this work we have highlighted the need of perceptually Tables2 to4 are tables of confusion. They show the orthogonal dimensions for multidimensional or multivariate relationship between sonified and clicked target fields not data sonification. We suggest five psychoacoustic quantities with a focus on the individual participant but on the that can serve as three orthogonal dimensions. Experimen- individual field. Here, each row represents a sonified target tal results show that these dimensions are learnable by field and the columns indicate the target as selected by inexperienced listeners under passive conditions. Partici- the participants. The elements in the table indicate how pants were able to interpret the direction and distance of a frequently each field was marked as the target field by the sonified location in all two-dimensional pairs of the three- participants. Consequently, the total of each row is 100 %. dimensional space. As our previous experiment on two- In addition to the numbers, the frequency is also indicated dimensional sonification Ziemer and Schultheis(2018b) has by gray level from white (0 %) to black (100%). The four shown, the accuracy was much higher when participants quadrants are separated by double lines to highlight how interacted with the sound instead of listening passively. Fur- many false clicks fall into the right quadrant. These tables thermore, its resolution had been proven to be very high and help to get a further impression of whether the distributions the axes had been perceived as linear. Interactive experiments of clicked targets differs significantly from random clicks, with the newly developed three-dimensional sonification will whether the participants performed similarly well in all three reveal whether the third dimension has the same qualities dimension pairs, and whether the distribution is roughly concerning accuracy, resolution and linearity. uniform across the whole two-dimensional spaces. One can clearly see the dark diagonal lines, which Outlook indicates that the target field was typically correctly As “(. . . ) is critical to examine performance longitudinally identified most frequently in all three groups. The correct when evaluating auditory display designs” Walker and field was identified in 25 to 90.9% of all trials. Only in Lindsay(2003), we are designing a game to motivate two to three out of 16 cases per group the most frequently users for long-term interaction with the sonification. chosen field did not coincide with the sonified target field. Progress on the game can be found on http:// Most confusions were between the target and other fields curat.informatik.uni-bremen.de/. In addition from the same quadrant. For most sonified targets, only three to interactive experiments to evaluate the sonification itself, to four fields have been clicked at all. Only for very few we plan to evaluate the benefit of the sonification in a targets, more than four different fields have been clicked potential application area, like an image-guided surgery by participants, namely t10 and t15 in group x-y, none in scenario. group x-y, and t9 and t14 in group z-y. Some targets were only confused with one other target. This was the case for t1, t5, t11 in group x-z and for t6, t7, t10 in group z- Key Points y. The x-y group identified the outermost fields well, i.e., • We discussed the problem of low interpretabil- t4, t8, t12 and t16, and did not click on them, when any ity due to a lack of orthogonality in multidimen- other target was sonified. The same is true for fields t4, tional/multivariate sonification t6, t9 and t13 for the x-z group and for t3, t4 and t9 • We identified a number of auditory attributes that seem in the z-y group. From visual inspection, all three tables perceptually orthogonal seem similar to each other. This observation is confirmed • We implemented them in a psychoacoustic sonifica- by Kendall’s τ test. After vectorization of the confusion tion matrices to one-dimensional vectors, the three show a fair but • Our experiment revealed that all three dimensions are highly significant rank correlation (τ = 0.56, p = 4 × 10−23 in fact orthogonal to each other between x-y and x-z, τ = 0.49, p = 2 × 10−18 between x- y and z-y, and τ = 0.54, p = 6 × 10−21). This observation References supports the finding from the ANOVA, i.e., that group did not have a significant effect on performance. The fair correlation Anderson J and Sanderson P (2004) Designing sonification for is owed to the fact that all three share the strong diagonal. effective attentional control in complex work domains. In: But the participants did not confuse the same fields in all Proc. Human Factors and Ergonomins Society 48th annual three groups. The two groups with the new z dimension meeting. New Orleans, LA. DOI:10.1037/e577082012-006. exhibit similarities with the x-y pair, which has already been Anderson JE and Sanderson P (2009) Sonification design for shown to be orthogonal. This suggests that each dimension complex work domains: Dimensions and distractors. Journal can be interpreted correctly during the presence of a second of Experimental Psychology: Applied 15(3): 183–198. DOI:10. dimension, which indicates that all three dimensions are 1037/a0016329. URL http://dx.doi.org/10.1037/ orthogonal. a0016329. All observations from the box plots5 to 10 and the Arfib D, Couturier J, Kessous, L and Verfaille V (2002) Strategies confusion matrices2 to4 draw a coherent picture: We of mapping between gesture data and synthesis model

Prepared using sagej.cls Ziemer and Schultheis 9

t1 t2 t3 t4 t5 t6 t7 t8 t9 t19 t11 t12 t13 t14 t15 t16 t1 80. 10. 10. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. t2 33.3 44.4 22.2 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. t3 20. 10. 60. 0. 0. 0. 10. 0. 0. 0. 0. 0. 0. 0. 0. 0. t4 0. 11.1 44.4 33.3 0. 11.1 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. t5 0. 0. 0. 0. 66.7 22.2 0. 0. 11.1 0. 0. 0. 0. 0. 0. 0. t6 0. 10. 0. 0. 10. 70. 10. 0. 0. 0. 0. 0. 0. 0. 0. 0. t7 0. 0. 0. 0. 42.9 14.3 42.9 0. 0. 0. 0. 0. 0. 0. 0. 0. t8 10. 0. 0. 0. 20. 0. 40. 30. 0. 0. 0. 0. 0. 0. 0. 0. t9 0. 11.1 0. 0. 11.1 0. 0. 0. 44.4 0. 33.3 0. 0. 0. 0. 0. t10 12.5 0. 0. 0. 0. 12.5 0. 0. 12.5 37.5 0. 12.5 0. 0. 0. 12.5 t11 0. 12.5 0. 0. 0. 0. 12.5 0. 12.5 0. 62.5 0. 0. 0. 0. 0. t12 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 37.5 37.5 0. 12.5 12.5 0. t13 0. 14.3 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 71.4 14.3 0. 0. t14 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 28.6 0. 0. 57.1 14.3 0. t15 10. 0. 0. 10. 0. 0. 0. 0. 10. 0. 0. 0. 10. 10. 50. 0. t16 0. 0. 0. 0. 0. 0. 0. 0. 0. 11.1 0. 0. 11.1 22.2 11.1 44.4 Table 2. Confusion matrix (in percentages) for the x-y-plane. The rows represent the sonified targets, the columns the selected field. The values indicate how often each field has been clicked when either of the targets was sonified.

t1 t2 t3 t4 t5 t6 t7 t8 t9 t19 t11 t12 t13 t14 t15 t16 t1 90.9 0. 9.1 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. t2 12.5 75. 0. 0. 0. 0. 12.5 0. 0. 0. 0. 0. 0. 0. 0. 0. t3 22.2 11.1 66.7 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. t4 0. 40. 20. 30. 0. 0. 0. 10. 0. 0. 0. 0. 0. 0. 0. 0. t5 0. 0. 0. 0. 87.5 0. 0. 12.5 0. 0. 0. 0. 0. 0. 0. 0. t6 0. 0. 0. 0. 60. 30. 0. 0. 0. 0. 10. 0. 0. 0. 0. 0. t7 0. 0. 0. 0. 0. 0. 75. 12.5 0. 12.5 0. 0. 0. 0. 0. 0. t8 0. 0. 0. 0. 11.1 11.1 11.1 66.7 0. 0. 0. 0. 0. 0. 0. 0. t9 0. 0. 0. 0. 9.1 0. 0. 0. 63.6 0. 27.3 0. 0. 0. 0. 0. t10 0. 0. 0. 0. 0. 0. 0. 0. 0. 71.4 0. 14.3 0. 0. 14.3 0. t11 0. 0. 0. 0. 0. 0. 0. 0. 25. 0. 75. 0. 0. 0. 0. 0. t12 0. 0. 0. 0. 0. 0. 0. 0. 0. 12.5 12.5 62.5 0. 12.5 0. 0. t13 12.5 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 37.5 50. 0. 0. t14 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 11.1 0. 0. 55.6 11.1 22.2 t15 0. 25. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 62.5 12.5 t16 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 12.5 0. 0. 12.5 75. Table 3. Confusion matrix (in percentages) for the x-z-plane. The rows represent the sonified targets, the columns the selected field. The values indicate how often each field has been clicked when either of the targets was sonified.

t1 t2 t3 t4 t5 t6 t7 t8 t9 t19 t11 t12 t13 t14 t15 t16 t1 55.6 11.1 0. 0. 11.1 22.2 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. t2 45.5 36.4 0. 9.1 9.1 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. t3 12.5 0. 75. 0. 0. 12.5 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. t4 0. 28.6 28.6 28.6 0. 14.3 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. t5 0. 0. 0. 0. 62.5 25. 12.5 0. 0. 0. 0. 0. 0. 0. 0. 0. t6 0. 0. 0. 0. 0. 77.8 0. 22.2 0. 0. 0. 0. 0. 0. 0. 0. t7 0. 0. 0. 0. 0. 0. 77.8 22.2 0. 0. 0. 0. 0. 0. 0. 0. t8 0. 0. 0. 0. 0. 12.5 12.5 62.5 0. 12.5 0. 0. 0. 0. 0. 0. t9 0. 0. 0. 0. 0. 0. 0. 0. 30. 20. 30. 10. 0. 10. 0. 0. t10 0. 0. 0. 0. 0. 0. 0. 0. 0. 30. 0. 70. 0. 0. 0. 0. t11 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 45.5 27.3 0. 0. 9.1 18.2 t12 0. 0. 0. 0. 0. 14.3 0. 14.3 0. 0. 0. 71.4 0. 0. 0. 0. t13 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 42.9 42.9 0. 14.3 t14 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 12.5 0. 12.5 25. 25. 25. t15 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 10. 10. 60. 20. t16 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 12.5 25. 12.5 50. Table 4. Confusion matrix (in percentages) for the z-y-plane. The rows represent the sonified targets, the columns the selected field. The values indicate how often each field has been clicked when either of the targets was sonified.

Prepared using sagej.cls 10 Journal Title XX(X)

parameters using perceptual spaces. Journal of Organised doi.org/10.2514/8.176. Sound 7(2): 127–144. DOI:10.1017/S1355771802002054. Garner WR (1974) The processing of information and structure. Aures W (1985) Berechnungsverfahren fur¨ den sensorischen Oxford: Lawrence Erlbaum. wohlklang beliebiger schallsignale (a model for calculating the Ghez C, Rikakis T, DuBois RL and Cook PR (2000) sensory euphony of various sounds). Acustica 59(2): 130–141. An auditory display system for aiding interjoint Barrass S (1994) A perceptual framework for the auditory display coordination. In: ICAD. Atlanta (GO). URL http: of scientific data. In: International Conference on Auditory //www.icad.org/websiteV2.0/Conferences/ Display. Santa Fe, pp. 131–145. URL http://hdl. ICAD2000/PDFs/LukeGhez.pdf. handle.net/1853/50821. Grau JW and Nelson DK (1988) The distinction between integral Barrass S (1997) Auditory Information Design. PhD Thesis, and separable dimensions: Evidence for the integrality of pitch Australian National University, Canberra. URL http:// and loudness. J. Experimental Psychol.: Gen. 117(4): 347–370. hdl.handle.net/1885/46072. DOI:10.1037/0096-3445.117.4.347. Black D, Hettig J, Luz M, Hansen C, Kikinis R and Hahn H Gray R (2011) Looming auditory collision warnings for (2017) Auditory feedback to support image-guided medical driving. Human Factors 53(11): 63–74. DOI:10.1177/ needle placement. International Journal of Computer Assisted 0018720810397833. URL http://doi.org/10.1177/ Radiology and Surgery 12(9): 1655–1663. DOI:10.1007/ 0018720810397833. s11548-017-1537-1. URL http://dx.doi.org/10. Grond F (2013) Listening-Mode-Centered Sonification Design 1007/s11548-017-1537-1. for Data Exploration. PhD Thesis, Universitat¨ Bielefeld, Bliss JP and Spain RD (2007) Sonification and reliability — Bielefeld. implications for signal design. In: ICAD. Montreal,´ pp. 154– Hellier EJ, Edworthy J and Dennis I (1993) Improving auditory 159. URL http://hdl.handle.net/1853/50028. warning design: Quantifying and predicting the effects of Bly S (1982) Sound and computer information presentation. PhD different warning parameters on perceived urgency. Human Thesis, University of California. DOI:10.2172/5221536. Factors 35(4): 693–706. Bovermann T, Rohrhuber J and de Campo A (2011) Laboratory Hermann T (2002) Sonification for Exploratory Data Analysis. methods for experimental sonification. In: Hermann T, PhD Thesis, Bielefeld University, Bielefeld. URL http: Hunter A and Neuhoff JG (eds.) The Sonification Handbook, //www.techfak.uni-bielefeld.de/ags/ami/ chapter 10. Berlin: COST and Logos, pp. 237–272. URL publications/media/Hermann2002-SFE.pdf. http://sonification.de/handbook/. Hermann T, Honer¨ O and Ritter H (2006) AcouMotion – An Bregman AS (1990) Auditory Scene Analysis. Cambridge, MA: Interactive Sonification System for Acoustic Motion Control, MIT Press. Lecture Notes in Computer Science, volume 3881. Berlin, Brewster S (2003) Nonspeech auditory output. In: Jacko JA and Heidelberg: Springer, pp. 312–323. DOI:10.1007/11678816 Sears A (eds.) The Human-Computer Interaction Handbook. 35. URL http://link.springer.com/chapter/ London: Lawrence Erlbaum Associates, pp. 220–239. 10.1007%2F11678816_35. Degara N, Kuppanda T and Nagel F (2013a) The walking game: Huang H, Wolf SL and He J (2006) Recent developments A framework for evaluating sonification methods in blind in biofeedback for neuromotor rehabilitation. Journal of navigation. In: Proceedings of the 4th Interactive Sonification NeuroEngineering and Rehabilitation 3(11). DOI:10.1186/ Workshop (ISon2013). Erlangen. 1743-0003-3-11. URL http://www.jneuroengrehab. Degara N, Kuppanda T, Neate T, Yang J and Torres AV (2014) com/content/3/1/11. Reproducible sonification for virtual navigation. In: 2014 Hunt A and Hermann T (2011) Interactive sonification. In: IEEE VR Workshop: Sonic Interaction in Virtual Environments Thomas Hermann JGN Andy Hunt (ed.) The Sonification (SIVE). pp. 35–40. DOI:10.1109/SIVE.2014.7006288. Handbook, chapter 11. Berlin: COST and Logos, pp. 273–298. Degara N, Nagel F and Hermann T (2013b) Sonex: An evaluation URL http://sonification.de/handbook/. exchange framework for reproducible sonification. In: Kramer G (1994) Some organizing principles for representing data Proceedings of the 19th International Conference on Auditory with sound. In: Kramer G (ed.) Auditory Display: Sonification, Display (ICAD2013). Lodz. URL http://hdl.handle. Audification, and Auditory Interfaces, Santa Fe Studies in net/1853/51662. the Science of Complexity, Proc. Vol. XVIII. Reading, MA: Ferguson J and Brewster SA (2017) Evaluation of psychoacoustic Addison-Wesley, pp. 223–252. sound parameters for sonification. In: Proceedings of Lichte W (1941) Attributes of complex tones. J. Exp. Psychol. 28: the 19th ACM International Conference on Multimodal 455–481. Interaction, ICMI ’17. New York, NY, USA: ACM. ISBN Lokki T and Grohn¨ M (2005) Navigation with auditory cues in a 978-1-4503-5543-8, pp. 120–127. DOI:10.1145/3136755. virtual environment. IEEE MultiMedia 12(2): 80–86. DOI: 3136783. URL http://doi.acm.org/10.1145/ 10.1109/MMUL.2005.33. 3136755.3136783. Marozeau J and de Cheveigne´ A (2007) The effect of fundamental Ferguson S, Cabrera D, Beilharz K and Song HJ (2006) frequency on the brightness dimension of timbre. Journal of Using psychoacoustical models for information sonification. the Acoustical Society of America 121: 383–387. In: 12th International Conference on Auditory Display Neuhoff JG (2004) Interacting perceptual dimensions. In: Neuhoff (ICAD2006). London. URL http://hdl.handle.net/ JG (ed.) Ecological Psychoacoustics, chapter 10. San Diego 1853/50694. (CA): Elsevier, pp. 249–269. Florez L (1936) True blind flight. Journal of the Aeronautical Neuhoff JG (2011) Perception, cognition and action in auditory Sciences 3(5): 168–170. DOI:10.2514/8.176. URL http: displays. In: Hermann T, Hunt A and Neuhoff JG (eds.)

Prepared using sagej.cls Ziemer and Schultheis 11

The Sonification Handbook, chapter 4. Berlin: COST and movements in stroke rehabilitation – a novel approach in Logos, pp. 63–85. URL http://sonification.de/ neurologic music therapy. Frontiers in Neurology 7: Article handbook/. ID: 106. DOI:10.3389/fneur.2016.00106. Neuhoff JG, Kramer G and Wayand J (2002) Pitch and loudness Scholz DS, Wu L, Pirzer J, Schneider J, Rollnik JD, Großbach M interact in auditory displays: Can the data get lost in the map? and Altenmuller¨ EO (2014) Sonification as a possible stroke Journal of Experimental Psychology: Applied 8(1): 17–25. rehabilitation strategy. Frontiers in Neuroscience 8: Article Noirhomme-Fraiture M, Scholler¨ O, Demoulin C and 332. DOI:10.3389/fnins.2014.00332. URL http://doi. Simoff SJ (2008) Complementing Visual Data Mining org/10.3389/fnins.2014.00332. with the Sound Dimension: Sonification of Time Schwarz S and Ziemer T (2019) A psychoacoustic sound design for Dependent Data. Berlin, Heidelberg: Springer Berlin pulse oximetry. In: 25th International Conference on Auditory Heidelberg. ISBN 978-3-540-71080-6, pp. 236–247. Display (ICAD 2019). Newcastle. DOI:10.21785/icad2019. DOI:10.1007/978-3-540-71080-6 15. URL https: 024. URL http://hdl.handle.net/1853/61504. //doi.org/10.1007/978-3-540-71080-6_15. Shepard RN (1964) Circularity in judgments of relative pitch. The Parseihian G, Gondre C, Aramaki M, Ystad S and Kronland- Journal of the Acoustical Society of America 36(12): 2346– Martinet R (2016) Comparison and evaluation of sonification 2353. DOI:10.1121/1.1919362. strategies for guidance tasks. IEEE Trans. Multimedia 18(4): Smith S (1990) Representing data with sound. In: Proc. IEEE 674–686. DOI:10.1109/TMM.2016.2531978. Visualization. Piscataway (NJ). Peres SC, Best V, Brock D, Frauenberger C, Hermann T, Neuhoff Stewart R (2010) Spatial Auditory Display for Acoustics and JG, Nickerson LV, Shinn-Cunningham B and Stockman T Music Collections. PhD Thesis, University of London. (2008) Auditory interfaces. In: P K (ed.) HCI Beyond the GUI. URL https://qmro.qmul.ac.uk/xmlui/handle/ Design for haptic, speech, olfactory and other nontraditional 123456789/413. interfaces, chapter 5. Burlington, MA: Morgan Kaufman, pp. Terhardt E (1981) Aspekte und moglichkeiten der gehorbezogenen¨ 147–195. schallanalyse und -bewertung. In: Fortschritte der Akustik, Rabenhorst DA, Farrell EJ, Jameson DH, Linton TD and DAGA ’81. Berlin: VDE-Verlag, pp. 99–110. Mandelman JA (1990) Complementary visualization and Towers J, Burgess-Limerick R and Riek S (2014) Concurrent 3-d sonification of multi-dimensional data. In: SPIE: Extracting sonifications enable the head-up monitoring of two interrelated Meaning from Complex Data: Processing, Display, Interaction. aircraft navigation instruments. Human Factors 56(8): 1414– Reh J, Hwang TH, Schmitz G and Effenberg AO (2019) 1427. DOI:10.1177/0018720814536443. Dual mode gait sonification for rehabilitation after unilateral Vasilijevic A, Vukic Z and Miskovic N (2016) Teleoperated hip arthroplasty. Brain Sci. 9(3, 66: Special Issue: trajectory tracking of remotely operated vehicles using spatial Rhythm Perception and Neural Plasticity). DOI:10.3390/ auditory interface. In: IFAC PapersOnLine, volume 49. pp. brainsci9030066. 079–102. Sanderson PM, Watson MO and Russell WJ (2005) Advanced Visell Y, Murray-Smith R, Brewster SA and Williamson J (2013) patient monitoring displays: Tools for continuous informing. Continuous auditory and tactile interaction design. In: Serafin Anesthesia & Analgesia 101(1). DOI:10.1213/01.ANE. K Stefania amd Franinovic (ed.) Sonic Interaction Design, 0000154080.67496.AE. chapter 3. Cambridge, MA: MIT Press, pp. 77–123. Schmitz G, Bergmann J, Effenberg AO, Krewer C, Hwang TH and Walker BN and Kramer G (2004) Ecological psychoacoustics and Muller¨ F (2018) Movement sonification in stroke rehabilitation. auditory displays: Hearing, grouping, and meaning making. Frontiers in Neurology 9: Article: 389. DOI:10.3389/fneur. In: Neuhoff JG (ed.) Ecological Psychoacoustics, chapter 6. 2018.00389. URL https://www.frontiersin.org/ Amsterdam: Elsevier, pp. 149–174. article/10.3389/fneur.2018.00389. Walker BN and Lindsay J (2003) Effect of beacon sounds on Schneider A (1997) Tonhohe–Skala–Klang.¨ Akustische, navigation performance in a virtual reality environment. In: tonometrische und psychoakustische Studien auf Proceedings of the 2003 International Conference on Auditory vergleichender Grundlage. Bonn: Orpheus-Verlag fur¨ Display (ICAD2003). Boston, MA, pp. 204–207. Systematische Musikwissenschaft. Walker BN and Lindsay J (2006) Navigation performance with Schneider A (2018a) Fundamentals. In: Bader R (ed.) Springer a virtual auditory display: Effects of beacon sound, capture Handbook of Systematic Musicology. Berlin, Heidelberg: radius, and practice. Human Factors 48(2): 265–278. Springer. ISBN 978-3-662-55004-5, pp. 559–603. DOI: Walker BN and Nees MA (2011) Theory of sonification. In: 10.1007/978-3-662-55004-5 30. Hermann T, Hunt A and Neuhoff JG (eds.) The Sonification Schneider A (2018b) Perception of timbre and sound color. In: Handbook, chapter 2. Berlin: COST and Logos, pp. 9–39. URL Bader R (ed.) Springer Handbook of Systematic Musiwology, http://sonification.de/handbook/. chapter 32. Berlin, Heidelberg: Springer, pp. 687–726. URL Watson M and Sanderson P (2004) Sonification supports eyes-free http://doi.org/10.1007/978-3-662-55004-5_ respiratory monitoring and task time-sharing. Human Factors 32. 46(3): 497–517. DOI:10.1518/hfes.46.3.497.50401. Schneider A (2018c) Pitch and pitch perception. In: Bader R Willems P, Noordmans H, van Overbeeke J, Viergever M, Tulleken (ed.) Springer Handbook of Systematic Musicology. Berlin, C and von der Sprenkel JB (2005) The impact of auditory Heidelberg: Springer Berlin Heidelberg. ISBN 978-3-662- feedback on neuronavigation. Acta Neurochir 147: 167–173. 55004-5, pp. 605–685. DOI:10.1007/978-3-662-55004-5 31. DOI:10.1007/s00701-004-0412-3. Scholz DS, Rohde S, Nikmaram N, Bruckner¨ HP, Großbach M, Williams SM (1994) Perceptual principles in sound grouping. In: Rollnik JD and Altenmuller¨ EO (2016) Sonification of arm Kramer G (ed.) Auditory Display: Sonification, Audification

Prepared using sagej.cls 12 Journal Title XX(X)

and Auditory Interfaces. Reading (MA): Addison Wesley, pp. Retrieval Evaluation, FIRE ’16. New York, NY, USA: 95–126. ACM. ISBN 978-1-4503-4838-6, pp. 1–7. DOI:10.1145/ Worrall D (2009) Sonification and Information. Concepts, 3015157.3015158. URL http://doi.acm.org/10. Instruments and Techniques. PhD Thesis, University of 1145/3015157.3015158. Canberra, Canberra. URL http://worrall.avatar. Zwicker E and Fastl H (1999) Psychoacoustics. Facts and Models. com.au/papers/phd/. second updated edition. Berlin, Heidelberg: Springer. DOI: Worrall D (2019) Sonification Design. Cham: Springer. ISBN 978- 10.1007/978-3-662-09562-1. 3-030-01497-1. DOI:10.1007/978-3-030-01497-1. Yeung ES (1980) Pattern recognition by audio representation of multivariate analytical data. Anal. Chem. 52: 1120–1123. DOI: 10.1021/ac50057a028. URL http://pubs.acs.org/ doi/pdf/10.1021/ac50057a028. Ziemer T (2017) Two-dimensional psychoacoustic sonification. In: Olbertz F (ed.) 33. Jahrestagung der deutschen Gesellschaft fur¨ Musikpsychologie (DGM). Hamburg, pp. 60–61. URL https://www.researchgate.net/publication/ 319778727_Two-dimensional_psychoacoustic_ sonification. Ziemer T (2020) Psychoacoustic Music Sound Field Synthesis, Current Research in Systematic Musicology, volume 7. Cham: Springer. ISBN 978-3-030-23033-3. DOI:10.1007/ 978-3-030-23033-3. Ziemer T and Black D (2017a) Psychoacoustic sonification for tracked medical instrument guidance. The Journal of the Acoustical Society of America 141(5): 3694. DOI:10.1121/ 1.4988051. URL http://dx.doi.org/10.1121/1. 4988051. Ziemer T and Black D (2017b) Psychoacoustically motivated sonification for surgeons. In: International Journal of Com- puter Assisted Radiology and Surgery, volume 12. Barcelona, pp. 265–266. URL https://link.springer.com/ article/10.1007/s11548-017-1588-3. Ziemer T, Black D and Schultheis H (2017) Psychoacoustic sonification design for navigation in surgical interventions. Proceedings of Meetings on Acoustics 30. DOI:10.1121/ 2.0000557. URL http://asa.scitation.org/doi/ abs/10.1121/2.0000557. Ziemer T and Schultheis H (2018a) A psychoacoustic auditory display for navigation. In: 24th International Conference on Auditory Displays (ICAD2018). Houghton, MI. DOI: 10.21785/icad2018.007. URL http://doi.org/10. 21785/icad2018.007. Ziemer T and Schultheis H (2018b) Psychoacoustic auditory display for navigation: an auditory assistance system for spatial orientation tasks. J. Multimodal User Interfaces Special Issue: Interactive Sonification. DOI:10.1007/s12193-018-0282-2. Ziemer T and Schultheis H (2019) Psychoacoustical signal processing for three-dimensional sonification. In: 25th International Conference on Auditory Displays (ICAD2019). Newcastle. DOI:10.21785/icad2019.018. URL https:// smartech.gatech.edu/handle/1853/61499. Ziemer T, Schultheis H, Black D and Kikinis R (2018) Psychoacoustical interactive sonification for short range navigation. Acta Acustica united with Acustica 104(6): 1075–1093. DOI:10.3813/AAA.919273. URL https://www.ingentaconnect.com/content/ dav/aaua/2018/00000104/00000006/art00018. Ziemer T, Yu Y and Tang S (2016) Using psychoacoustic models for sound analysis in music. In: Proceedings of the 8th Annual Meeting of the Forum on Information

Prepared using sagej.cls