Quick viewing(Text Mode)

Sound Effects for a Silent Computer System

Sound Effects for a Silent Computer System

Proc. of the 5th Int. Conference on Digital Audio Effects (DAFX-02), Hamburg, Germany, September 26-28, 2002

SOUND EFFECTS FOR A SILENT SYSTEM

Christian Müller-Tomfelde

IPSI - Integrated Publication and Information Systems Institute Fraunhofer IPSI, Dolivostr. 15, D-64293 Darmstadt, Germany [email protected]

ABSTRACT system and to fulfill learned expectations (see also related realizations [4]). In the following the context of this project and a This paper proposes the sonification of the activity of a computer typical situation of sound during the interaction with a system that allows the user to monitor the basic performance computer are described. Then, the future of noise-free computer parameters of the system, like CPU load, read and write activity systems is anticipated, followed by the description of a realized of the hard disk or network traffic. Although, current computer software prototype for the proposed sound feedback. systems still produce acoustic background noise, future and emerging computer systems will be more and more optimized with respect to their noise emission. In contrast to most of the 2. HYBRID ENVIROMENTS concepts of auditory feedback, which present a particular sound On the continuum between the real world environment and a as a feedback to a user’s command, the proposed feedback is complete virtual environment are several approaches in between mediated by the running computer system. The user’s interaction combining the properties and advantages of the two ends of the stimulates the system and hence the resulting feedback offers continuum. Milgram called these approaches along the more realistic information about the current states of continuum ‘mixed reality’ and build a taxonomy for the different performance of the system. On the one hand the proposed visual displays, based on video and computer display techniques sonification can mimic the acoustical behavior of operating [5]. In the area of a general computer workplace environment the components inside a computer system, while on the other hand, continuum starts with the traditional desktop computer new qualities can be synthesized that enrich interaction with the environment and finishes in immersive, three dimensional device. Different forms of sound effects and generation for the environments, like described in [6]. Among these approaches so proposed auditory feedback are realized to experiment with the called hybrid environments combine on one hand more direct usage in an environment of silent computer systems. interaction and manipulation of the user with data represented by while on the other hand objects of the environment 1. INTRODUCTION are still tangible and become augmented with information technology. In the past various research has been done to extend the visual desktop metaphor for computer environments with audio feedback and thus to enrich the interaction with computers. Today, audio facilities are available in every standard desktop computer environment, i.e., a sound card is standard in personal computers and audio feedback is integrated into popular operating systems and applications. Different concepts were developed to map sound effects to user input events to enrich the user interaction with computers. For example, with the concept of Earcons an event is translated into a short musical sequence of tones, which has to be learned to understand their meaning by the listener [1]. While the concept of Auditory Icons introduced in [2] and [3] uses everyday sounds of the environment that are well-know by the user to convey information about events and states within a computer. Now, instead of adding sounds as a direct feedback to the interaction in a computer environment the emphasis and focus of this paper lies more on the sounds that the computer produces as a device itself and as a reaction to the input. The computer systems of the near future will loose their technically constrained noise aura and this can lead to more Figure 1: An example for a silent computer system in a comfortable work atmosphere but also to irritation during the hybrid environment: the ConnecTable® is build upon interaction. The imitation or a sort of a rebirth of the sounds of computer components that emit no noise (Design: computer systems can help the user to monitor the state of the Fraunhofer IPSI, Wiege, Wilkhahn).

DAFX-227 Proc. of the 5th Int. Conference on Digital Audio Effects (DAFX-02), Hamburg, Germany, September 26-28, 2002

8Mark Weiser coined for these kind of computer environments something is happening due to his command. The real operating also the term ‘ubiquitous computing’ [7] emphasizing the aspect components of the computer system emit sound, that has typical of the real space as an interface for the interaction with characteristics. The sound generation block in Figure 2 should information. In a environment like described in [9] the objects comprise all these effects a standard computer system. In the within a room, like chairs, tables and walls become hybrid, i.e., chain of actions during this sequence also the click of the they have a significant role in the real world and also function as keyboard can be identified as an early feedback for the user that an interface to interact with information in a virtual world in the the key is/was pressed. In Figure 2 this is represented by the computer. For cooperative work situations the hybrid direct arrow (through the application) between the input event environment acts as a traditional room which guarantees flexible and the sound generation. This feedback does not give the user and fast group interaction and at the same time free face-to face any hints about the states of the computer system, e.g. whether communication. The augmentation of the objects with the command is executed or not. This is the same in the case of a information technology provides an additional virtual space in synthetic sound feedback with a samples sound or musical motif which the users can cooperate in parallel. In Figure 1 the assigned to the input event. This is currently accomplished by ConnecTable is depicted as a typical example for a hybrid some system settings of popular computer operating systems and artifact. The small desk is designed to sit and to stand in front of applications. Examples for this kind of feedback are the selection it. The desktop is based on a computer display on which the user of the items for the pull-down menu or the opening and closing can interact with a pen. All other components of the computer of applications. In the standard setting there is no sound feedback are integrated in the case aside the column of the ConnecTable. considered that represents a inherent state of the computer over The system works completely wireless and is entire free from time (expect perhaps for the situation when the wireless infrared typical computer noise from cooling fans, spinning hard disks connection temporally is broken down. This is illustrated by and, of course keyboard clicks. Hence, the ConnecTable lopped sound that occurs during the time of the dropout). But functions as a test object to experiment with sound effects for a currently the inherent states computer system that depends hybrid artifact based on a silent computer system. The name indirectly on the input event of the user, as shown in Figure 2 ConnecTable refers to a mechanism that couples co-located still bases on the sound generation that is the result of physical displays of two desks to one homogenous workspace [8]. behavior of mechanical parts within the computers.

3. INTERACTION SOUND FEEDBACK 3.1. The disappearing sounds

A common example of the sound feedback of a traditional In this paper, the assumption is made, that future computer computer system should illustrate the sound feedback processing systems become more and more inaudible and finally noise-free, triggered by an input event. The user presses the key as demonstrated with the ConnecTable [8][9]. Hence, the well combination ‘Ctrl-s’ on the keyboard and forces the currently known acoustical behavior will vanish and sound artifacts focussed application to store the edited document. typically produced by a computer system will disappear. In the recent history of the computer industry two products are prominent examples with respect to the decreasing noise emission in computer environments. On one hand the application Cube is a computer system that is powered by a unit without an input event additional cooling fan. A chimney like construction of the central sound arrangement of the power supply unit allows a complete passive realization of the cooling of this unit. Passive or alternative generation keyboard stroke methods of cooling are possible and applied for the central mouse click performance processing unit (CPU) in other standard computer systems by network request system using a large heat sinks as well as liquid cooling techniques. On processing demand the other hand computer components like the latest hard disks of IBM can be controlled by a parameters for the acoustic load management. The trade-off between high data access disk i/o performance and low noise emanation (quiet seek mode) can be network i/o adjusted to meet on the requirements of the application context of this computer component in the work environment [10]. The consequences of the disappearing sounds are tow-sided: In a Figure 2: The input event is processed by an application: time of constantly increasing noise pollution in the environment In contrast to the direct call of the sound generation [11] it is welcomed that noise can be dramatically reduced in the unit, the sonification mediated by the performance nearest surrounding of a computer workplace. But the lost of the system is indirect and involves the characteristics of the acoustic aura of computer systems can also lead to irritation and underlying hardware system. uncertainty of the users during the interaction with the computer. In the context of product design for home appliances Heleen The real computer system acknowledges the save-command with Engelen mentioned that the sound of devices should be designed a transient noise burst that is caused by the positioning of the to be more pleasent to the users than they currently are [12]. She write head of the spinning hard disk during the save operation. states that „.by taking away the sound completely, the products This sound functions as a distinct hint for the users that really lost a part of their character. They lost a part of their

DAFX-228 Proc. of the 5th Int. Conference on Digital Audio Effects (DAFX-02), Hamburg, Germany, September 26-28, 2002 communication to the consumer, which can, when not solved in 3.3. Performance parameters another way, cause problems”. To achieve the most realistic sound feedback for the performance states of a computer system, a low latency of the response of the 3.2. The reappearing sounds parameters is important. For this case, the performance data are In the same time in which the computer devices are getting more retrieved via kernel functions of the Windows and more invisible or ubiquitous [13] also the audible properties and mapped to the sound generation unit that allows sound and characteristics are getting lost or vanish. In some extreme synthesis in real-time. The currently chosen performance items cases there could be even no more a physical reason for the represent the most interesting data items of the list of available production and emanation of a sound. In the author’s institute, items of the performance system under : examples of noise-free computer systems for the integration into future office environments [9] were built (second generation of • Hard disk i/o in Bytes per second the CommChairs and ConnectTables [14]) and allowed first • CPU load in percent experiences with noise-free computer systems. • Network i/o in Bytes per second Usually, the user ignores sound artifacts and understands them as waste, i.e., they are accepted as side products of the Other parameters like number of threads and mutex variables of a primary function or activity of the computer device. From the process can be also accessed by the programming interface but history of technology, however, there are examples that illustrate are less interesting for this proposed sound feedback. the persistence of sounds even when the physical reason does not The real sound emission of hard disks is a composite of a exist anymore, because of certain improvements in technology. continuous part, due to the rotation of the drive, and a more One example is the feedback of the dialed numbers on telephone transient part, e.g. created by data access operations. The CPU devices with noise bursts due to the mechanical operation and the load represented here in percent is the sum of the demand of all cross-talk between the sending and receiving channel. For these running processes. Usually this parameter is not audible to the sounds there is no more technical reason anymore because of the user in real computer environment, expect only indirect via an improved digital technology in the area of telecommunication index (for semiotic classification see [1],[15]), like the increasing devices. The other example is the sound of the indicator in cars speed of the rotation of the cooling fan of a . The number that originally was the sound of the mechanical relay that of bytes sent to or received by the computer is also a parameter alternate between the on and off state of the indicator lights. that has no correspondence within the real computer Although, the mechanical relay is replaced by silent transistor environment, but it has certain correlation to other parameters, technology the useful sound feedback still persists due to e.g., when downloading a file from the network to the local hard additional sound generation. These examples underline the disk. importance of the sound feedback that emerge in a situation, The programming infrastructure to access the parameter of where the technology development introduce new and more the computer system via the Windows operating system silent way of operation during the interaction. (Performance Data Helper, PDH) allows also to retrieve the The proposed reappearing sounds for computer systems can states of remote machines by simply modification of the be understood and realized as a class of Auditory Icons, when the performance query string. So non-standard sonification beyond typical computer noise is imitated, or as a class of parameter real acoustical behavior of computer systems is possible so that mapping methods, where a one-to-one mapping between the the sound of a remote computer system, e.g. a file server could be points in the data and sound representation space exists [15]. Bill transmitted and imitated on a local one. Gaver used Auditory Icons in his ‘SonicFinder’ to support the user’s interaction of the ‘desktop’ of the computer screen with sound of the ‘everyday listening’ (e.g., deleting a file in the trash 3.4. Related Work can) [2]. In contrast to this direct sound feedback of the user’s This work on the silent computer system is stimulated from interaction on the computer screen, like scraping sounds during different disciples and research areas, like computer science, dragging object over the desktop surface an analog sound human-computer interaction (HCI) and processing. feedback for the silent computer system is more indirect and In 1992 Jonathan Cohen presented in the ICAD community emphasis the reaction of a ‘system’ to the user’s input. (international conference on auditory displays) the ‘ShareMon’ The original ambient character of the sound feedback should application that uses the audio channel as an alternative modality be retained when the sounds are the synthetically added and the to convey to the user the states of the shared network resources offer to the perception should not attract the attention of the user of his computer [17]. Cohen used Auditory Icons like sounds of constantly and so does not become unpleasant for the ‘knocking on a door’, ‘foot steps’ and a ‘door slams’ to map the user/listener. Another possible feedback for the interaction with a remote logon and logoff procedure and the amount of file sharing hybrid artifact was demonstrated in [16]. The ‘silent’ interaction activity on a Macintosh computer. His work was motivated by with stylus on a computer graphic tablet was acoustically the limited visual display space on computer screens and the idea modeled and the audible qualities of the surface textures trigger a to convey background information about the computer via the cross-modal perception of different areas of interaction. audio medium. Another work by the artist Natalie Jeremijenko in 1995 at the computer science laboratory of Xerox PARC illustrates a more ambient and also hybrid approach to display the local area network activity. The so-call ‘dangling string’ was a plastic cable that hangs from a small electric motor from the

DAFX-229 Proc. of the 5th Int. Conference on Digital Audio Effects (DAFX-02), Hamburg, Germany, September 26-28, 2002 ceiling. Each bit of information causes a tiny twitch of the motor, metaphor of a audio device is used to convey the parameter via so that high network traffic made the cable twisting so fast that it modulating an audio stream. got finally also audible [7]. In 1997 Ishii and Ullmer used also sound in an architectural space called ambientROOM to convey 4.2. Analog representation the hits on certain web pages on an internet server by mapping these events to single rain drops. The sound of a heavy rain For the analog representation a generator is indicates that the web pages is well visited and attracts the users modulated with a low frequency rectangular oscillator which attention [18]. The work of Beth Mynatt at Xerox PARC from speed is controlled by the percentage of the CPU load. Idle states 1998 is called Audio Aura and represents a personal mobile of the CPU will sound like smooth swelling waves of water on audio information system that delivers to the user peripheral the beach. Therefore an additional low pass filter reduces ‘sharp information about arrival of e-mails, the presence of colleagues edges’ of the modulating rectangular function in the idle and their activity in the computer environment. These abstract or situation. A rising load of the CPU will increase the speed of the more high-level parameters of interaction with a computer LFO so that finally the sound generator resembles to the system serve as events to build a ‘sonic ecology’ for each user. rhythmic noise of a car engine. This sound will then be filtered The different events are mapped into an auditory scene, e.g., an by a recursive comb filter structure with a fixed resonance incoming e-mail is presented by seagull cries and various beach frequency. The amount of network traffic is mapped to the gain bird depending of the particular sender and the overall group of the feedback coefficient of the comb filter structure so that activity are mapped to surf and volume and from hence a sonic higher activity on the network interface becomes audible by more background [19]. harmonic filtering of the ‘CPU load’ sound (subtractive synthesis). While the CPU load parameter is more perceptible in the time domain, the network traffic is represented the frequency 4. SOUND PRESENTATION domain as a fixed resonance with variable gain. While the overall The presentation of sounds and sound effects for a silent sound is not formed by an additive synthesis process of different computer system build upon standard audio facilities of sound generators the resulting sound of this generator appears traditional computer system. To achieve a maximum of more as one single sound object. coherence in the perception of the location of the ‘object that sounds’ and the location of the real sound emanation the 4.3. Metaphor representation should be as close as possible to the silent computer. The best solution would be to integrate the loudspeaker into the To convey the parameters of the CPU load and disk i/o activity to hybrid artifact. The alternative of a virtual sound projection with the user of the silent computer the metaphor of traditional record multiple always would lead to a particular optimal player is used. Therefore it is assumed that the computer user is area of perception (sweet spot). This is only suitable for a static listening to musical streams which then are manipulated by the sound presentation like usually used in multi-media computer performance parameters of the computer. The parameter of the environments but not for a hybrid environment as described in CPU load is used to control the speed of the ‘turntable’ system section 2. so that the idle state is equivalent to the normal speed, while increasing load of the processes on the CPU leads to a decreasing of the rotation speed of the turntable. Only a few percent in the 4.1. Sound generation decrease of the speed are needed to be perceptible. The other The generation of sounds for the inherent states of the computer parameter of the disk i/o performance affects the sound stream by system can follow different concepts and can vary between an additional artifacts in the playback of the ‘record’. Small analog and more abstract or symbolic auditory representation of asperities on the record surface produce artifacts during the the parameter from the performance system [20]. The goal of the sampling of the record with a needle. This cracking analog sound generation would be to imitate the noise that is influence is added to the sound stream to convey the read and usually produced by the computer. These sounds are well-known write activity on the hard disk of the computer. Other standard by the users and are part of their ‘everyday listening’ in the sound effects can be used to convey this performance parameter workplace environment. More abstract and symbolic sound like chorus, reverb or distortion effects [23][22]. generation would use musical sequences, auditory scenes or effect on audio streams, that do not have any more a direct 5. IMPLEMENTATION relation between the parameter of the computer system and the resulting sound. By using a metaphor an additional sign referent In a first version, a program was created that combines the can create a link between the parameter of the computer system monitoring of the performance states of a computer and the and the sound or that is presented to the user (for sound generation. The programming interface (performance data detailed descriptions see [21]). Like the desktop metaphor for the helper, PDH) of Windows 2000 was used to get access to the arrangement of graphical objects on the computer screen an performance parameter. The resulting program was designed to auditory metaphor of a sounding device or an auditory scene can run in the background on the system tray. The sound generation support the user’s interaction and his mental model of the system unit was integrated into the program to have as much as possible activity. For the silent computer system two approaches are control of the parameter mapping and direct effectiveness in real- developed that follow on one hand a more analog representation time. The PortAudio application programming interface was and generation of sound for the inherent computer states with used to have access and control of the audio stream i/o in the certain allusion to real noises. While on the other hand a computer environment [24]. A set of signal processing objects

DAFX-230 Proc. of the 5th Int. Conference on Digital Audio Effects (DAFX-02), Hamburg, Germany, September 26-28, 2002 were written ANSI-C and used to build the basic filter operations [8] Tandler P., Prante, Th., Müller-Tomfelde, C., Streitz, N., and signal manipulations for the sound generation and effects. As Steinmetz, R., ConnecTables: Dynamic Coupling of nearly all programming environments and libraries this approach Displays for the Flexible Creation of Shared Workspaces. In of signal processing objects also uses block-based signal Proceedings of the 14th Annual ACM Symposium on User processing. Well-known signal processing algorithms were Interface Software and Technology (UIST'01), Orlando, implemented and applied to build a real-time controllable sound Florida, USA, November 11-14, 2001, CHI Letters 3(2), object and sound effects [25]. While including network based ACM Press, pp. 11-20. communication into the program it would be possible to remotely [9] Streitz, N., Tandler, P., Müller-Tomfelde, C., Konomi, S., involve a more flexible and powerful sound processing Roomware: Towards the Next Generation of Human- environments, like e.g. Max/MSP [26] for the initial design Computer Interaction based on an Integrated Design of Real phase of the sounds effects. The problem of the self-reference of and Virtual Worlds, In: J. A. Carroll (Ed.): Human- the sound generation process has to be solved in future. The Computer Interaction in the New Millennium, Addison required processing capacity the sound generation has to be Wesley, 2001. pp. 553-578. excluded otherwise the generation process produces sound that [10] IBM Feature Tool, http://service.boulder.ibm.com/ represents its own activity on the CPU. Some mappings of the storage/hddtech/ibmftool_userguide.pdf parameter are extremely depended on the hardware configuration [11] Wrightson, K., An Introduction to Acoustic Ecology. of the computer system and have to by adjusted individually. So Soundscape, Volume 1, Number 1, Spring 2000, pp. 12-13. for example the network traffic rate can vary from 1Mbps up to [12] Engelen, H., Sound Design for Consumer Electronics. In 100Mbps. Proceedings "Stockholm, Hey Listen!" Conference on Acoustic Ecology, Stockholm, June 9-13, 1998. [13] Weiser, M., Designing Calm Technology, PowerGrid 6. CONCLUSION Journal, v101, 1996, http://powergrid.electriciti.com/1.01. The described sound effects for a silent computer system are [14] Müller-Tomfelde, C. and Reischl W., Communication initial research activities in the context of future and emerging Chairs: Examples of Mobile Roomware Components, In: computer systems that will produce no more a noise at all. The ACM Conference on Human Factors in Computing Systems disappearing sounds allow to make those sounds reappearing that (CHI ’98), Los Angeles, California, USA, April 18-23, are useful and important for the interaction with hybrid object in 1998. CHI-Summary. pp. 267-268. a hybrid environments. An analog representation and the use of a [15] Barrass, S., Kramer, G., Using sonification. record player metaphor are used to demonstrate the variety of Systems, Special Issue on Audio and Multimedia, 7(1): 23– possibilities for a reappearing sound aura of a silent computer 31, 1999. system. I would like to thank Steffen Halma for his programming [16] Müller-Tomfelde, C. and Münch, T., Modeling and work on the performance parameter and the integration of the all Sonifying Pen Strokes on Surfaces, In Proceedings of the source code into one program. COST G-6 Conference on Digital Audio Effects (DAFX-01), Limerick, Ireland, December 6-8, 2001. [17] Cohen, J, Monitoring background activities, In: G. Kramer 7. REFERENCES (Ed.), Auditory display, sonification, audification and auditory interfaces. Proceedings of the 1th ICAD´92, Santa [1] Blattner, M. M., Sumikawa, D. A. and Greenberg, R. M., Fe Institute, Addison-Wesley, 1994, pp. 499-531. Earcons and Icons: Their Structure and Common Design [18] Ishii, H, Ullmer , B., Tangible Bits: Towards seamless Principles, In Human Computer Interaction, 4(1), 1989, pp. Interface between People, Bits and Atoms In Proceedings of 11-44. the ACM Conference on Human Factors in Computing [2] Gaver, W., The SonicFinder: An Interface That Uses Systems (CHI ‘97) pp. 234 – 241. Auditory Icons, In Human Computer Interaction, 4(1), [19] Mynatt, E.D., Back, M., Want, R., Baer, M., and Ellis, J. B. 1989, pp. 67-94. 1998. Designing audio aura. In Proceedings of the ACM [3] Gaver, W., Synthesizing Auditory Icons. In Proceedings of Conference on Human Factors in Computing Systems (CHI the InterCHI ’93,24-29 April 1993, pages 228-235. ’98), Los Angeles, CA, Apr. 18–23, pp. 566–573. [4] Müller-Tomfelde, C. and Steiner, S., Audio-Enhanced [20] Kramer, G. An Introduction to Auditory Display. In: G. Collaboration at an Interactive Electronic Whiteboard. In Kramer (Ed.), Auditory display, sonification, audification Proceedings. of 7th International Conference on Auditory and auditory interfaces. Proceedings of the 1th ICAD´92, Display, ICAD, Espoo, 2001, pp. 267-271. Santa Fe Institute, Addison-Wesley, 1994, pp. 1-77. [5] Milgram, P. and Kishino, F., A Taxonomy of Mixed Reality [21] Familant, M.E. and Detweiler, M.C., (1993). Iconic Visual Displays, IEICE Transactions on Information Reference: Evolving Perspectives and an Organising Systems, Vol E77-D (12), Dec. 1994. Framework, International Journal of Man-Machine Studies, [6] Eckel, G., Application for the CyberStage Spatial Sound 39, 705-728. Server, In Proceedings of the AES 16th International [22] DAFX software, http://echo.gaps.ssr.upm.es/COSTG6/ Conference: Spatial Sound Reproduction, Rovaniemi, [23] Zölzer, U., Digital , J. Wiley & Finland, 1999 April 10-12, pp. 478-484. Sons, Chichester 1997. [7] Weiser, M., The computer for the twenty-first century. [24] PortAudio, http://www.portaudio.com/ Scientific American, Vol. 265, No. 3, pp. 94-104, 1991. [25] Roads, C., The Computer Music Tutorial, 3. Edition, The MIT Press, 1996.

DAFX-231 Proc. of the 5th Int. Conference on Digital Audio Effects (DAFX-02), Hamburg, Germany, September 26-28, 2002

[26] Max/MSP, Cycling 74, http://www.cycling74.com.

DAFX-232

Top View