Playing a Game with Supplemental Modalities

University of Nevada, Reno

Playing A Game With Supplemental Modalities

A thesis submitted in partial fulﬁllment of the requirements for the degree of Master of Science in Computer Science and Engineering

Daniel Ramos

Eelke Folmer, Ph. D./Thesis Advisor

THE GRADUATE SCHOOL

We recommend that the thesis prepared under our supervision by

DANIEL RAMOS

entitled

Playing A Game With Supplemental Modalities

be accepted in partial fulfillment of the requirements for the degree of

MASTER OF SCIENCE

Eelke Folmer, Ph. D., Advisor

Sergiu Dascalu, Ph. D., Committee Member

Jeffrey Hutsler, Ph.D., Graduate School Representative

Marsha H. Read, Ph. D., Associate Dean, Graduate School

May, 2011

Abstract

Video games are presented to a user primarily using visual feedback. Although many games contain sound and haptic feedback, these modalities are usually secondary in nature. Most secondary stimuli are used to enhance the player’s experience and do not contain information that will tell the user what input to provide and when. Therefore, a person that has an impeded ability to perceive the primary stimuli, whether it be because of limitations of the input device (e.g. a mobile phone), a temporary sensory impairment (e.g. a noisy room), or a permanent impairment (e.g. blindness), will potentially be at a disadvantage when trying to play video games. Previous studies have shown that supplemental forms of feedback can improve the usability of software and games, and cues represented in multiple, simultaneous modalities can be detected faster, more accurately, and at lower thresholds than when presented separately. This thesis presents user studies on the use of supplemental feedback to reduce errors in playing Bingo, a game which is typically played by an older demographic who are more likely to suﬀer from sensory impairments such as low vision or hearing. ii

Contents

Abstract i

List of Figures iii

List of Tables iv

1 Introduction 1 1.1 Overview ...... 1 1.2 Interaction Model for Games ...... 2 1.3 Sensory Substitution and Alternate Modalities ...... 3 1.4 Research Questions ...... 4 1.5 Why Bingo? ...... 5 1.6 Research Methods ...... 5 1.7 Thesis Structure ...... 7

2 Supplemental Soniﬁcation of a Bingo Game 8

3 An Analysis of Multimodal Representation of a Bingo Game 15

4 Conclusion 27 4.1 Results ...... 27 4.2 Fun and Fairness ...... 27 4.3 Future Work ...... 28 4.4 Applications ...... 28

Bibliography 29 iii

List of Figures

1.1 Interaction model for games...... 2 1.2 Iterative user studies...... 6

2.1 An electronic dauber for playing Bingo...... 8 2.2 A screen shot of the Bingo Simulator...... 10 2.3 The away count test for each card...... 11 2.4 Spectrograms for the four audio icons...... 12

3.1 An electronic dauber for playing Bingo...... 17 3.2 A screen shot of the Bingo Simulator (version 1)...... 18 3.3 The away count test for each card (version 1)...... 19 3.4 Spectrograms for the four audio icons...... 20 3.5 Tacton patterns...... 22 iv

List of Tables

2.1 Results of the Bingo simulation...... 12

3.1 Results of phase 1 Bingo simulation...... 21 3.2 Combined results from phase 1 & 2 Bingo simulation...... 23 1

Chapter 1

Introduction

1.1 Overview

In today’s world there is no arguing that computers are essential to everyday life. A recent example is the rise in popularity of smartphones. A growing number of people are utilizing mobile devices, like smartphones and tablets, for their computing needs. Mobile Internet devices outsold PCs by 150 million units in 2009 [1] and show no sign of slowing down.

With more than 76% of the U.S. population [8] using computers and the Internet, it is critical that software developers ensure that the applications running on those devices are accessible to everyone.

Software is usually created for two primary purposes: (1) to relieve a user from a task by automation or (2) to provide functionality that supports a user in performing a task

[11]. Increasingly, a third area of electronic entertainment has been gaining popularity. In the past few decades, video games have become a more mainstream pastime.

Games are increasingly attractive to people because unlike books, movies, and music, a game requires interaction, and playing multiple times can produce diﬀerent results each time. According to the Entertainment Software Association, 67% of homes in America own either a console and/or PC used to run entertainment software. 42% of heads of households report they play games on wireless devices such as a cell phone or PDA. Video games have become a large business, generating 10.5 billion dollars in sales in 2009 alone [6].

In contrast, entertainment is not the sole objective of video games. Serious games are 2

Figure 1.1: Interaction model for games. designed to communicate information or create a learning experience for the player. For instance, the National Oceanic and Atmospheric Administration (NOAA), along with many other organizations, use the virtual world of Second Life to provide learning opportunities and discussions about science and technology [2].

Video games used for educational purposes are often very eﬀective because rather than explain a concept, a user learns through action. Games can be designed to simulate a real life situation which the user can experience and learn from in a controlled manner (i.e. provide goals, give the user resources to accomplish the goals, and then provide feedback)

[13].

1.2 Interaction Model for Games

If we examine how a user interacts with a game, we can see it is a feedback loop in which the game provides one or more stimuli to the user which, in turn, causes the user to provide input to the game (see Figure 1.1). This loop continues until the goal is attained or the game ends. An example would be in a fighting game when the player is face to face with an enemy. He or she can choose to fight, defend, or flee. In order to break down and analyze how people play games, Yuan et al. [14] created a list of steps common to most games. 3

This game interaction model consists of:

Receive stimuli: The game provides stimuli in one or more diﬀerent forms which • can be categorized as “primary” or “secondary”. Primary stimuli are what the user

needs to perceive in order to accomplish goals in the game. Most games use visual

feedback as the primary stimulus. Secondary stimuli are not required to play the

game, but if the player is not able to perceive these stimuli, he or she may suﬀer from

a reduced gaming experience.

Determine response: Based on the stimuli perceived by the player, he or she • determines the appropriate responses to accomplish the goal or task.

Provide input: After determining the response to the game’s stimuli, the user must • physically perform some action through an input device to interact with the game.

Typically, a video game involves interaction with a user interface that generates, primarily, visual feedback on some device. Games also use other modalities, such as auditory and haptic feedback, but these alternate modalities are rarely used as a means to indicate to the user as to what he or she should do and when. Regardless of how a game or other software communicates, a user with an impeded ability to perceive the primary stimuli will have a more diﬃcult time accessing and using it. This could include a user with a temporary impairment, such as trying to play a game in a noisy environment or in a dark room. Handheld devices, such as smartphones or portable gaming units, also can impose impairments for the sake of portability. An example would be a smartphone’s reduced screen size or lower powered speakers. Finally, users with permanent impairments could be reluctant to use software if they are unable to perceive the primary stimuli.

1.3 Sensory Substitution and Alternate Modalities

Sensory substitution [7] is deﬁned as transforming the characteristics of one sensory modality into stimuli of another sensory modality. Humans typically use three of their senses when interacting with software: 4

Sight: Interpreting information from the eﬀects of visible light reaching the eye. • The human eye has a large spatial resolution and is capable of visualizing millions of

diﬀerent colors. The eye also has a high temporal resolution, meaning humans can

detect changes in lighting with a duration lower than 100 milliseconds [4].

Hearing: Interpreting information from the eﬀects of air vibrations reaching the • ear. The frequency range of human hearing is 16 Hz to 20 kHz. Locating a sound

can be diﬃcult if both ears receive the same signal at the same time (i.e. when a

sound originates from directly in front, directly behind, or directly above) [3]. Thus,

compared to the eye, a human can see better than he or she can hear.

Touch: Interpreting information from the eﬀects of pressure on the skin. Although • humans have a large hearing frequency range, the range of frequencies that can be

detected by the skin is much lower, typically 10Hz to 400Hz [10]. Despite the more

limited frequency range, humans can detect consecutive tactile stimuli as fast as 5

milliseconds apart [9], which is much faster than the duration noted above for sight.

The beneﬁts of sensory substitution by using alternate modalities helps people with impairments use software they would otherwise be unable to. This beneﬁt is not limited to people with disabilities. A cell phone is an example of using alternate modalities when the primary modality is not available or would be inappropriate. For instance, a cell phone can vibrate to notify a person of a call when having the phone ring would be distracting or rude (i.e. in a movie theater).

1.4 Research Questions

There have been many studies into making operating systems and other serious software easier to use for impaired persons, but the area of accessible games is still relatively new.

The focus of this thesis is to examine the usability of software related to playing the game of

Bingo. Usability is defined as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of 5 use [5]. Specifically, we investigate how supplemental forms of feedback (other than visual) will help a player reduce errors and improve efficiency when playing the game. Bingo is played by a wide demographic, but the most well known demographic are the elderly who are more likely to have visual or auditory impairments. Bingo can also potentially be played in noisy environments which might impair a person’s ability to play the game. We expect adding supplemental modes of feedback will make the game more accessible to people with impairments.

1.5 Why Bingo?

As games have evolved from Pong in the 1970s to the massively multiplayer games of today, they have rapidly grown in complexity. Pong was played with a paddle containing a single wheel. Today’s games are often played with full keyboards or controllers that have 12 or more buttons.

We chose Bingo because it is, at its core, a very simple game. At any moment in time a player can determine the “state” of his or her cards by examining the numbers called and comparing them to the numbers on the card. This state-based approach, with an element of randomness, is common in the world of gaming (e.g. in Tetris you can determine how you are doing by looking at how many lines you have on the screen, but the next piece to be played is decided at random). It is ﬁnite because eventually any card will win given enough numbers are called. It is controllable because the rate at which numbers are called is solely dependent on the person or device calling them. Finally, Bingo is fun to play and a very popular pastime. Charitable Bingo is played in over 30 states and provinces in North

American and earned over 1.8 billion dollars in 2009 [12].

1.6 Research Methods

Research methodology deﬁnes the activity of research, including the process, measuring progress, and how to determine whether that research was successful. To determine the usefulness of alternate modalities in a Bingo game we utilized user studies. We used us- 6

Figure 1.2: Iterative user studies. ability testing to gather data by recording users as they perform pre-deﬁned tasks with an application that was being evaluated. The process was iterative and consisted to two phases in which the data from the ﬁrst phase was evaluated and used as input for improvement for the second phase of study. The motivation for two phases was to eliminate some of the possible alternate modalities to keep the user studies feasible.

The sequence of our research consisted of four steps (see Figure 1.2):

1. Hypothesize: For the ﬁrst phase we chose a research area, examined existing work,

and proposed a theory. The second phase consisted of building upon the ﬁrst phase

and more research into existing work.

2. Design: We designed user studies and associated computer programs that would help

us determine if our theories were correct.

3. Test: We recruited subjects for our user studies and collected data.

4. Evaluate: We evaluated the data collected from the user studies in order to ﬁnd the

most eﬀective supplemental feedback, use that to reﬁne our theory, and restart the

process at step 1. 7

1.7 Thesis Structure

The body of this thesis consists of two articles. The articles are organized in chronological order (i.e. phase 1, then phase 2). Following the articles is a concluding chapter. These articles can be found in the following chapters:

Chapter 2 - Daniel Ramos, Eelke Folmer. Supplemental Soniﬁcation of a Bingo Game. • To appear in Foundations of Digital Games, Bordeaux, France, June 2011.

Chapter 3 - Daniel Ramos, Eelke Folmer. An Analysis of Multimodal Representation • of a Bingo Game. Submitted to the Journal of Entertainment Computing, April 2011. 8

Supplemental Soniﬁcation of a Bingo Game

Daniel Ramos Eelke Folmer GameTech International Player-Game-Interaction Lab 8850 Double Diamond Parkway University of Nevada, Reno Reno, NV 89521 1664 N. Virginia St., Reno, NV 89557 [email protected] [email protected]

ABSTRACT Visual cues are typically used in video games to indicate to the player what input to provide and when. Cues represented in multiple modalities that are presented simultaneously can be detected at lower thresholds, faster, and more accurately than when presented separately in each modality. This characteristic has not been explored in playing video games to reduce errors. This paper explores the use of supplemental audio feedback to reduce errors in playing Bingo, a game which is typically played in crowded and noisy environments by a demographic, which-due to their age-are more likely to suffer from sensory impairments such as low vision or hearing impairments. A user study explored Figure 1: An electronic dauber for playing three different types of sonification (pitch, timbre, Bingo. c 2011 GameTech International, Inc. All and audio icons) versus using no sonification and rights reserved. found that supplemental sonification using timbre or audio icons significantly reduces players’ errors. the result of a particular player action or are part of game, such as music, and they rarely contain cues Keywords that the player must interpret to decide what input Sonification, Multimodal Feedback, Games to provide. Lack of sight is therefore considered a greater barrier to playing a game than playing Categories and Subject Descriptors without audio or haptic feedback. H.5.2 [User Interfaces]: Auditory Feedback Sonification (also known as auditory display) is the process of displaying data in an audio format (other General Terms than traditional speech) [7]. Sonification may use Human Factors, Performance changes in pitch, volume, timbre, patterns or frequency to represent data. A Geiger counter is a 1. INTRODUCTION well-known example that uses the frequency of au- Video games rely, primarily, on visual feedback to dible clicks to indicate the level of radiation. Soni- indicate to the player what input to provide and fication has been pioneered in sensory substitution when [29]. Though games may provide sound and [5] to communicate information to users who are haptic feedback, these are typically used to indicate visually impaired [16, 26]. Although sonification is still considered in its infancy, it has become an emerging modality of information representation for able-bodied users. Various studies with users performing choice response tasks (i.e. providing a particular input based on a certain stimulus) have shown that a stimulus represented in multiple modalities simultaneously can be detected at lower thresholds, faster, and more accurately than when information FDG’11 June 29-July 1, 2011, Bordeaux, France is provided separately in each modality [14, 18]. This characteristic has been recently explored in the 9

domain of complex data representation and inter- Bingo can be traced back, in some form or another, pretation, where studies found that supplemental to Italy in the 1500s when it was played as a lottery sonification, e.g., in addition to visual representa- game [24]. In the United States, Bingo is mostly tion is beneficial [22, 25]. played as a charitable game and it is usually the domain of churches, youth organizations, and other Sonification has only been sparsely explored in the non-profits. Bingo is also played for profit by casi- domain of video games. Supplemental sonification nos and Native American Bingo halls. Despite its of cues, that indicate to the player what input to mostly charity status, Bingo is big business. The provide and when, could reduce errors in playing National Association of Fundraising Ticket Manu- games. In this paper we explore supplemental soni- facturers reports that in 2009 over 1.8 billion dollars fication in playing electronic Bingo. This paper is was spent on charitable Bingo in the United States organized as follows. Section 2 provides background [11], which dwarfs any current AAA video game ti- and Section 3 discusses related work. Section 4 dis- tle. Although Bingo is often assumed to be a game cusses the design of a simulator used in user studies that is only played by“little old ladies”, recent stud- that are discussed in Section 5. Section 6 and 7 dis- ies show the stereotype of the average Bingo player cuss the results and future work, and the paper is has been steadily changing. It is true that the ma- concluded in Section 8. jority of players are female (57%), but only 11% are age 65 or above [2]. In the UK, the proportion of players under 45 years of age has increased from 46 2. BACKGROUND percent to 62 percent in the last decade [3], where Bingo is a game of chance, where players compete players aged 18 to 24 currently make up 20% of the against one another for a prize. A player receives Bingo demographic. This growth is primarily at- one or more pre-printed cards that contain 24 num- tributed to the emergence of online Bingo, which bers (one of the squares is usually designated the currently makes up 20% of the UK market. “free space” and is not a number) arranged in a 5 x 5 matrix. The player must match his or her cards to Playing video games can be considered to be a form the numbers randomly drawn for the game. If the of performing choice response tasks [29], which are drawn numbers match a particular pattern on the perceptual-motor tasks that are typically used in player’s card, the player calls out the word “Bingo!” psychology user studies to study the content, du- to alert other players of a possible win. Typical pat- ration, and temporal sequencing of cognitive op- terns include a horizontal, vertical, or diagonal line, erations. When playing a video game, the player though there are many other patterns including re- provides input based on the feedback the game pro- quiring all the numbers to be matched up to cover vides, upon which the game may generate new feed- the entire card. Single games often have multiple back based on the players input, which may elicit Bingos; for example, the players first play for a sin- new responses from the player. For example, in a gle line; after that, play goes on until a full card is first person shooter, an approaching enemy elicits a covered. All wins are checked for accuracy before particular response from the player, such as shoot- the win is officially confirmed and a new game is ing or fleeing. begun. The game mechanisms of Bingo have been used in educational settings to teach math [9] or The mechanics of Bingo represent a simple choice biochemistry [28]. response task in a gaming context. This task consists of the player marking the called number, if it Players typically play with multiple cards to in- exists on the player’s card, and calling a Bingo if crease their chances of winning. To mark cards that card matches a particular pattern. It is in- faster the players usually use special markers called teresting to study the effectiveness of supplemental daubers. Cards used for playing Bingo are typically sonification in a game context as this adds time printed on paper but, increasingly, electronic ver- pressure and competition, which is different from sions of Bingo are played on portable devices called the contexts in which sonification has been stud- an electronic dauber (see Figure 1), which allows ied in the past. Supplemental sonification of how for increasing the number of cards the player can close a player is to achieving a Bingo may be use- play to near unlimited. In some jurisdictions, reg- ful in playing Bingo to avoid a “sleeper”, which oc- ulations call for players to insert the called number curs when the player does not realize he or she has into the electronic dauber upon which all cards will a winning card. This occurs occasionally because: be marked, whereas other jurisdictions allow au- (1) Bingo halls and casinos can be noisy environ- tomatic play. The electronic dauber shows a lim- ments where it can be hard to follow the game; (2) ited number of cards that are closest to achieving due to their specific demographic, players are more a Bingo and the electronic dauber will identify the likely to suffer from sensory impairments such as winning pattern and show “Bingo” on the screen, low vision or hearing impairments; and (3) play- but the player is still required to call the Bingo to ers play the game using a large number of cards, stop the game. which makes it more difficult to identify a winning 10

Figure 2: A screen shot of the Bingo Simulator. pattern. a dance game that is played using fingers and a keyboard where different audio icons indicate what There have been several studies to determine the input to provide. AudioQuake [4] is an FPS that best methods and techniques for representing in- uses earcons [8] to alert the player to an object or formation using audio for different types of appli- event. cations. Some methods include using pitch, volume, tempo, timbre, auditory icons [13], i.e, ab- We identified only one game that uses supplemental stract sounds used in operating systems to alert the sonification, e.g., sonification in addition to visual user, and earcons [8], e.g., structured sounds that representation of information. Terraformers [27] is follow musical conventions. Many video games use an FPS that can be played by players with visual real life sounds (explosions and footsteps) as audio impairments as well as able-bodied players. Ter- cues but this is not considered sonification (nor is raformers uses an audio compass using pitch, and a speech). sonar like mechanism is used for conveying how far objects are in front of the character. 3. RELATED WORK Sonification has primarily been explored in sensory 4. BINGO SIMULATOR substitution for creating audiogames that can be To evaluate what type of sonification works best played by players with visual impairments. An ex- for playing Bingo, we developed a simulator appli- tensive overview of audiogames can be found in cation that can simulate a standard Bingo game. Yuan [29]. Most audiogames use audio cues, but This application is a Windows program written in the following games use sonification. The puzzle C# using Microsoft’s .NET Framework and Win- game the Towers of Hanoi has been sonified [20] dows Presentation Foundation technologies. where timbre, pitch and frequency of a pulse signal are used to individuate the size and location In the simulator, each player gets three cards with of the discs. A sonified version of the boardgame randomly generated numbers in each square (see Go [23] exists where timbre is used to indicate black Figure 2). Even though most electronic daubers and white stones, and pitch indicates their distance can support a large number of cards, only a small with respect to the center of the board. Speed Sonic number of cards are usually displayed on the screen Across the Span [21] is a platform game where ob- (those that are closest to a Bingo). The motivation jects, such as a platform or animals, emanate differ- for the use of three cards is players most commonly ent audio cues, and changes in pitch and frequency play with that many cards (See Figure 1), but elec- are used to convey the distance and height of these tronic daubers can render up to 24 cards simulta- objects. Metris [15] is a musical version of Tetris neously. The pattern needed to achieve Bingo is where pitch, frequency, and length of an audio cue a straight line either with all called numbers in a are used to indicate the type and orientation of row, column, or diagonal. When the player presses the different puzzle pieces. Finger Dance [17] is the “Start Ball Calls” button the application will 11

start drawing random numbers from 1 to 75 every dle C note to represent a certain away count. six seconds and display them in a grid below the The instruments used were a piano, cello, or- cards. Six seconds is a common interval to allow gan, and pan flute. paper players enough time to daub their cards. The Audio Icons: One of four completely differ- last ball called and sample patterns are displayed • next to the grid. To mark a called number on his ent audio cues is played to represent a cer- or her card, the player must type that number with tain away count. We used the following au- the keyboard and press enter or click the “Daub” dio icons: a dog barking, a jackhammer, a button. Each card will then be daubed in sequence cash register bell ring, and an audience clap- from left to right, and a green background will be ping. The cash register and audience clapping rendered behind the daubed number (see Figure 2). were used for low away counts to signal that If the user gets five daubs in a straight line, he or a Bingo was close to happening. she wins and the game restarts. Volume was also considered, but previous sonifica- 4.1 Sonification Strategies tion studies found [25, 30] volume to be inferior to In Bingo, the number of daubs needed to achieve pitch and timbre. Sonification types such as in- a Bingo is known as the “away count”, e.g., if the creasing the frequency between audio cues are not player requires two daubs to make a five number applicable since we use discrete sonification, e.g. we straight line, then the player’s away count is two briefly play one sound to convey the value of the and hence, for straight line patterns on a 5x5 card, variable as opposed to a continuous representation. the away count varies between 1 and 4. We seek to sonify the away count for each card. Although soni- All sounds were selected from open source sound fication is generally useful for enhancing program libraries, and the pitch shifts of were created using usage, there are some potential pitfalls that could Audacity [1] audio software where pitch was itera- cause problems. This is more likely to occur when tively increased approximately 150% for each cue. sonifying multiple variables and mapping them into For both timbre and audio icons, we selected instru- one audio stream. Neuhoff, Kramer, and Weyand ments and audio icons that sounded as different as [19] performed a study to investigate the interac- possible from each other. Figure 4 shows the re- tion between pitch and volume sonification meth- spective spectrograms for the used audio icons that ods. They showed that pitch and volume interact reveals their differences. All sounds had a length of to mask the actual data being sonified. Specifically, approximately 1 second. rising intensity sounds were perceived to change more than equivalent falling intensity sounds. In Each time a number is daubed on the player’s cards, our study we will avoid this problem because we the away count may change. We only sonify changes will sequentially sonify variables using one type of in the away count, rather than the current value for sonification at a time using the “Ears-Lead-Eyes” the away count for each card, as this could poten- [6] sonification design pattern. tially overwhelm the player with audio feedback, which may be detrimental to the their gameplay Sonification solutions have been cataloged using a experience. If a card’s away count is changed, the technique known as “design patterns”, which is a sound corresponding to that away count is played concept that has been pioneered in describing ob- while the card is highlighted. Figure 2 shows that ject oriented design solutions [12], but patterns col- the away count for the middle card changes and lections for other domains exist, such as interaction that this card is highlighted while sonified. In the design for video games [10]. The Ears-Lead-Eyes case where all away counts for all cards change, the design pattern involves designing a sonification so- sounds are played back to back in the six-second lution for when users need to be notified of intermit- interval between ball calls. tent, but critical, information while they are busy with another task using their eyes. This is helpful because it reduces the need to keep switching visual attention between different areas in an application or keep the user from potentially missing important information. For our study, we explore the following three types of sonification:

Pitch: An audio cue is played at diﬀerent • pitches (99Hz, 119Hz, 156Hz, and 193Hz) to represent the away count. A lower frequency indicates a low away count. Figure 3: The away count test for each card. Timbre: A diﬀerent instrument plays a mid- • 12

4.2 Measures While playing a game, players are randomly tested. Table 1: Results of the Bingo simulation. The game is paused and a screen that blocks the Type Error σ Time (ms) σ (ms) view on the cards pops up upon which the player NONE 0.241 0.19 16,554 22,677 is asked to provide the away count on each of their PITCH 0.105 0.14 9,879 5,672 cards (see Figure 3). Players type in the away count TIMBRE 0.074 0.11 9,149 8,510 using the keyboard. AUDIO 0.059 0.06 6,954 1,815 ICON The idea behind testing for away count is that, although our hypothesis is that sonification may help in good health, with no hearing or vision impair- avoid a sleeper, it is difficult to test for a sleeper as ments (some had corrected vision) that could pos- that can only be observed over a large number of sibly impede their ability to play the game. Each games. However, it can be assumed that the more user was given a brief explanation and walk-through aware the player is of the current away count for of the application, a demonstration of the different each card, the less likely a sleeper will occur. sounds, and subjects were able to play a number of games using different sonification techniques. After the player is tested on each method, the application ends and the results are recorded to a log file. Subjects were tested alone in a room using a lap- We record for each game (using one unique type of top. There were no other background noises and sonification): (1) total playing time; (2) number of the laptop’s built-in speakers (4Ω, 2 Watt, stereo) away count prompts; (3) time it took the player to were used for sonification, which was motivated by answer the away count prompts; (4) away count; that electronic daubers systems often have similar and (5) player provided away count for each card. speakers. The effectiveness of the three different types of sonification (pitch, timbre, audio icons) as well as using no sonification was evaluated using a within-subjects design. The users played through four games, each using one of the four sonification methods, in random order. The game would end if the user achieved a Bingo or was tested a total of four times, which ever came first. On average it takes 24 ball calls to achieve a Bingo, and we as- signed the random test as such that there were, on average, 6 ball calls between each test.

After the tests, the subjects ﬁlled in a brief questionnaire where they were asked which soniﬁcation method they felt helped them the most in determining the cards’ away counts.

6. RESULTS Performance data were gathered from the logs and Table 1 lists the results from our study. Our analysis focuses on two variables: (1) error rate (second column) which is the ratio between incorrectly answered away counts and total away counts; and (2) time to complete away count prompts (fourth column). The third and the fifth columns list the found standard deviations. Figure 4: Spectrograms for the four audio icons. For all tests, α was set at 0.05. When using an ANOVA with repeated measures with a Greenhouse- 5. USER STUDY Geisser correction, the mean errors were statisti- The Bingo Simulator application was evaluated with cally significantly different (F 1.574,24 = 4.67 p < nine adult volunteers (2 female, age M = 41.2 SD 0.05). Post-hoc tests revealed that using no soni- = 12.8). Subjects were recruited from a local gam- fication yields a significantly larger number of er- ing company that specializes in developing Bingo rors than timbre (p = 0.047) and audio icons (p = systems, and, as such, all subjects could be con- 0.034) but no significant difference was found be- sidered expert Bingo players. We opted for testing tween pitch and using no sonification (p = 0.085). this with expert Bingo players to avoid having to No significant difference was found between timbre accommodate a learning phase. All subjects were and audio icons (p = 0.521). Due to our small 13

sample size, no corrections were applied. Using best modality to use for multimodal representation. the same ANOVA, the mean values for time were The newest electronic daubers also allow for pro- not found to be statistically significantly different viding vibrotactile feedback as they contain one or (F 1.049,24 = 1.036 p > 0.05). more vibrotactors. Future research will focus on ex- ploring the efficacy of supplemental representation Results from our questionnaire revealed that all using sonification versus haptification (vibrotactile subjects felt that supplemental sonification –in general– feedback) as well as their combined effects. improved their ability to play Bingo. Out of the nine participants five preferred audio icons; two preferred the use of pitch; one preferred timbre; and 8. CONCLUSION In this paper we explore the supplemental sonifi- one user preferred timbre as well as pitch. cation of information required to play the game of Bingo. Bingo halls and casino environments are 7. DISCUSSION AND FUTURE WORK often crowded and potentially noisy, and Bingo is The results of our user study show that supple- typically played by players who-due to their age-are mental sonification of visual information increases more likely to suffer from visual or hearing impair- performance in performing choice response tasks ments. Supplemental sonification may be useful for that rely upon interpreting this visual information. Bingo to avoid players not realizing they have won. Our findings could potentially affect other types A user study with nine subjects assessed the perfor- of games. For example, puzzle games like Bejew- mance of three different types of sonification (pitch, eled or Drop 7 often use visual indicators, such as timbre, and audio icons) as well as using no sonifi- a progress bar, that indicates how far the player cation. Results show that sonification using timbre is in the current level before moving to the next. and audio icons significantly reduces player errors. Such values could be sonified using the Ears-Leads- Sonification could be used for other types of games, Eyes pattern and discrete audio icons could be used such as puzzle games, to reduce player errors. to indicate different levels, which could make the player more alert of an upcoming level transition, and which may increase player’s performance. References [1] Audacity: Free audio editor and recorder, Sonification can already be facilitated on most elec- http://audacity.sourceforge.net, access- tronic daubers. Though in some jurisdictions no date: 2-3-2011. sounds may be played while playing Bingo, many electronic daubers have a headphone jack where [2] Everything you need to know about bingo and sonification could be facilitated using a headset, pull tabs, http://www.arrowinternational. which would allow the player to hear the sonifi- com/images/docs/allaboutbingo.pdf, access cation but would also leave one ear free for listen- date: 1-18-2011. Tech. rep., Arrow Interna- ing to the called numbers. One could also argue tional, Inc., 2007. that using supplemental feedback, in addition to [3] Bingo - uk - april 2009, http://oxygen. visual feedback, gives a user an unfair advantage mintel.com/sinatra/oxygen/display/id= over those players who only use one form of stim- 394689. Tech. rep., Mintel International uli. Complaints could also arise stating that sonifi- Group Ltd., 2009. cation produces noises that could annoy or discourage neighbors. [4] Atkinson, M. T., Gucukoglu, S., Machin, C. H. C., and Lawrence, A. E. Making the Audio icons and timbre gave the best results with mainstream accessible: redefining the game. regard to error. The semiotics of what audio icon to In sandbox ’06: Proceedings of the 2006 ACM use for representing what value could significantly SIGGRAPH symposium on Videogames (New affect performance and error. For example, we used York, NY, USA, 2006), ACM, pp. 21–28. the cash register bell and applause for low values of the away count, sounds which the player is more [5] Bach-y Rita, P., and W Kercel, S. Sen- likely to associate with winning. Had we used dif- sory substitution and the human-machine in- ferent sounds, maybe players would not have made terface. Trends in Cognitive Sciences 7, 12 these associations and we may have observed dif- (Dec 2003), 541–6. ferent performances. [6] Barrass, S. Sonification design patterns. In Bingo is typically played in crowded and poten- Proceedings of the 2003 International Confer- tially noisy environments and a key demographic ence on Auditory Display, pp. 170–175. that plays bingo is more likely to suffer from sensory impairments. Though multimodal representa- [7] Barrass, S., and Kramer, G. Using sonifi- tion using supplemental sonification can be useful cation. Multimedia Sysems. 7 (January 1999), in these contexts, audio feedback may not be the 23–31. 14

[8] Brewster, S., Wright, P., and Edwards, [20] olof Hellstrom, S., and Winberg, F. A. Experimentally derived guidelines for the The quest for auditory direct manipulation:the creation of earcons. In Adjunct Proceedings of sonified towers of hanoi. In In Proceedings of HCI (1995), vol. 95, Citeseer, pp. 155–159. the 3rd International Conference on Disabil- ity, Virtual Reality and Associated Technolo- [9] Cheng, H. N., Deng, Y.-C., Chang, S.-B., gies (2000), pp. 75–81. and Chan, T.-W. Edubingo: Design of multi- level challenges of a digital classroom game. [21] Oren, M. A. Speed sonic across the span: Digital Game and Intelligent Toy Enhanced building a platform audio game. In CHI ’07: Learning, IEEE International Workshop on 0 CHI ’07 extended abstracts on Human factors (2007), 11–18. in computing systems (New York, NY, USA, 2007), ACM, pp. 2231–2236. [10] Folmer, E. Usability patterns in games. In Future Play ’06: Proceedings of the 2006 con- [22] Pauletto, S., and Hunt, A. Interac- ference (2006). tive sonification of complex data. Interna- tional Journal of Human-Computer Studie. 67 [11] Franke, R. 2009 Annual Report on Charity (November 2009), 923–933. Gaming. Tech. rep., National Association of Fundraising Ticket Manufacturers, 2009. [23] Pederson, M., and Muhlberger, R. In- visible territory: Sonifying the game of go. [12] Gamma, E., Helm, R., Johnson, R., and In Australian Computer Music Conference Vlissides, J. Design Patterns: Elements of ACMC2005 (Brisbane, 2005). Reusable Object-Oriented Software. Addisson- Wesley, Toronto, Ontario. Canada, 1995. [24] Snowden, R. Gambling Times Guide to Bingo. Gambling Times, 1986. [13] Gaver, W. W. Auditory icons: using sound in computer interfaces. Human-Computer- [25] Song, H. J., and Beilharz, K. Aesthetic Interaction 2 (June 1986), 167–177. and auditory enhancements for multi-stream [14] Hershenson, M. Reaction time as a measure information sonification. In Proceedings of the of intersensory facilitation. Journal of Experi- 3rd international conference on Digital Inter- mental Psychology 63 (Mar 1962), 289–93. active Media in Entertainment and Arts (New York, NY, USA, 2008), DIMEA ’08, ACM, [15] InspiredCode. Music tetris: pp. 224–231. http://www.inspiredcode.net/metris.htm, 2008. [26] Walker, B. N., and Mauney, L. M. Uni- versal design of auditory graphs: A compar- [16] Massof, R. W. Auditory assistive devices ison of sonification mappings for visually im- for the blind. E. Brazil and B. Shinn- paired and sighted listeners. ACM Trans. Ac- Cunningham, Eds., Boston University Publi- cess. Comput. 2 (March 2010), 12:1–12:16. cations Production Department, Boston Uni- versity Publications Production Department, [27] Westin, T. Game accessibility case study: pp. 271–275. Terraformers - a realtime 3d graphic game. In Proc. of the The Fifth Int. Conference on Dis- [17] Miller, D., Parecki, A., and Douglas, ability, Virtual Reality and Associated Tech- S. A. Finger dance: a sound game for nologies (2004). blind people. In Assets ’07: Proceedings of the 9th international ACM SIGACCESS con- [28] Willmott, C. J. R. Revision bingo. Bio- ference on Computers and accessibility (New chemistry and Molecular Biology Education York, NY, USA, 2007), ACM, pp. 253–254. 29, 5 (2001), 193 – 195. [18] Miller, J. Divided attention: evidence for [29] Yuan, B., Folmer, E., and Harris, F. C. coactivation with redundant signals. Cogn Game accessibility; a survey. Universal Access Psychol 14, 2 (Apr 1982), 247–79. in the Information Society 10, 1 (2010). [19] Neuhoff, J., Kramer, G., and Wayand, [30] Zhao, H., Plaisant, C., Shneiderman, B., J. Pitch and loudness interact in auditory dis- and Lazar, J. Data sonification for users with plays: Can the data get lost in the map? Jour- visual impairment: A case study with georef- nal of Experimental Psychology Applied 8, 1 erenced data. ACM Trans. Comput.-Hum. In- (2002), 17–25. teract. 15 (May 2008), 4:1–4:28. 15

An Analysis of Multimodal Representation of a Bingo Game

Daniel Ramos

GameTech International, 8850 Double Diamond Parkway Reno, NV 89521

Eelke Folmer Player-Game-Interaction Lab, University of Nevada, Reno, 1664 N. Virginia St., Reno, NV 89557

Abstract As indicated by their name, video games are primarily represented to a player using visual cues. Studies have shown that cues represented in multiple modalities, that are presented simultaneously, can be detected at lower thresholds, faster, and more accurately than when presented separately in each modality. This paper presents two user studies of the use of supplemental feedback to reduce errors in playing Bingo, a game which is typically played by a demographic, which-due to their age-are more likely to suffer from sensory impairments such as low vision or hearing. The first user study explored three different types of sonification (pitch, timbre, and audio icons) versus using no sonification. The second user study was expanded to include haptic icons in conjunction with the best sonification modes from the first study (haptic, timbre & haptic, audio icons & haptic). In both studies we found that supplemental feedback significantly reduces players’ errors, with the best modes being audio icons (in the first study) and haptic (in the second study). Keywords: Sonification, Haptic, Tacton, Multimodal Feedback, Games

1. Introduction

Alternate modalities in video games is a research area that is still in its infancy. Although many current games contain sound and haptic feedback, most of these are used to enhance the player’s experience (e.g. music, audio effects, and using a controller rumble so the user can “feel” an explosion) and not to help the user make choices or otherwise increase the player’s performance. Visual cues are still the primary way a user experiences a game, thus a user with a visual impairment would have a more difficult time playing than a user with an auditory impairment [1]. In this paper we will use alternate modalities to determine if a player’s errors are reduced when these methods are presented in addition to standard visual cues. In video games, there are a number of ways a player could make an error. Besides inputting the incorrect response, there is also the possibility that the player does not provide input in the time frame required, or completely misses that a response is needed. We theorize that multiple modalities will help decrease the chance the player would delay or miss the necessary input. Sonification (also known as auditory display) is the process of displaying data in an audio format (other than traditional speech) [2]. Sonification can use various different techniques to represent data, for example, changes in pitch, volume, timbre, patterns, or frequency. Sonification has been pioneered in sensory substitution [3] to communicate information to users who are visually impaired [4, 5]. Haptics is the use of force feedback devices to present information to be sensed by the skin, muscles, and joints. For example, the use of vibration in cell phones to alert the user to a phone call

Email addresses: [email protected] (Daniel Ramos), [email protected] (Eelke Folmer) 16 when the phone is in “silent mode”. In this paper, we refer to our use of haptic icons as “tactons”. Tactons are structured tactile icon or vibrotactile messages, which can be developed by manipulating multiple dimensions (e.g. intensity, rhythm, and spatial location) [6]. Previous studies have shown that a stimulus represented in multiple modalities simultaneously can be detected at lower thresholds, faster, and more accurately than when information is provided separately in each modality [7, 8]. In this paper we explore supplemental use of soniﬁcation and tactons in playing electronic Bingo. This paper is organized as follows. Section 2 provides background and Section 3 discusses related work. Section 4 discusses the design of simulators used in user studies that are discussed in Section 5. Section 6 includes discussion and future work, and the paper is concluded in Section 7.

2. Background

Bingo is a game of chance, where multiple players compete against one another for a prize. Players receive one or more sheets of pre-printed cards that contain 25 squares arranged in a 5 x 5 matrix. Inside those squares are 24 numbers (one of the squares is usually designated the “free space” and does not contain a number). There are other types of cards that are slight variations on the game (e.g. the squares contain two numbers instead of one or there are two free spaces instead of one). The player must match his or her cards to the numbers randomly drawn for the game by the Bingo caller. If the drawn numbers match a particular pattern on the player’s card, the player calls out the word “Bingo!”, to alert other players of a possible win. The most common patterns include horizontal, vertical, or diagonal lines, though there are many other patterns that form various shapes and designs. All wins are checked for accuracy before the win is confirmed and the next game starts. Bingo can be traced back to Italy in the 1500s when it was played as a lottery game [9]. In the United States, Bingo is mostly played in churches, youth organizations, and other non-profits to benefit charities. Increasingly, Bingo is also played for profit by casinos and Native American tribes. Despite its mostly charity status, Bingo is big business. The National Association of Fundraising Ticket Manufacturers reports that in 2009, over 1.8 billion dollars was spent on charitable Bingo in the United States [10], which dwarfs any current AAA video game title. Bingo is often pictured in popular culture as the domain of old people (and predominantly women). Studies show the stereotype of the average Bingo player has been steadily changing. While 57% of players are women, only 11% are 65 or above [11]. The UK has experienced similar demographic changes. The proportion of players under 45 has increased from 46% to 62% in the last decade [12], where players age 18 to 24 currently make up 20% of the Bingo demographic. The emergence of online Bingo, which currently makes up 20% of the UK market, is credited with this growth. In order to increase their chance at winning, most people play multiple Bingo cards at the same time. This is why most cards are printed on sheets of three or more (up to 18). In last decade, the emergence of computers in everyday life has injected itself into even this past time. Many states now allow players to utilize computers known as “electronic daubers” (see Figure 1) to aid in keeping track of their cards. Electronic daubers allow the player to keep track of many more cards then they normally would be able to. In the United States, there are many different jurisdictional bodies and their rules for the use of electronic Bingo differs widely. Some require the player to manually type the called number into the device, while others allow the device to automatically detect ball calls. In all cases, the user is still required to notify others of his or her win by yelling “Bingo!”, but most jurisdictions allow the device to tell the user when he or she has won. Bingo represents a simple choice response task in a gaming context [1]. When playing a video game the player provides input based on the stimuli the game provides, which then generates more input from the player and so on. For example, in a game when an enemy approaches, the player may choose to either fight or run away. For Bingo, the task consists of the player marking the called number, if it exists, on his or her card and then calling out “Bingo” if that card matches a particular pattern being played for. Supplemental feedback of how close a player is to achieving a Bingo may 17

be useful to avoid the possibility that the player does not realize he or she has a winning card. This occurs occasionally because: (1) Bingo halls and casinos can be noisy environments where it can be hard to follow the game; (2) due to their specific demographic, players are more likely to suffer from sensory impairments such as low vision or low hearing; and (3) players play the game using a large number of cards which make it more difficult to identify a winning pattern.

3. Related Work

Sonification has primarily been explored in sensory substitution for creating audiogames that can be played by people with visual impairments. An extensive overview of audiogames can be found in Yuan [1]. There have been several studies to determine the best methods and techniques for representing information using audio. Some methods include using pitch, volume, tempo, timbre, auditory icons (i.e. abstract sounds used in operating systems to alert the user) [13], and earcons (i.e. structured sounds that follow musical conventions) [14]. Most audiogames use audio cues, but we identified only one game that uses supplemental sonification. Terraformers [15] is a first person shooter that can be played by people with visual impairments as well as able-bodied players. Terraformers uses an audio compass using pitch, and a sonar like mechanism is used for conveying how far objects are in front of the character. We are not aware of any mainstream games that use haptic feedback as a form of primary stimulus. Most games only use haptic feedback as a secondary stimulus to enhance the player’s experience. In many video games, an explosion is related to the user via vibrations in addition to visual and auditory feedback. The line between primary and secondary stimuli might be blurred in cases where vibrations can represent when the player has done something wrong. For example, in a racing game, using haptic feedback to signal when the player has drifted off the track and into grass or dirt. Recently there have been several games created to allow users with visual impairments to play games using haptic feedback as a primary stimulus. Blind Hero [16] is a modification to Guitar Hero [17] that uses a glove to provide tactile feedback on which inputs the player must provide, while still 18

Figure 2: A screen shot of the Bingo Simulator (version 1). The away count for the middle card has changed and is soniﬁed.

playing the original music. VI-Tennis [18] is an exergame that provides audio and haptic cues for playing the game of tennis using a Wii remote. Similar is VI-Bowling [19], which uses haptic cues to help the player aim a bowling ball down the lane to determine where to throw the ball.

4. Bingo Simulator

The program we designed to determine how useful tactons and sonification are in a bingo game is known as the Bingo Simulator. The Bingo Simulator generates three random cards for use in each game of Bingo (see Figure 2). Three cards are normally what a player would see when using an electronic dauber, though, usually, electronic daubers can display many times that. The pattern needed to achieve Bingo is a straight line either with all called numbers in a row, column, or diagonal. When the player presses the “Start Ball Calls” button, the application will start drawing random numbers from 1 to 75 every six seconds and display them below the cards on what is called a “flashboard”. The last ball called and possible patterns are displayed next to the flashboard. To mark a number on his or her card, the player must type that number with the keyboard/keypad and press enter or click the “Daub” button. Each card will then be daubed in sequence from left to right and a green background will be rendered in the number’s square if it is present on the card. If the user gets five daubs in a straight line, they win and the Bingo Simulator moves to the next game and mode of feedback.

4.1. Version 1 For our ﬁrst study, version 1 of the Bingo Simulator was created. Version 1 is a Windows program written in C# using Microsoft’s .NET Framework and Windows Presentation Foundation technologies. It runs on any standard desktop or laptop computer running Windows XP or higher. For our user study we used a laptop with built-in 4Ω, 2 Watt, stereo speakers.

4.2. Version 2 For our second study, version 2 of the Bingo Simulator was written in C# using the .NET Compact Framework and Windows CE 6.0 R3. It runs on the GameTech ExplorerTM portable electronic dauber. The Explorer is a custom-built tablet with a 10.2” touch screen display and 10-digit, back-lit keypad. It contains two coin vibration motors, sitting behind the keypad, that operate at a maximum frequency of 54Hz. The Explorer also contains 4Ω, 1 watt, stereo speakers. 19

We chose the Explorer for the 2nd phase because we wanted to utilize its vibration motors, and having built-in motors was a more realistic option than having a user wear a haptic device or attach motors to the outside of a laptop chassis.

4.3. Measures

Figure 3: The away count test for each card (version 1).

In Bingo, the number of daubs needed to achieve a Bingo is known as the “away count” (e.g. if the Bingo pattern is a straight line, and the player has four squares daubed, then he or she is one square away from winning). Each time a number is daubed on the player’s cards, the away count may change. Depending on the Simulator’s mode, if a card’s away count is changed, the sound and/or tacton corresponding to that away count is played while the card is highlighted. The Bingo Simulators will take up to two seconds to present the supplemental feedback (depending on the version). This allows all cards to potentially play the sounds/tactons back to back at still remain in the six-second interval between ball calls. In order to measure the effectiveness of multiple modalities in a bingo game, the Bingo Simulator randomly asks the user what he or she thinks the away count is on each of his or her three cards. When the test occurs, the cards are hidden and a dialog pops up asking for the users responses (see Figure 3). The user types the responses using the keyboard/keypad. The user is tested a total of four times per mode or until a Bingo is achieved, whichever comes first. To prevent too short of an interval between tests, the simulator will have a minimum of five and a maximum of nine ball calls between each test. After the player progresses through each mode, the application exits and the results are recorded to a log file. We record for each game (using one unique type of sonification and/or haptic feedback): (1) total playing time; (2) number of away count prompts; (3) time it took the player to answer the away count prompts; (4) away count; and (5) player provided away count for each card. We theorize that providing alternate modalities for away count will help the user avoid missing that they have a Bingo (also known as “sleeping” a Bingo). It is difficult to test for a sleeper as that can only be observed over a large number of games. We anticipate that using multiple modalities will decrease the rate at which users make mistakes in their away counts answers and thus increase efficiency of detecting winning patterns.

5. User Studies

5.1. Phase 1 Feedback Strategies For the first phase of user studies, we seek to sonify the away count for each card. Although sonification is generally useful for enhancing program usage, there are some potential pitfalls that could cause problems. This is more likely to occur when sonifying multiple variables and mapping them into one audio stream. Neuhoff et al. [20] performed a study to investigate the interaction 20 between pitch and volume sonification methods. They showed that pitch and volume interact to mask the actual data being sonified. Specifically, rising intensity sounds were perceived to change more than equivalent falling intensity sounds. In our study, we will avoid this problem because we will sequentially sonify variables using one type of sonification at a time. Sonification solutions have been cataloged using a technique known as “design patterns”, which is a concept that has been pioneered in describing object oriented design solutions [21], but patterns collections for other domains exist, such as interaction design for video games [22]. The Ears-Lead- Eyes design pattern involves designing a sonification solution for when users need to be notified of intermittent, but critical, information while they are busy with another visual task. This is helpful because it reduces the need to keep switching visual attention between different areas in an application or keep the user from potentially missing important information. For our first study we explore the following three types of sonification: Pitch: An audio cue is played at different pitches (99Hz, 119Hz, 156Hz and 193Hz) to represent • the away count. A lower frequency indicates a low away count. Timbre: A different instrument plays a middle C note to represent a certain away count. • The instruments used are a piano, cello, organ, and pan flute. We chose instruments where the middle C note pitch sounded similar, but where the tone “quality” or “color” was easily distinguished. Audio Icons: One of four completely different audio cues are played to represent a certain • away count. We use the following audio icons: a dog barking, a jackhammer, a cash register bell ring, and an audience clapping. The cash register and audience clapping were used for low away counts to signal that a Bingo was close to happening. We selected audio icons that sounded as different as possible from each other. Volume was also considered but previous sonification studies found [23, 24] volume to be inferior to pitch and timbre. Sonification types such as increasing the frequency between audio cues are not applicable since we use discrete sonification (e.g. we briefly play one sound to convey the value of the variable as opposed to a continuous representation). All sounds were selected from open source

Figure 4: Spectrograms for the four audio icons. sound libraries, and pitch shifts of were created using Audacity [25] audio software where pitch was iteratively increased approximately 150% for each cue. Figure 4 shows the respective spectrograms for the used audio icons that reveals their diﬀerences. All sounds had an average length of 1.5 seconds. In the case where all three cards’ away counts change, the sounds are played back to back in the six-second interval between ball calls.

5.2. Phase 1 User Study The Bingo Simulator (version 1) application was evaluated with nine adult volunteers (2 female, age M = 41.2 SD = 12.8). Subjects were recruited from a local gaming company that specializes in 21

Table 1: Results of phase 1 Bingo simulation.

Type Error σ Time (ms) σ (ms) NONE 0.241 0.19 16,554 22,677 PITCH 0.105 0.14 9,879 5,672 TIMBRE 0.074 0.11 9,149 8,510 AUDIO ICONS 0.059 0.06 6,954 1,815

developing Bingo systems, and, as such, all subjects could be considered expert Bingo players. All subjects were in good health, with no hearing or vision impairments (some had corrected vision) that could possibly impede their ability to play the game. Each user was given a brief explanation and walk-through of the application, a demonstration of the different sounds, and subjects were able to play a number of games using different sonification techniques. Subjects were tested alone in a room, to minimize background noises, using a laptop. The laptop’s built-in speakers were used for sonification, which was motivated by that electronic daubers systems often have similar speakers. The effectiveness of the three different types of sonification (pitch, timbre, audio icons) as well as using no sonification was evaluated using a within-subjects design. The users played through four games, each using one of the four sonification methods, in random order. The game would end if the subject achieved a Bingo or was tested four times, which ever came first. After the test, the subjects filled in a brief questionnaire where they were asked which sonification method they felt helped them remember the cards’ away counts most effectively.

5.3. Phase 1 Results Performance data were gathered from the logs and Table 1 lists the results from our study. Our analysis focuses on two variables: (1) error rate (second column) which is the ratio between incorrectly answered away counts and total away counts; and (2) time to complete away count prompts (fourth column). The third and the fifth column list the found standard deviations. For all tests, α was set at 0.05. When using an ANOVA with repeated measures with a Greenhouse-Geisser correction, the mean errors were statistically significantly different (F 1.574,24 = 4.67 p < 0.05). Post-hoc tests revealed that using no sonification yields a significantly larger number of errors than timbre (p = 0.047) and audio icons (p = 0.034), but no significant difference was found between pitch and using no sonification (p = 0.085). Also, no significant difference was found between timbre and audio icons (p = 0.521). Due to our small sample size, no corrections were applied. Using the same ANOVA, the mean values for time were not found to be statistically significantly different (F 1.049,24 = 1.036 p > 0.05). Results from the questionnaire revealed that all subjects felt sonification, in general, helped them play Bingo. The majority of the participants preferred audio icons; but two preferred the use of pitch; one user preferred timbre; and one user preferred timbre as well as pitch.

5.4. Phase 2 Feedback Strategies For the second study we expanded upon our previous work. We selected the two most effective sonification methods based on our previous results and added tactons. Our expectation is increased multimodal representation will further decrease the users’ error rate in detecting away counts. We chose the Explorer because it can provide stimuli in three modalities with its screen, speakers, and vibration motors. Recently, there have been studies to determine the effectiveness of using vibration as a method of non-visual feedback. Brown et al. [26] performed a study where they investigated the perception of tactons using vibro-tactile “roughness”, described as amplitude modulated sinusoids, together with different vibration rhythms. Their experiments showed the rhythm based tactons were identified over 90% of the time and the roughness tactons were recognized 80% of the time. In our study, we 22

Away Count 2

0 200 400 600 800 1,000 1,200 1,400 Time (ms)

Figure 5: Tacton patterns.

will focus only on rhythm since the Explorer’s vibration motors can only run at a constant frequency and rhythm seems to be the more effective method. Qian et al. [27] investigated using vibration pulse tactons 4.2 to 7 seconds in length. The study included tacton pairs that used fixed and dynamic durations & intervals. They found that the most effective pair was two static tactons that had the same interval, but different durations (200 milliseconds vs. 800 milliseconds). Based on this finding we will be using similar tactons were we mainly vary the duration of the pulses. For our second study we explore the following three feedback modes: Haptic: The Explorer’s coin vibration motors will activate in a different pattern to represent • a certain away count (see Figure 5). One away is one continuous 1.5 second duration pulse. Two away is two 600 millisecond pulses with a 300 millisecond pause in between. Three away is three 300 millisecond pulses with two 300 millisecond pauses in between. Four away is four 200 millisecond pulses with three 233 millisecond pauses in between. Timbre & Haptic: The timbre sounds are the same as in phase 1. The Explorer’s coin • motors are activated during the note in the same patterns as haptic mode.

Audio Icons & Haptic: The audio icons are the same as in phase 1. The Explorer’s coin • motors are activated during the cue in the same patterns as haptic mode. For this phase, we matched the length of the tactons to the average duration of the sounds (1.5 seconds). Each tacton/sound played is followed by a 500 millisecond pause for a total away count feedback length of two seconds. This pause between cards allows the user to feel/hear when the current card is ﬁnished before the next card could potentially start.

5.5. Phase 2 User Study For our second user study we used the same volunteers from our previous study. Two months had past since the first phase. Each user was given an overview of the Bingo simulator (version 2) and a demonstration of the haptic modes that had been added since they had last used the simulator. The users were tested alone in a room, to minimize background noises, with the Explorer. The effectiveness of the three different types of feedback (haptic, timbre & haptic, audio icons & haptic) was evaluated using a within-subjects design. The users played games using each one of the three 23

Table 2: Combined results from phase 1 & 2 Bingo simulation.

Type Error σ Time (ms) σ (ms) NONE 0.241 0.19 16,554 22,677 PITCH 0.105 0.14 9,879 5,672 TIMBRE 0.074 0.11 9,149 8,510 AUDIO ICONS 0.059 0.06 6,954 1,815 HAPTIC 0.0089 0.027 6,975 3,179 HAPTIC + TIMBRE 0.0656 0.101 7,539 3,446 HAPTIC + AUDIO ICONS 0.111 0.18 7,913 4,224

feedback methods, in random order. The game would end if the subject achieved a Bingo or was tested four times, which ever came ﬁrst. After the test, the subjects were interviewed where they were asked which mode they felt helped them remember the cards’ away counts most eﬀectively.

5.6. Phase 2 Results Performance data were gathered from the logs and Table 2 lists the results from our study combined with the first study. As in phase 1, our analysis focuses on two variables: (1) error rate which is the ratio between incorrectly answered away counts and total away counts; and (2) time to complete away count prompts (fourth column). The third and the fifth column list the found standard deviations. For all tests, α was set at 0.05. When using an ANOVA with repeated measures with a Greenhouse-Geisser correction, the mean errors were statistically significantly different (F 2.567,48 = 4.448 p < 0.05). From both our studies we can say timbre, audio icons, haptic, and timbre & haptic were significantly better than none (p = 0.047, p = 0.034, p = 0.04, p = 0.012), and pitch and audio icon & haptic were not better than none (p > 0.05). Haptic had the lowest error rate and this was significantly better than none, pitch, and audio icons, but not significantly better than timbre, timbre & haptic, and audio icons & haptic. There may be some overload effects as combined techniques (such as audio icons & haptic) were not significantly better than any of the other techniques nor using no technique. Timbre & haptic was only significantly better than none, but not significantly better than any of the other techniques. Due to our small sample size, no corrections were applied. Using the same ANOVA, the mean values for time were not found to be statistically significantly different (F 1.462,48 = 0.592 p > 0.05). Results from the interviews revealed all subjects felt supplemental feedback, in general, helped them play Bingo. Approximately half of the participants preferred haptic while the other half preferred timbre & haptic; one participant preferred audio icons & haptic. Some subjects felt the sonification and haptic feedback combined was too much information, indicating that the combination of visual, auditory, and physical modalities might result in cognitive overload for some. A previous study by Kalyuga [28] has shown that redundant modalities might impose an additional, unnecessary cognitive load which interferes with learning. Those that favored haptic-only said the number of vibrations directly indicating away count was more effective than having to connect an abstract sound to the count.

6. Discussion and Future Work

From the results of both users studies we see some form of supplemental feedback reduces user errors, but not efficiency. Unlike most other kinds of software, where the user is only concerned with accomplishing the task in the most efficient manner, an important aspect in games is their “fun factor”. In the context of games, faster does not necessarily mean better, but the question could be asked, “Does less errors mean more fun?” Pinelle et al. [29] point out that errors can be expected in 24 games because part of the challenge is to learn new skills to overcome previous mistakes. There is potential for some users to see alternate forms of feedback as less fun because it makes detecting a Bingo “easier”. The same can be said for using an electronic dauber instead of playing Bingo using paper. Our findings could potentially affect other types of games. For example, puzzle games like Bejeweled often use visual indicators, such as a progress bar, that indicate how far the player is in the current level before moving to the next. Such values could be indicated using sonification or tacton patterns. Alternately, supplemental feedback could be used to indicate the row or column where two of the same jewels are next two each other, signaling to the player that the third jewel needed to complete the pattern might be near by, which may increase the player’s performance. We utilized the Explorer for the second user study because of its vibration motors. The Explorer has only recently been released by GameTech and is one of, if not the first, portable electronic dauber to have the potential to use haptic feedback. Even though we used two different pieces of equipment in our user studies, we feel that the similarity of the speakers in both the Explorer and the laptop would minimize any sonification bias that might appear in the results. One of the most limiting factors in the second user study was that the Explorer’s vibration motors have a fixed frequency. Future work could include using a different vibration motor that would allow control of frequency so the tactons could encode two pieces of information (intensity and rhythm). A study by Brown and Kaaresoja [30] found a 72% recognition rate for intensity-based tactons in tests with mobile phones.

7. Conclusion

In this paper we explored the supplemental feedback of information required to play the game of Bingo. Bingo halls and casino environments are often crowded and potentially noisy and Bingo is often played by players who-due to their age-are more likely to suffer from visual or hearing impairments. Supplemental feedback may be useful for Bingo to avoid players not realizing they have won. Two user studies, with nine subjects, assessed the performance of six different types of feedback (pitch, timbre, audio icons, haptic, timbre & haptic, audio icons & haptic) as well as using no feedback. Results show that sonification using timbre & audio icons and use of “tactons” significantly reduces player errors.

References

[1] B. Yuan, E. Folmer, F. C. Harris, Game accessibility; a survey, Universal Access in the Information Society 10 (2010). [2] S. Barrass, G. Kramer, Using soniﬁcation, Multimedia Systems 7 (1999) 23–31.

[3] P. Bach-y Rita, S. W Kercel, Sensory substitution and the human-machine interface, Trends in Cognitive Sciences 7 (2003) 541–6. [4] R. W. Massof, Auditory assistive devices for the blind, Boston University Publications Produc- tion Department, Boston University Publications Production Department, Boston, USA, 2003, pp. 271–275.

[5] B. N. Walker, L. M. Mauney, Universal design of auditory graphs: A comparison of soniﬁcation mappings for visually impaired and sighted listeners, ACM Trans. Access. Comput. 2 (2010) 12:1–12:16. [6] S. Brewster, L. Brown, Tactons: structured tactile messages for non-visual information display, in: Proceedings of the ﬁfth conference on Australasian user interface-Volume 28, Australian Computer Society, Inc., pp. 15–23. 25

[7] J. Miller, Divided attention: evidence for coactivation with redundant signals, Cogn Psychol 14 (1982) 247–79. [8] M. Hershenson, Reaction time as a measure of intersensory facilitation, Journal of Experimental Psychology 63 (1962) 289–93. [9] R. Snowden, Gambling Times Guide to Bingo, Gambling Times, 1986.

[10] R. Franke, 2009 Annual Report on Charity Gaming, Technical Report, National Association of Fundraising Ticket Manufacturers, 2009. [11] Everything you need to know about bingo and pull tabs, Technical Report, Arrow International, Inc., 2007.

[12] Bingo - UK - April 2009, http://oxygen.mintel.com/sinatra/oxygen/display/id=394689, Technical Report, Mintel International Group Ltd., 2009. [13] W. W. Gaver, Auditory icons: using sound in computer interfaces, Hum.-Comput. Interact. 2 (1986) 167–177. [14] S. Brewster, P. Wright, A. Edwards, Experimentally derived guidelines for the creation of earcons, in: Adjunct Proceedings of HCI, volume 95, Citeseer, pp. 155–159. [15] T. Westin, Game accessibility case study: Terraformers - a realtime 3d graphic game, in: Proc. of the The Fifth Int. Conference on Disability, Virtual Reality and Associated Technologies. [16] B. Yuan, E. Folmer, Blind hero: enabling guitar hero for the visually impaired, in: Assets ’08: Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility, ACM, New York, NY, USA, 2008, pp. 169–176. [17] Wikipedia, http://en.wikipedia.org/wiki/Guitar_Hero, 2011. [18] T. Morelli, J. Foley, E. Folmer, VI-Bowling: a tactile spatial exergame for individuals with visual impairments, in: Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility, ASSETS ’10, ACM, New York, NY, USA, 2010, pp. 179–186. [19] T. Morelli, J. Foley, L. Columna, L. Lieberman, E. Folmer, VI-Tennis: a vibrotactile/audio exergame for players who are visually impaired, in: Proceedings of Foundations of Digital Interactive Games (FDG’10), Monterey, California.

[20] J. Neuhoﬀ, G. Kramer, J. Wayand, Pitch and loudness interact in auditory displays: Can the data get lost in the map?, Journal of Experimental Psychology Applied 8 (2002) 17–25. [21] E. Gamma, R. Helm, R. Johnson, J. Vlissides, Design Patterns: Elements of Reusable Object- Oriented Software, Addisson-Wesley, Toronto, Ontario. Canada, 1995. [22] E. Folmer, Usability patterns in games, in: Future Play ’06: Proceedings of the 2006 conference.

[23] H. Zhao, C. Plaisant, B. Shneiderman, J. Lazar, Data sonification for users with visual impairment: A case study with georeferenced data, ACM Trans. Comput.-Hum. Interact. 15 (2008) 4:1–4:28. [24] H. J. Song, K. Beilharz, Aesthetic and auditory enhancements for multi-stream information sonification, in: Proceedings of the 3rd international conference on Digital Interactive Media in Entertainment and Arts, DIMEA ’08, ACM, New York, NY, USA, 2008, pp. 224–231. [25] Audacity: Free audio editor and recorder, http://audacity.sourceforge.net, 2011. [26] L. Brown, S. Brewster, H. Purchase, A first investigation into the effectiveness of tactons (2005). 26

[27] H. Qian, R. Kuber, A. Sears, Towards identifying distinguishable tactons for use with mobile devices, in: Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility, ACM, pp. 257–258. [28] S. Kalyuga, When using sound with a text or picture is not beneﬁcial for learning, Australian Journal of Educational Technology 16 (2000) 161–172.

[29] D. Pinelle, N. Wong, T. Stach, Heuristic evaluation for games: usability principles for video game design, in: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, ACM, pp. 1453–1462. [30] L. Brown, T. Kaaresoja, Feel who’s talking: using tactons for mobile phone alerts, in: CHI’06 extended abstracts on Human factors in computing systems, ACM, pp. 604–609. 27

Chapter 4

Conclusion

4.1 Results

Based on the two user studies we performed, we see that supplemental and alternate repre- sentations of information required to play the game have significant benefits. In all cases, the users in the studies had reduced error rates regardless of the type of supplemental feedback. The results show that pure haptic feedback was the most effective modality because it led to the lowest, statically significant error rate. Many of the users noted they preferred the haptic feedback over sonification and the combination of the two became overwhelming to some.

4.2 Fun and Fairness

As discussed in the previous article, unlike most other kinds of software, an important aspect in games is their “fun factor”. Errors can be expected in games because part of the challenge is to learn new skills to overcome previous mistakes. There is potential for some users to see alternate forms of feedback as less fun because it makes detecting a Bingo

“easier”. The same can be said for using an electronic dauber instead of playing Bingo using paper.

One could also argue that using supplemental feedback, in addition to visual feedback, gives a user an unfair advantage over those just using one form of stimuli. Complaints could also arise stating that soniﬁcation produces noises that could annoy or discourage neighbors 28

(i.e. if a player hears the person sitting next to them is one away they could become less interested if they feel they have no chance of winning). Perhaps that reason could be a deciding factor for the use of tactons over soniﬁcation, besides the better performance, as it is not intrusive because only the person holding the device can perceive them.

4.3 Future Work

The next step in this research would likely include performing more user studies in real world environments (e.g. casinos and bingo halls) with an older demographic or people with disabilities to get qualitative feedback. Soniﬁcation and haptic feedback are already being applied to other types of games, but it is usually not a primary stimulus. Another possibility for continuing research is looking at other games or genres to apply the research

(e.g. slot games, non-casino or casual games, serious games, etc.).

4.4 Applications

Since the second phase of user studies involved an actual electronic dauber (GameTech’s

Explorer), the results of the study could be applied in the near future. Most electronic daubers have the ability to produce sound and the Explorer has haptic motors built-in, so

GameTech could potentially implement supplemental feedback in their Bingo system with minimal eﬀort.

In order to accommodate people who do view supplemental feedback as making the game less fun, the Explorer could have an option to turn soniﬁcation and/or tactons on or oﬀ independent of each other. This is also important for legal reasons. There are some jurisdictions in the United States that do not allow an electronic dauber to make any sound at all (or only produce sounds in certain situations). This toggle capability would also be important if jurisdictions begin to regulate haptic feedback. Currently, we are not aware of any jurisdictions that regulate the use of haptic feedback. 29

Bibliography

[1] Chart of the Week: Mobile Internet Devices Outsell Computers, http://www.practicalecommerce.com/articles/ 1654-Chart-of-the-Week-Mobile-Internet-Devices-Outsell-Computers, access- date: 4-16-2011.

[2] SciLands Virtual Continent, http://www.scilands.org/, access-date: 4-2-2011.

[3] SparkNotes - Sensation and Perception, http://www.sparknotes.com/psychology/ psych101/sensation/section3.rhtml, access-date: 4-13-2011.

[4] Webvision, http://webvision.med.utah.edu/, access-date: 4-13-2011.

[5] ISO 9241-11:1998 - Ergonomic requirements for oﬃce work with visual display terminals (VDTs) – Part 11: Guidance on usability. 1998.

[6] Entertainment Software Association. 2010 Essential Facts About the Computer and Video Game Industry, http://www.theesa.com/facts/pdfs/ESA_Essential_Facts_ 2010.PDF, access-date: 4-13-2011.

[7] Paul Bach-y Rita and Stephen W Kercel. Sensory substitution and the human-machine interface. Trends in Cognitive Sciences, 7(12):541–6, Dec 2003.

[8] U.S. Census Bureau. Internet Use in the United States: October 2009, http://www. census.gov/population/www/socdemo/computer/2009.html, access-date: 4-2-2011.

[9] R.W. Cholewiak and A.A. Collins. Sensory and physiological bases of touch. 1991.

[10] R.W. Cholewiak and M. Wollowitz. The design of vibrotactile transducers, 1992.

[11] Eelke Folmer. Software Architecture Analysis of Usability. PhD thesis, University of Groningen, Department of Computer Science, 2005.

[12] Roger Franke. 2009 Annual Report on Charity Gaming. Technical report, National Association of Fundraising Ticket Manufacturers, 2009.

[13] M. Kebritchi, A. Hirumi, et al. Examining the pedagogical foundations of modern educational computer games. Computers & Education, 51(4):1729–1743, 2008. ISSN 0360-1315.

[14] Bei Yuan, Eelke Folmer, and Frederick Harris. Game accessibility: a survey. Universal Access in the Information Society, 10:81–100, 2011. ISSN 1615-5289.