MSN GAMES – Badges Usability #1 Date: 8/16/05 Table of Contents To: Zone Digital Objects In-Brief

Find Usability & Playtest Reports at http://usertesting

MSN GAMES – Badges usability #1 Date: 8/16/05 Table of Contents To: Zone Digital Objects In-Brief ...... 1 From: Liz Sanocki Key Findings ...... 2 CC: Games User Research Team Usability Issues ...... 2 Feature Enhancements: ...... 14 Questionnaire Summary and Data ...... 15 Participants and Procedures ...... 16 Screen Shots ...... 17

In-Brief Introduction: The usability of MSN games badges was tested August 1 – 3, 2005 with ten participants aged 40–59 years. The test focused on the discoverability and understandability of badges and the end-to-end badge experience; navigating the game-end screen and the badges album.

Participants were casual gamers -- some played web games occasionally and others played 10-20+ hours/week. A couple also played some console or PC games. None had experienced badges on MSN games. Two were Club Pogo members.

Videos of all sessions can be found at:

Overall Results: Badges were well liked. Participants went from being ambivalent about badges at the beginning of the usability testing session to being very enthusiastic about them at the end. Some of the participants were looking forward to going home and collecting more badges. Others recognized the attraction of badges for other people, but were not “collectors” themselves (or chose not to be).

The results suggest that once users are exposed to badges, the program will sell itself. However, we need to raise awareness of badges before users get into and play their games. 80% of the participants failed to use the Quit button, suggesting that users are likely to miss exposure to badges on the game end screen.

The results also suggest that badges need to be easy to use. The following video clip emphasizes the importance of making badges approachable and easy to use:

Moving forward, it is important to promote the appealing aspects of badges to entice customers to try them. It is also important to allow users to access their albums when not playing a game. Recommendations include providing links directly to the badge album and a list of games with badges on the home page, game hub pages and the individual game pages (default.htm). Real-time notification of badge awards while playing games is also recommended. Features considered important to add to badges included the ability to “use badges to enter sweepstakes” and to “unlock new levels of a game”.

Detailed problem areas are outlined below. For the most part recommended solutions may be addressed with small changes to the existing system.

Confidential Information - Do not forward outside of MGS or MGS Partners 1 Find Usability & Playtest Reports at http://usertesting

Key Findings Severity 1: • When not signed in, eight of 10 participants lost their badges when they quit a game.

Severity 2: • Two participants lost badges by accessing the album (and thus signing in) from the Learn About Badges article page. • Two of the participants were unable to view badges on the end-game screen and the album because Flash was not installed on their computers. • Three of 10 participants assumed that their Badge Albums would update automatically during games as they earned new badges. • Six of 10 participants failed to use navigation elements in the Badges Album. • When participants failed to close the Available Badges popup window, it became “lost” behind other browser windows, causing the link to Available Badges to appear broken.

Severity 3: • Six of eight participants had difficulty determining how many points were required for their next badge when using the Available Badges popup window (Badges.htm). • Half of the participants expected links in the “Latest Additions” section of the badge album to allow them to view their earned badges for that game. • Three of the participants wanted to view all of their earned badges in one place. • Six of 10 participants were unsure where clicking the links in the Badge Album would take them. • Six of 10 participants failed to find a list of badge-enabled games.

Issues discovered during testing but not experienced by participants: • A user’s total number of games played needs to appear somewhere besides the game end screen (e.g. in the badges album), as total games played is used as a criterion for earning special promotional badges. • It was too easy for users to get an excessive number of opened browser windows. • The ability to the first badge in a game varied with games – it was easy to earn the first badge in Cubis and difficult to earn them in Combo Chaos and Bejeweled 2.

Usability Issues Table 1. The following are usability issues, sorted by Severity and Game Facet. Issues were derived from participant statements and direct observations. Key to Usability Severity Ratings: Severity 1: An issue that blocks a substantial number of players from continuing the game. E.g., Players are unable to start a game. Nearly impossible task during the initial experience. Severity 2: An issue that blocks a substantial number of players from making use of an important feature or accomplishing an important task. E.g., Unclear solution. Undiscoverable feature. Severity 3: An issue that hinders some users from accomplishing a task or goal. E.g., Inconsistent or misleading text. Players don't understand a non-essential aspect of the game. Delays. Severity 4: An issue that causes some users to become mildly frustrated or express minor complaints about a game element. E.g., Suggested Feature. Minor or infrequent complaint.

Confidential Information - Do not forward outside of MGS or MGS Partners 2 Find Usability & Playtest Reports at http://usertesting

Findings, Evidence, Recommendations, Fixes Severity UI Area

1. When not signed in, eight of 10 participants lost their badges when they 1 Game Window quit a game.

Participants were asked to play the first two levels of the game Zuma and then to “end the game”. Eight of 10 participants (P1, P2, P3, P4, P7, P8, P9, P10) failed to use the Quit button to end the game, and failed to read (or heed) the popup dialog warning them that their badges wouldn’t be saved. This resulted in their failing to collect any earned badges and effectively prevented them from learning about badges. o Two of the participants noticed the Quit button and used it to end their games (P5, P6). o None of the participants mentioned noticing badges before getting to the game-end screen. o After failing to use the Quit button, participants were directed to use it. Only then did they discover the existence of badges. o After discovering badges, one of the participants assumed that the information in the left of the game window automatically updated when badges were earned (P10).

Recommendations: Provide users with a way to experience badges that does not depend on viewing the game-end screen. Encourage users to sign in before they play a game. Consider the following: • Add content to the Home page and default game web pages to extol the virtues of badges and of signing in. • Add a sign-in button to the game page so users can sign-in after they have started playing. Clicking the Sign-in Button would pause the game. • Add dynamic badge award notification during the game that is presented in the left side of the game window (the location dedicated to badge information). • If these solutions fail, provide an optional sign-in page inserted in-line before launching the game installer. With this, provide a “don’t ask me this again” option (e.g. check box).

2. Two participants lost badges by accessing the album (and thus signing 2 Game Window in) from the Learn About Badges article page.

Four of the participants signed in to view badge albums without being prompted to do so (this is good!). When this happened, two of the four participants lost badges because they signed in from the “Learn About Badges” article page, having accessed the article from the “i” icon on the game end screen (this is bad L). Failing to see their newly earned badges in the album that had previously appeared on the end- game screen lead participants to think they had lost their badges. Recommendation: Consider sharing new badge data more globally on the server when users are not signed in and end game screen is still open. Modify the existing system to allow non-signed in users to get badges when they sign in from any location on the site (not just the game end screen as the site works today. If users close the game window without signing in, they lose the badges (badges earned in the game are removed from the server).

3. Two of the participants were unable to view badges on the end-game Game screen and the album because Flash was not installed on their 2 Window & computers. Badge Album

If flash was not installed, badges failed to appear. Empty white space remained instead, giving the game- end screen and badge album a strange, broken-looking appearance. If it was the participants’ first time viewing the game-end screen there was no way for them to know that something was missing. Although participants successfully followed instructions to install Flash, it required going to the Macromedia web site to do it. Recommendation: o When Flash is not installed, consider inserting a static version of the badge image where the flash-generated badges would normally appear. The empty white space makes the game-end

Confidential Information - Do not forward outside of MGS or MGS Partners 3 Find Usability & Playtest Reports at http://usertesting

screen appear broken. o Find a way to get flash installed more seamlessly without taking the user away from the MSN games site.

Badges 4. Three of 10 participants assumed that their Badge Albums would update 2 automatically during games as they earned new badges. Album

Once participants had signed in and were playing games, they returned to their Badge Albums expecting to see an updated display of their badges. They assumed that the information in the album automatically updated (in real time) while a game was being played (P5, P8, P9). After seeing that the badge album had failed to update, one of the participants pressed the F5 key to refresh the display. Recommendation: Consider allowing the badge album to update with new badges as a game is being played. In this way badges are not lost if users quit using the red “X” in the upper right of their game windows. (see related issue 1)

Album 5. Six of 10 participants failed to use navigation elements in the Badges 2 Album. Navigation

Participants experienced difficulty navigating within the badge album, especially when it opened in a small browser window. They failed to click on album navigation elements, suggesting that they did not look clickable.

• One participant (P5) never used the Album navigation at all. • Three participants failed to use navigation until after being instructed to maximize the size of the browser window (P5, P6, P7). • Six of the participants failed to click the elements (Home, Puzzle, Word, Action & Arcade) even after hovering over them with the mouse (P1, P2, P3, P5, P6, P7). • Six of the participants used the navigation once but then forgot to use it later, suggesting that had forgotten that it was clickable (P2, P3, P6, P7, P8, P9). • Two of the participants failed to scroll the album (P5, P6).

Recommendations: Consider the following: • Change the visual design to make the tab navigation look more “clickable” so they jump out of the page as navigation elements. Add tool tips. • Change “Home” label on tab navigation to say “Badges Home”. • Rearrange the layout of the badge album to encourage users to scroll down the page. • Group navigation elements together in one place near the tab navigation (including the link to “view all puzzle games”). This has the added benefit of pushing the Trophy Case lower on the page putting it more directly across from the badges it features. • Change the name of links from “Back to puzzle games” to “View puzzle games”. Many users will get to an album game page directly from an end-game screen, so saying “back to…” is confusing.

Confidential Information - Do not forward outside of MGS or MGS Partners 4 Find Usability & Playtest Reports at http://usertesting

6. When participants failed to close the Available Badges popup window, it Available became “lost” behind other browser windows, causing the link to 2 Badges Popup Available Badges to appear broken. Window

Some of the participants thought that links to view available badges were broken. In fact, they had clicked away from the Available Badges popup window instead of closing it, completely hiding the popup window. When links to the popup window were clicked again, they appeared to fail because the window was already opened. However, the link was not wired to change the z-order, bringing focus back to the popup window. This problem was exacerbated the presence of multiple browser windows. Participants typically had multiple versions of the Badge Album opened, a game window opened, a default game page opened and the MSN games Home page opened. Recommendation: o Update links to the Available Badges popup window to work when it has been left opened and has lost focus. o Work to reduce the number of instances of the Badge Album that are allowed to open at one time; consider refreshing windows rather than launching new instances of them.

7. Six of eight participants had difficulty determining how many points were Available required for their next badge when using the Available Badges popup 3 Badges Popup window (Badges.htm). Window

The Available Badges popup window (Badges.htm) used the same graphics for all games and the visual variation between badge “levels” was small, thus making the popup window difficult for participants to use (see screen shot, below). Six of eight participants had difficulty extracting information from the Available Badges popup window (P1, P2, P3, P4, P5, P7). Using similar graphics for all badges created a redundancy in badge appearance which became visual clutter and was an impediment to pulling out important information, such as point requirements for a badge.

Confidential Information - Do not forward outside of MGS or MGS Partners 5 Find Usability & Playtest Reports at http://usertesting

Recommendations: Consider showing static, simplified or low contrast versions of the actual badges in the Available Badges window (as Pogo does). Alternatively, remove the graphics and show the point requirements alone.

8. Half of the participants expected links in the “Latest Additions” section Album of the badge album to allow them to view their earned badges for that 3 Summary game. Area

Links in summary area went to unexpected places. Participants expected links for recently earned badges in the “latest additions to ’s badge album” to go to that game in the Badge Album, not to launch the Available Badges popup window (P4, P5, P6, P7, P8). Participants assumed that since they were in the album they should see information about the badges they had earned. Thus they were surprised with those links opened the Available Badges (badges.htm) popup window.

Confidential Information - Do not forward outside of MGS or MGS Partners 6 Find Usability & Playtest Reports at http://usertesting

Recommendation: Provide a way for users to view their earned badge information rather than the information about what could be earned. Consider changing the destination of the links to go to that game’s place in the badges album, not the Available Badges popup window.

9. Three of the participants wanted to view all of their earned badges in one Album place. 3 Summary Area

Some of the participants wanted to view all their earned badges in one place (P4, P5, P7). They hovered over text in the badge summary area, such as “Total Badges Won” expecting it to be hot. They expected the text to be clickable and to link to a page showing all their earned badges.

Recommendations: Consider creating a single page where users can view all of their badges, linked from the Total Badges Earned text in the badge album summary section.

Badges 10. Six of 10 participants were unsure where clicking the links in the Badge 3 Album would take them. Album

Participants were unsure where navigation links went. When recognized as navigation, participants were unsure where links went, or what they did (P1, P2, P3, P5, P6, P7).

Confidential Information - Do not forward outside of MGS or MGS Partners 7 Find Usability & Playtest Reports at http://usertesting

• Some thought navigation links went to the game (P7, P?). • Two of the participants said that the Home link went to MSN games home page (P4, P7). • Some participants expected the game name link in right pane to go to the game, not another place in the badge album (e.g. when viewing all puzzle games) (P5, P?).

Badges 11. Six of 10 participants failed to find a list of badge-enabled games. 3 Album

Some participants failed to find the game list that is found within the album (P2, P3, P6, P7, P8, P9). Lists were available only in one place. Lists were accessed by clicking the game type links in the album, or by clicking the “Back to Puzzle games” link when viewing badges for a puzzle game. Thus, the problem was exacerbated by any difficulty participants had discovering album navigation. Recommendations (some belong to above issues): • Consider grouping navigation elements together in one place (including the link to “view all puzzle games”). • Change the name of links from “Back to puzzle games” to “View puzzle games”. Many users will get to an album game page directly from an end-game screen, so saying “back to…” is confusing. • Consolidate with other album nav elements (such as “back to puzzle games”) in one area of the page. Make it look more “clickable” so it jumps out of the page as a nav element. Add tool tips.

Game 12. The badge image in the left of the game window did not look clickable. 4 Window

Clicking the badge image in the left of the game window opens the Badges.htm popup window. However, participants tended not to click in it as a way to find more information about winning game badges. One of the participants mentioned aloud that this image did not look like it was clickable (P4). Recommendation: Consider changing the image so users think it is more clickable.

13. The order of the badge types shown in the Available Badges popup Available window (Badges.htm) was switched from that shown on the badge 4 Badges Popup Album. Window

The order that the badges appeared in the Available Badges popup window was reversed from that shown in the Badges Album: Mastery Badges are show first on Badges.htm and second in the Album (see below). Mastery Badges are also referred to as Special Game Badges in the Album. As is, when there are no veteran or special promotional badges earned, the placement of the Mastery badges appears wrong – as though it is in the wrong place. Order shown in the popup window: Mastery Lifetime Special Badges

Order shown in the Album: Lifetime Mastery/Special Badges

Confidential Information - Do not forward outside of MGS or MGS Partners 8 Find Usability & Playtest Reports at http://usertesting

Recommendations: • Make the order that the badge types are the same in the Album and in Badges.htm. • Clarify the difference between, and the placement of, Special Badges and Special Game Badges.

Badges 14. Five of 10 participants expected to find a list of badge-enabled games 4 Album

Half of the participants wanted a way to determine what games had badges available, such as a page with all games listed, that appeared outside the Album and that did not require signing in (P5, P6, P8, P9, P10). While it was intuitive to go to the Album after playing a game (to view newly earned badges), it wasn’t intuitive to go to there to find a badge-enabled game to play. Recommendations: Consider adding a link on the Home page to a list of games that have badge awards associated with them. Link to this page from any special badge promotion article pages.

Badges 15. Two of the participants mentioned that the coolness of badges was 4 difficult to notice; it was limited to display in the Trophy Case. Album

The “coolness” factor of badges was hidden or less apparent because participants only saw “coolness” in the trophy case and only for the ones that had been earned (unearned badges were not shown at all in the album) (P4, P8). Combined with a lack of distinction in the appearance of the badges on the Available Badges popup window (see issue #7) participants incorrectly assumed that badges all looked like the ones shown in the Available Badges popup window. Finally, the use of question marks as placeholders for Lifetime badges was misleading. Participants seemed to think that those were what they got when they achieved that cumulative score. Recommendations: Consider the following: • Allow users to view some sort of “animations” of their earned badges outside the trophy case (e.g. in a smaller size with limited animations). • Show simplified static images of real badges in the Available Badges popup window. • Provide a more obvious teaser for unearned badges in the Album and reduce the (visual)

Confidential Information - Do not forward outside of MGS or MGS Partners 9 Find Usability & Playtest Reports at http://usertesting

compelling nature of the Lifetime score placeholders.

16. Requirements for special badge challenges were difficult to quickly Special extract from the text on the article pages. 4 Challenge articles

All five participants correctly guessed what clicking links to Badge challenges would show. All participants expected to find requirements for a badge challenge when they clicked the links to Cubis, Mah Jong and Jewel Quest. However these requirements were difficult to quickly extract from the text. More importantly, all five participants expected to find a link to play the game on the article page. All participants requested a large “Play Now” button at bottom of page challenge description page.

Recommendations: Consider changes shown in the screen shot, below, which includes: • Place something like a Play Now button at the bottom of the special challenge article pages. • Summarize the challenge requirements in a bulleted list (or the like) so users get the info fast. • Remove the “back to ” link from the link in upper right of the page. It is confusing when users come to this page from the Home Page. Alternatively, remove the text “back” and “main” to make this link less confusing.

17. All participants found and understood the meaning of badges-related My Game links in the My Game Center section of the new home page. N/A Center (Home Page)

Five of the participants (P6, P7, P8, P9, P10) viewed the new Home Page, paying special attention to the My Game Center section of the page. All five participants easily found the badge-specific area on the new Home Page. All correctly guessed where the links in this section went.

Confidential Information - Do not forward outside of MGS or MGS Partners 10 Find Usability & Playtest Reports at http://usertesting

Recommendation: Based on comments from other parts of the study, consider providing a list of badge-enabled games (and their available badges) in this section too.

Badges 18. None of the participants were able to find their badge albums from N/A outside a game. Album

All 10 participants were unable to find their badge albums unless they had just finished playing a game and were on a game end screen or were in the Learn About Badges article. Recommendation: This has been fixed with release of new Home Page (Yay!)

Issues discovered by the User Research Engineer but not experienced by participants Findings & Explanations UI Area

1. A user’s total number of games played needs to appear somewhere besides the Badges game end screen (e.g. in the badges album), as total games played is used as a Album criterion for earning special promotional badges.

Recommendation: Consider adding total games played to the game specific information pages in the badges album, somewhere near where users’ total lifetime points are displayed.

2. It was too easy for users to get an excessive number of opened browser windows. Web Site

By the end of a testing session some of the participants had 10 or more browser windows opened when they finished their study. Half of these windows were usually instances of the Album. Recommendation: Find a way to refresh and redisplay the Badge Album when links to it are clicked multiple times.

3. The ability to the first badge in a game varied with games – it was easy to earn the Game first badge in Cubis and difficult to earn them in Combo Chaos and Bejeweled 2. Window

Recommendation: Consider revisiting criteria for earning a first badge in the games.

Issues unrelated to badges UI Area Installer There were 10 participants. Of them they used: WinXP (N=7) or Win98 (N=3). • Two of the participants failed to install Active X controls. They needed to be told to click the yellow bar at the top of the browser window (P1, P2). • Two of the participants figured out how to install Active X controls after some searching or research reading support articles. Both closed the installer window and tried it a second time (P3, P4). • Three of the participants figured out how to install Active X controls after some initial confusion and reading the text on the page (P5, P6, P7)

Confidential Information - Do not forward outside of MGS or MGS Partners 11 Find Usability & Playtest Reports at http://usertesting

• Three of the participants got right away (P8, P9, P10 – all WinXP users). Participants had difficulty finding it in its current location due in part because they encountered this error before the installation dialog -- they expected these instructions to precede the “click install” explanation. The yellow bar example is also very low contrast and was difficult for participants to read.

Recommendation: Consider putting the “yellow bar” section of the installer page higher on the page and make it more visible (see illustration below). How it appears today:

Concept for making the “yellow bar” more noticeable:

Confidential Information - Do not forward outside of MGS or MGS Partners 12 Find Usability & Playtest Reports at http://usertesting

Nickname creation

1. One participant decided not to create a nickname – she had “too many already,” and assumed that she would need to remember it to sign into the site in the future. Nothing on the page stated why it was useful to have a nickname. Nothing made it clear that it was not a sign in name. Recommendation: Consider stating on the Nickname creation page why it is important or useful to have one, and that the site “remembers your Nickname for you”.

2. The Nickname page failed to make it clear that users were not creating a new Passport (P8). 3. (This was my observation only, it was not experienced by a participant) The continue button on the newsletter announcement signup page is in the wrong place. It should be below the Continue button.

How the Continue button currently appears:

Confidential Information - Do not forward outside of MGS or MGS Partners 13 Find Usability & Playtest Reports at http://usertesting

How the Continue button should appear: