Elbuen.Org, the Website of El Buen Samaritano

Final Usability Report Elbuen.org, the website of El Buen Samaritano Episcopal Mission

Henry Stokes/Prentiss Riddle INF 385P 4/21/2005

NOTE: This report was developed for a class project at the School of Information, UT Austin. It was not developed in consultation with El Buen Samaritano. Please do not redistribute.

1 STOKES/RIDDLE

INF 385P 2 STOKES/RIDDLE

How this report is structured

The usability findings within this report are offered in three levels of detail. For the reader interested in only the overall impressions, a one-page Executive Summary is offered on the next page. For the reader interested in summaries of all the findings, there is the body of the paper, in particular the specific usability findings in Section 5.3. Finally, all the detail of the methods and the raw data are provided in the Appendices, for only the reader interested in the smallest detail.

The sections are numbered as follows:

1. Executive Summary 2. Introduction 3. Methodology 4. Results 5. For more information Appendices: A. Welcome and Instructions B. Background Questionnaire C. Task Descriptions D. Post-Test Questionnaire E. Post-Test Questionnaire Data F. Performance Data

1.Executive Summary

This report describes a study of the usability of elbuen.org, the website of the El Buen Samaritano Episcopal Mission. The study was conducted by two graduate students in an introductory course in usability at the School of Information of the University of Texas at Austin in April, 2005.

The study tested the use of the site by five representative users, including both English and Spanish speakers. The participants were asked to carry out a set of tasks representative of the needs of clients, donors and volunteers at El Buen Samaritano. Both their comments and test monitors' observations were recorded as they performed the tasks.

Participants were able to complete most of the tasks and reported that they found the site easy to use, but they did report some problems, and there were some tasks which they could not complete. Specific problems include:

 Unclear navigational labeling  Unclear placement of information  Confusion regarding services branded in Spanish  Inconsistencies between English and Spanish content

We make recommendations about how to address each of these problems.

2.Introduction

2.1. Study Purpose

The purpose of this study was to test the usability of the elbuen.org website, using representative users.

2.2. Study Methods and Context

In this study we employed an “end-user test” method, wherein individual representative users were tested, one at a time. Test participants were seated at a computer displaying the elbuen.org website, were given some representative tasks to perform, and were observed as they carried out these tasks. This was a “find-and-fix” test, more interested in identifying potential usability problems than in establishing baseline performance data against which later releases could be compared.

2.3. Study Summary

Five representative users were tested one at a time on the elbuen.org website. We wished to collect performance data (error rates), satisfaction data (via a questionnaire), and, most importantly, particular areas of potentially poor usability. Through meetings with the El Buen Samaritano team we agreed on a set of tasks concerned with locating factual information on the website which would be of interest to the organization's client, donors and volunteers. Questions we hoped to answer by studying these tasks include:

 Will users in all segments be able to navigate the site and find what they are looking for?

 Will users in all segments be able to understand the content once they find it?

 Will information presented in both languages be consistent?

 Will bilingual users be able to switch easily among equivalent content in both languages?

Although we collected general satisfaction data, we did not attempt to measure the website's effect on users' attitudes toward El Buen Samaritano and its services. We recommend a separate marketing study for that purpose.

2.4. User Profile

Members of the El Buen Samaritano community come from the widest possible range of socioeconomic backgrounds. Some speak only Spanish, others only English, and many speak both with varying degrees of mastery. Many have only a primary education or no formal education at all. Many have never used a computer or the Internet and find information online second-hand with the help of friends or family. Some are recent graduates of El Buen Samaritano's own computer literacy programs.

The organizational roles of users of the elbuen.org site are varied as well. For the purposes of this study we segmented them into clients, donors and volunteers.

Because some categories of users would be hard to recruit in the general population of Austin (in particular, low- income Spanish speakers of limited education who nevertheless have sufficient computer skills to use the web) we relied on El Buen Samaritano to recruit participants for the study. We stressed to their team the importance of finding representative users and to the users the importance of being frank about any problems they found with the website. In order to ensure that participants represented a cross-section of potential users, our background questionnaire asked about languages, educational background and computer usage. In order not to raise privacy concerns we did not inquire in detail about other socioeconomic factors beyond noting participants' roles with El Buen Samaritano (i.e., clients vs. others).

3.Methodology

3.1. End-user Test Method

3.1.1. Participants

As can be seen in figure 1 below, we tested five participants. Two were volunteers, one was a donor and two were clients of El Buen Samaritano. Both of the clients had grown up bilingual in English and Spanish and continued to speak both languages at home while using primarily English at work. The remaining participants were monolingual English speakers. All participants had extensive experience with computers and used the web on a daily basis. All were college-educated.

Figure 1: Test participants

Participant Languages Computer use Education no. role At home At work Growing up How At Web? long? home?

P1 volunteer en en en years yes daily college

P2 volunteer en en en years yes daily college

P3 donor en en en years yes daily college

P4 client en/sp en en/sp years yes daily college

P5 client en/sp en en/sp years yes daily college

3.1.2. Procedure

The test was conducted in an office at the El Buen Samaritano facilities, a location chosen to meet the needs of clients who might have transportation difficulties in reaching a more neutral test location and because it offered childcare for participants accompanied by their families. The test room consisted of an office with a single computer. Because no separate observation room was available, the test moderator stayed in the room and interacted with the test participant as necessary while observing and taking notes.

Participants were welcomed and made comfortable in the test room and signed an informed consent form. They then listened as he read a scripted explanation of the test and administered a short background questionnaire (Appendices A and B). Because some participants had literacy issues, questionnaires were administered orally. If there were no questions, they were given the task list (Appendix C) and asked to interact with the website and carry out the tasks. They were encouraged to think aloud as they worked, to ask for help if necessary and to feel free to skip to the next question or leave the test at any time if they wished. Both the explanation of the test and the written task list emphasized that it was the website being tested, not the participants, and that any problems they discovered would be used to help improve the site and were not a reflection on the participants or on El Buen Samaritano. After going through the tasks the participants were given a short post-evaluation questionnaire (Appendix D), given a gratuity (an El Buen Samaritano t-shirt and, in the case of clients, $40 to cover their time, including time spent on transportation). They were escorted to their waiting families if needed and thanked for their participation. The session took less than 60 minutes.

3.1.3. Task Scenarios

We worked with the El Buen Samaritano team to derive a set of tasks (Appendix C) that would be representative of users of the website. The tasks consisted of using the site to find specific factual information of use to clients, donors and volunteers:  Specific services offered (e.g., exercise classes, immunizations).  Times and locations (class schedules, street address).  How to find more information.

4. Results

This section summarizes the data collected during the study sessions. We include three kinds of data here: 1. Performance data: participants' successes and failures at completing the tasks. 2. Satisfaction data: participants' reported enjoyment of the site. 3. Usability findings: usability problems discovered both from participants' comments and from test moderators' notes, along with our recommendations. These are summaries. More complete data can be found in the appendices, as noted.

4.1. Performance Data

For each task we recorded whether the participants' success in one of three possible states: the participant completed the task in a straightforward fashion, the participant completed the task but reported or displayed confusion in the process, or the participant was unable to complete the task. We did not record timing data.

Note that of our five participants, one chose the option open to bilingual participants of repeating the tasks in the second language. Hence the table shows a total of six trials for five participants.

Task English (4 trials) Spanish (2 trials)

Complete Completed Unable to Completed Completed Unable to d task task with complete task task with complete confusion task confusion task

1 1 3 0 0 1 1

2 3 1 0 1 0 1

3 4 0 0 1 0 1

4 3 1 0 1 0 1

5 4 0 0 2 0 0

6 0 3 1 2 0 0

Total across 15 8 1 7 1 4 tasks (62.5%) (33.3%) (4.2%) (58.3%) (8.3%) (33.3%) (percentage within language)

Table 4.1: Performance data.

4.2. Satisfaction Data

Students answered a post-test questionnaire (Appendix D) at the completion of the last task. The detailed answers are in Appendix E. Their comments can be summarized as follows:

All participants described the site as "easy" to use, despite some specific difficulties.

All participants who commented on the aesthetics of the design had positive things to say about it ("clean," "colors good," etc.)

Negative comments focused on usability issues which we address in section 5.3 below.

4.3. Usability Findings

As this study was a “find-and-fix” usability study, the findings in this section constitute the most important results in the study.

4.3.1. Positive Findings

Finding: The left menu bar was an easy way to navigate. Participants found it familiar and used it immediately with no difficulty. Finding: Users had no trouble locating contact information such as the street address. This information was placed in the footer of every page and in the 'Contact Us' section.

4.3.2. Problematic Findings

Problem: Unclear labeling

Description: Participants often did not know which category in which to find the specific information they were looking for. Names such as Health Care, Primary Needs, and Primary Care were difficult to differentiate. At times, participants stated that they could not discern the association between the desired information and the category it was labeled with. Trial-and-error "shot-in-the-dark" guessing was sometimes necessary.

In the Spanish site, some navigational labels appeared in English ("Programs/Services") and others used unfamiliar Spanish terminology ("Portal" for home).

Recommendation: Develop better terminology that users are more likely to understand. Pay close attention to the choosing Spanish labels that will be understood by speakers of multiple varieties of Spanish. Add a search feature.

Problem: Unclear placement

Description: Information was not necessarily arranged on a page in a way that made sense to users. Important information required scrolling without cues on the page to that effect. Links were sometimes made small and hidden within text blocks.

For example, donor and volunteer information appear together on one long page with no internal navigation, and to make matters worse, motivational information about existing volunteers is in the way of information about how to volunteer.

Recommendation: Find ways to isolate and clarify information, attract user's eye for minimal searching. Break up long pages or provide internal links.

Problem: Monolingual branding on a bilingual site

Description: Monolingual English-speaking participants were surprised by the names of the "Vida y Salud" and "Promotores" programs and, without knowing their meaning, had no way to know what content lay behind them.

Recommendation: Supplement Spanish-"branded" labels with additional labeling. For example: "Vida y Salud lifestyle classes".

Problem: Distracting content

Description: Participants complained that motivational material such as the staff bios and "Look Who's Volunteering" were intrusive and confusing, getting in the way of task-oriented use of the site. Recommendation: Place factual and task-oriented content more prominently. Provide visual and layout cues to help users distinguish it from background and motivational content.

Problem: Inconsistency between English and Spanish content

Description: Some items on the English site were missing from the Spanish site, such as food pantry information.

Recommendation: Perform inventory to ensure that all information available in one language is also available in the other, particularly factual and task-oriented information.

Problem: Information about a particular service is scattered

Description: Locations, schedules, and other details about a particular service were often scattered in different places. For example, participants expected to be able to click on a class such as Citizenship to see its schedule and were surprised that instead they needed to scroll to the bottom of the page and click on a separate link to a comprehensive schedule. Similarly, donor information about the food pantry was on one page, whereas information about its hours and services was on another.

Recommendation: Lump more complete information about a given service in one spot, considering what a client, donor or volunteer will need to know to use or contribute to the service.

Problem: Inconsistent navigation menus

Description: Some labels on the left-hand navigation menu had popout sub-menus and others did not.

Recommendation: Make the navigation menu consistent, particularly for sections which could benefit from popouts, such as the donor/volunteer page.

5.For More Information

If you have any questions about anything contained in this report, please contact: Henry Stokes [email protected] Prentiss Riddle [email protected]

Appendix A – Welcome and Instructions

Thank you very much for coming in today. My name is ______. My partner and I are students at the School of Information at the University of Texas. We are taking a class on how to evaluate products to see how easy they are to use. Today we are conducting a study to test the usability of a website.

I will guide you through today's session and interview you about your reactions to the site. Keep in mind, we are not evaluating you in any way, we are only interested in your experience with the site. Throughout the session, I will encourage you to freely express your opinions, to comment on what is clear and what is unclear to you, and in particular, what you find confusing or difficult to understand.

If you are ready, I will take you to the opening page of the website and give you a list of tasks to perform with the site. Please perform as many of the tasks as you can. Take your time and don't worry if you have problems. Remember, we are testing the website, not you or your abilities.

As you work, please think aloud. If you run into a problem you can ask me for help. You are also free to skip to the next question or end this test at any time.

Do you have any questions before we begin?

Appendix B - Background Questionnaire

Administered orally to accommodate participants with literacy issues.

Questions given in italic type should be asked as stated. Clarifications may be improvised as necessary. Use the space on the right to record additional comments by the participant. For checkboxes (), check all that apply.

Name: ______

Participant segment: (Clients will be identified as such by El Buen Samaritano.)  Client (For non-clients) What do you do for El Buen Samaritano?  Donor  Volunteer  Other: ______

Languages:

What languages do you usually speak at home?  English  Spanish Other: ______

What languages do you usually speak at work?  English  Spanish Other: ______

What languages did you speak growing up?  English  Spanish Other: ______

Computer literacy:

How long have you used a computer? Number: ______ Weeks  Months  Years

Do you have a computer at home?

 Yes  No

How often do you go online to use the Web?  Daily  Weekly  Monthly  Less

Education:

How many years did you go to school or what grade did you finish? (Record as stated by participant)  Years/grade ______ Elementary/primaria  High school/secundaria  College/university

Appendix C – Task Descriptions

Please perform as many of the tasks on the El Buen Samaritano website as you can. Take your time and don't worry if you have problems. We are testing the website, not you or your abilities.

As you work, please think aloud. If you run into a problem you can ask us for help. You are also free to skip to the next question or end this test at any time.

1. Find out what day of the week the Citizenship Class meets.

2. Find out whether El Buen Samaritano offers immunizations.

3. Find out who to call to discuss volunteering.

4. Find out what kinds of food are accepted for the food pantry.

5. Find El Buen Samaritano's street address.

6. Find out what kinds of exercise classes are offered.

(Optional) If you speak both English and Spanish, find the same list of exercise classes in Spanish.

Appendix D – Post-test Questionnaire

Administered orally in order to accommodate participants with literacy issues.

Name: ______

1. Would you say the website was easy or hard to use?

2. Were there parts of it which were particularly hard?

3. Was there information which you couldn't find?

4. Were there particular things about the site which you liked?

5. Were there particular things about the site which you didn't like?

6. Additional probes about specific problems:

7. Is there anything you'd like to add?

Appendix E – Post-test Questionnaire Data

Here are the participants' replies to the post-test questionnaire.

1. Would you say the website was easy or hard to use? P1: Easy for the most part; not much info to have to decide from. P2: Relatively easy. P3: Easy, but couldn't find exercise class. P4: Pretty easy. P5: Easy but I didn't find everything.

2. Were there parts of it which were particularly hard? P1: No. P2: [Trying to locate the] exercise class -- didn't get an immediate impression of [where to look]. P3: No. P4: What category to use? P5: No, but a search box would help.

3. Was there information which you couldn't find? P1: No. P2: No. P3: Yes, the exercise class. P4: The food pantry [in Spanish]. P5: Yes, food items [the food pantry] in Spanish. And I couldn't return home in Spanish.

4. Were there particular things about the site which you liked? P2: Nice, consistent pages but didn't look exactly alike. P3: It was clean. P4: Like photographs, sidebar, switching back and forth between English and Spanish. P5: The fonts were okay and the colors were good.

5. Were there particular things about the site which you didn't like? P2: Navigation bar -- not apparent that there is secondary navigation. P3. Links were hard to see.

P4: Too much white space on right. P5: The inconsistency of information between English and Spanish, broken links, and no search box.

7. Is there anything you'd like to add? P5: In this community, kids may be web guides for their parents [so the design should keep children in mind].

Appendix F – Performance Data

Raw performance data for each participant. Note that participant 5 attempted to perform each task twice, in English and in Spanish. Key: Y The participant completed the task in a straightforward fashion. YC The participant completed the task but reported or displayed confusion in the process. N The participant was unable to complete the task.

Task P1 P2 P3 P4 P5 P5 English English English Spanish English Spanish

1 YC Y YC YC YC N

2 YC Y Y Y Y Y

3 Y Y Y Y Y N

4 Y Y YC N Y N

5 Y Y Y Y Y Y

6 YC YC N Y YC Y

Additional notes on tasks:

Task 1: Find out what day of the week the Citizenship Class meets.  Participant expected to be able to click on a class (e.g., Citizenship) to see schedule. Failing that, it was not obvious to scroll down to find the schedule.  Participants were surprised that all schedules are on one (separate) page and that there is not a schedule page for each class.  Class schedule missing in Spanish.  "Programs/Services" labeled in English on Spanish site.  Participant confused by testimonial staff bios.

Task 2: Find out whether El Buen Samaritano offers immunizations.  Unclear labeling - participant had difficulty knowing which one to go to ("Health Care" or "Primary Care"?)

 Found easier because participant remembered option ("Health Care") from previous task

Task 3: Find out who to call to discuss volunteering.  Surprised by "Look Who's Volunteering" before information on how to volunteer.  Expected secondary sub-menu under Donate/Volunteer in left column.  Participant looked for a link at the top of the Volunteer/Donate page to sections of that page.

Task 4: Find out what kinds of food are accepted for the food pantry.  Would not have expected to find details of food pantry on volunteer page.  Submenu of Donate/Volunteer could include "Money", "Food".  Did not associate "Primary Needs" with food pantry.  Food pantry and list of food items missing from Spanish site.

Task 5: Find El Buen Samaritano's street address.  Answer available on footer of every page AND on Contact page (good thing)  Participant wanted a map

Task 6: Find out what kinds of exercise classes are offered.  English speakers surprised by "Vida y Salud" in Spanish (monolingual branding on a bilingual site)

INF 385P 22