Imdb.Com Usability Test Kit HCDE 417 Fall 2016

IMDb.com Usability Test Kit HCDE 417 Fall 2016

Group 4 Andy Truong Karin Vaughan Lydia Davison Monica Lee

November 8th, 2016

Introduction Our group plans on investigating IMDb, short for Internet Movie Database. IMDb is an online database that provides a myriad of information for entertainment such as movies, TV shows, and celebrities. IMDb provides information on ratings, plot summaries, and reviews. IMDb can be accessed online or through their mobile app.

Purpose

IMDb can help users find a movie they may be interested in watching. Our objective is to study and improve the usability of navigation and make the information architecture and categorization clear and consistent to create a logical and efficient flow through the website. For the purposes of this study, we will be testing the usability of the website through the desktop version of IMDb. Table of Contents

Pre-Screening Background Questionnaire 2

Consent Form 6

Introductory Script 7

Study Script 8

Pre-Test Questionnaire 10

Task Scenarios 12

Post-Task Questionnaires 14

Post-Test Questionnaire 15

Data-Logging Forms 17

Pre-Screening Background Questionnaire [Note: The survey will be executed using Google Forms and Facebook.]

Title for Facebook post: Enter for a chance to win a $20 Amazon gift card! We need your help understanding movie watching habits.

We are a team of students at the University of Washington seeking participants for a usability study on movie watching habits. If you are interested in helping us, please fill out this short screening survey below. Respondents will be contacted based on their compatibility with the study’s intended user group.

Please note that this study is for educational purposes only, and participation is voluntary. You must be over 18 years old. The study will take place at the University of Washington Seattle campus and will require a commitment of approximately one hour. Refreshments will be provided and you will be entered into a drawing to win a $20 Amazon gift card.

Video and audio will be recorded during the test session. All personal information will remain confidential.

1. Please provide us with your name and contact information. We will contact you by email if you are selected to participate in our study. Name: Email: Phone (optional):

2. Preferred gender: ___ Female ___ Male ___ Prefer not to answer

3. Age: ___ 18-24 ___ 25-34 ___ 35-44 ___ 45-54 ___ 55-64 ___ 65-74 ___ 75+

4. Have you ever gone online to look up information about movies, TV shows, or actors? ___ Yes ___ No

a. If Yes: i. How often do you use a laptop or desktop computer? ___ Daily ___ Once a week (a few times a month) ___ Once a month ___ A few times a year ___ Never or N/A

ii. How often do you use the following sites? Please rate each site from 0-4. (0 is never and 4 is very frequently). ___ Wikipedia ___ Fandango ___ IMDb ___ Online Newspaper ___ Rotten Tomatoes ___ Other (please specify): ______

iii. Do you contribute content about movies, TV shows, actors, or entertainment to the following sites? ___ Wikipedia ___ Fandango ___ IMDb ___ Online Newspaper ___ Rotten Tomatoes ___ Other (please specify): ______

b. If No: i. Are there information sources, other than online, that you use to look up information about movies, TV shows, or actors? ___ Yes ___ No

1. If yes, please explain below:

a. How often do you use these sources? ___ Daily ___ Once a week (a few times a month) ___ Once a month ___ A few times a year ___ Never or N/A

b. How often do you use a laptop or desktop computer? ___ Daily ___ Once a week (a few times a month) ___ Once a month ___ A few times a year ___ Never or N/A

5. Please indicate which days and times you’re available (drop down calendar) ___ Tuesday 11/15/16, 9:00am ___ Tuesday 11/15/16, 10:00am ___ Tuesday 11/15/16, 11:00am ___ Tuesday 11/15/16, 12:00pm ___ Tuesday 11/15/16, 1:00pm ___ Tuesday 11/15/16, 2:00pm ___ Tuesday 11/15/16, 4:00pm ___ Tuesday 11/15/16, 5:00pm ___ Tuesday 11/15/16, 6:00pm ___ Wednesday 11/16/16, 5:00pm ___ Wednesday 11/16/16, 6:00pm ___ Thursday 11/17/16, 9:00am ___ Thursday 11/17/16, 10:00am ___ Thursday 11/17/16, 11:00am ___ Thursday 11/17/16, 12:00pm ___ Thursday 11/17/16, 1:00pm ___ Friday 11/18/16, 12:00pm ___ Friday 11/18/16, 1:00pm ___ Friday 11/18/16, 2:00pm ___ Friday 11/18/16, 3:00pm ___ Saturday 11/19/16, 9:00am ___ Saturday 11/19/16, 10:00am ___ Saturday 11/19/16, 11:00am ___ Saturday 11/19/16, 12:00pm ___ Saturday 11/19/16, 1:00pm ___ Saturday 11/19/16, 2:00pm ___ Saturday 11/19/16, 3:00pm ___ Saturday 11/19/16, 4:00pm

___ Saturday 11/19/16, 5:00pm

Termination Script:

Thank you for taking the time to complete our survey. We appreciate your willingness to participate in our usability sessions. If you meet our participant criteria, we will contact you within the next 72 hours to confirm the time for your usability session.

Best UW Students

Consent Form

I agree to participate in the study conducted by the IMDb.com Usability Test Group at the University of Washington, Human Centered Design and Engineering Department.

During this study: ● I will be asked to perform certain tasks on a computer ● I will be interviewed regarding the tasks I’ve performed ● I will be recorded through audio and video during the session

I understand and consent to the use and release of the recording by IMDb.com Usability Test Group at the University of Washington, Human Centered Design and Engineering Department. I understand that the information and recording is for research purposes only and that my name and image will not be used for any other purpose. I relinquish any rights to the recording used by IMDb.com Usability Test Group at the University of Washington, Human Centered Design and Engineering Department without further permission.

Participation in this usability study is voluntary. All information and recordings will remain strictly confidential. The descriptions and findings may be used to help improve the IMDb.com website design. At no time will my name or any other identification be used. I have the ability to withdraw consent to the experiment and stop participation at any time.

Below is my signature indicating that I have read and understood the information on this form and that any questions I might have about the session have been answered.

Date: ______

Participant’s printed name: ______

Participant’s signature : ______

Thank you!

We appreciate your participation. 6

Introductory Script [Before reading Introductory Script]

❏ Welcome participant to testing area ❏ Have IMDb.com opened in a Google Chrome window on the laptop ❏ Clear browsing data

Start time: ______

“Thank you for taking time to participate in this study, we really appreciate it.

I’m and this is . We are students in the Human Centered Design and Engineering (HCDE) major at the University of Washington. will be just be taking notes today and I will be facilitating the study.

We are studying the usability of the IMDb website. IMDb is a popular online movie database to find more information about movies and TV. During this study, we will ask you to complete 3 tasks on the website. You may take a break at any time. Please remember that we are testing the site, not you. Any difficulties you may have are because it wasn’t designed in a way that makes sense to you.

During the tasks, I can clarify what you are asked to do, but otherwise will remain silent. Try to complete the tasks as if you were at home. You can spend as little or as much time you want on each of them. Again, we are not testing you, we are testing the site. Also, as we go along the tasks, I'm going to ask you to think out loud as much as possible. By that, we mean that we’d like you to speak your thoughts as often as you can so that we know what’s going through your mind.

If it’s okay with you, we will be taking screen, audio, and video recordings for a better understanding of behavior on the website. Please take a look at this consent form and sign if you agree with the conditions.”

❏ Give participant the consent/video form to sign ❏ Once signed, start up Lookback and Skype on the laptop ❏ Connect with the note-taker’s laptop and set up Skype’s screen-sharing

Before we begin, do you have any questions?

Study Script [Before reading Task Scenarios]

❏ Read Introductory Script ❏ Read Think Aloud Warmup Script

Okay, now we can get you started on this laptop here. On the laptop screen, you will see the homepage of the IMDb website.

❏ See table below for each participant’s task order

Participant Task Order

P1 - new user 1, 2, 3, 4

P2 - new user 2, 1, 3, 4

P3 - infrequent user 1, 2, 3, 4

P4 - infrequent user 2, 1, 3, 4

P5 - somewhat infrequent user 1, 2, 3, 4

P6 - somewhat infrequent user 2, 1, 3, 4

Now, take a few moments to observe the IMDb homepage. Feel free to scroll around, but please do not click on anything just yet. ● What do you think this website is for? ● What do you think you can do on this page? ● What strikes you about this site?

❏ Give the participant 2 - 3 minutes to explore the homepage and questions

Thank you. Now I’ll proceed to ask you to try doing some tasks on this website, and I will read each task out loud. Again, as much as possible, it will help us if you can try to think out loud while you perform these tasks so that we know what you’re thinking about.

❏ Read task scenario out loud ❏ Allow participant to proceed until they accomplish the task or they get very frustrated ❏ After each task, ask participant to complete the post-task questionnaire ❏ Ask any additional questions based on your observations

❏ Repeat for each task or until time runs out

Thank you, that was very helpful.

Now that we’ve finished exploring the IMDb website, I have a brief questionnaire that I would like you to complete. The information that you provide is for our use only, and your name will not be stored with the questionnaire data.

❏ Have participant fill out Post-Test Questionnaire

Now that we’re done, do you have any comments or questions for me about today’s session?

❏ Remind participant about the incentive for participating in this usability test (they’re entered in drawing for $20 Amazon gift card) ❏ Stop recording on Lookback, end Skype screen-sharing, and save the files ❏ Thank participant and escort them out

Pre-Test Questionnaire Participant #___

1. How often do you watch movies or TV shows? ___ Only a couple times a year ___ Once a month ___ Once a week ___ Every day

2. What helps you determine what movies or TV shows to watch? Please explain.

3. Have you used IMDb before? ___ Yes ___ No

If yes:

How often do you use IMDb? ___ Every day ___ About once a week ___ Once a month ___ A few times each year ___ Never

When was the last time you used IMDb? ___ This week ___ About a month ago ___ 2-3 months ago ___ More than 3 months ago

What do you use IMDb for? ___ Finding movies in theaters now ___ Finding movies/TV shows related to ones you like ___ Watch movie trailers ___ Looking up the cast of movies/TV shows ___ Finding trivia information about movies/TV shows

___ Reading movie reviews ___ Find the latest news ___ Other: Please explain below

What device or platforms to you check IMDb on? ___ Phone browser ___ Mobile application ___ Tablet ___ Laptop ___ Desktop

In what environment do you usually look at IMDb? ___ At home ___ Outside of the house (mall, school, etc.) ___ Alone ___ Around other people ___ Other: Please explain below

Do you prefer IMDb over other movie databases like Rotten Tomatoes or Wikipedia? Explain:

If not, we will ask:

a. What do you think it can be used for?

b. What do you use to look at movies and entertainment?

Task Scenarios Think Aloud Warmup Script “During this study, we ask that you think aloud as you complete each task. Tell us what you like or dislike, what frustrates you, what you’re looking for, etc. To get a better idea of the level of detail in thinking aloud that we would like to hear, I have a short think aloud warmup: count how many windows are in your house or apartment, and walk me through your process. I’m not really interested in how many windows you have, but I am interested in how you go about doing this task.”

If level of detail is satisfactory, say, “Great, thank you. It would be really helpful if you walked me through your thinking during the tasks with the same level of detail.”

Task #1

Scenario It’s Halloween night, and you and your friends want to stream a horror movie on your laptop, but cannot decide which movie to watch. Find out what kinds of horror movies there are on IMDb.

Starting State IMDb.com homepage

Steps 1. Find the horror genre list 2. Find a movie within the horror genre that has at least a 7.0 rating 3. Select a horror movie to watch 4. View the plot of the movie 5. After you’ve looked over the movie’s page, go back to the IMDb homepage

Goals See if the navigation of the website is intuitive and observe the different ways a user can navigate to complete the task.

Task #2

Scenario Earlier, you and your friend watched “The Conjuring 2” for Halloween night. You’re now trying to find another movie to watch and you remembered that you really enjoyed the actress who played Judy Warren in the movie. Find another movie with that actress in it.

Starting State IMDb.com homepage

Steps 1. Find the actress who plays Judy Warren 2. Find another movie with that actress 3. Select the movie 4. View the actress’s character in that movie 5. Once you’ve viewed the actress’s new character, go back to the IMDb homepage

Goals Assess the content organization of the website and see if the navigation is intuitive for users.

Task #3

Scenario You heard one of your friends talking excitedly about a movie called “The Nice Guys.” You think it sounds interesting, so you try to look it up yourself to see what’s so good about it.

Starting State IMDb.com homepage

Steps 1. Find the movie “The Nice Guys” 2. Look at the reviews of “The Nice Guys” 3. View what genres the movie is labeled as 4. Based on the information provided, decide if you want to watch “The Nice Guys” by choosing to watch the movie on Amazon video 5. After you’ve looked over the movie’s page, go back to the IMDb homepage

Goals 1. Assess the detailed information for “The Nice Guys” to get an understanding for why your friend may have found it interesting. 2. Assess the ease of use of the search feature of the website and observe the types of information users look for in a movie.

Task #4

Scenario Think of a movie that you recently saw. Find a movie similar to it in order to help you decide on what movie to watch next.

Starting State IMDb.com homepage

Steps 1. Find the movie that was recently watched 2. View the information on that movie’s page 3. Find a movie similar to the movie that was recently watched 4. View reviews of that movie

Goals Assess how easy it is to find related movies.

Post-Task Questionnaires After each task is completed, we will ask participants to answer the following questionnaire:

Please indicate your level of agreement/disagreement with each of the following statements. Statement Strongly Disagree Neither Agree Strongly Disagree Agree or Agree Disagree

I was able to complete the task without having to remember the information on previous screens.

I felt confident in my ability to understand where I was within the website.

I felt confident in my ability to locate the information I needed.

The navigation bar was helpful in completing my task.

Post-Test Questionnaire Post-Test Questions

Once the think-aloud testing has been completed, will we ask users to fill out the post-test questionnaire and ask several questions about their overall experience using IMDb:

Please indicate your level of agreement/disagreement with each of the following statements. Statement Strongly Disagree Neither Agree Strongly Disagree Agree or Agree Disagree

I was able to complete the tasks without having to remember the information on previous screens.

I felt confident in my ability to understand where I was within the website.

I felt confident in my ability to locate the information I needed.

The navigation bar was helpful in completing my tasks.

The section headers made the content easy to identify.

I feel the content is organized in a way that makes sense.

How satisfied are you with the overall experience of the IMDb.com website?

1 2 3 4 5 Not at all Slightly satisfied Moderately Very satisfied Completely satisfied Satisfied

After participants fill out the questionnaire, we will also ask them several questions including: ● How did you feel about the tasks overall in terms of difficulty? ● Is there any areas, specifically, where you struggled? ○ How do you think IMDb can be improved to help you with this? ● If users previously said “yes” to knowing about IMDb in the pre-test questions, we will ask: ○ How has testing IMDb changed your view of IMDb? ○ How do you feel about IMDb as a whole?

○ Has your opinion of IMDb changed now that you’ve tested out the desktop site? ● If users previously said “no” to knowing about IMDb in the pre-test questions, we will ask: ○ Now that you’ve heard about it, do you think you would use it in the future? ■ If so, how would you use it? ■ If not, why not? ● What was the navigation through the website like? Please describe your experience.

Data-Logging Forms

Tasks Successful Number Number Number Number Number of Number of Completion of of of times of times Positive Negative (yes/no) Clicks Errors Backtrack Search Comments Comments Function used