Subject: Usability report on TripAdvisor

TripAdvisor Marketing and Design team,

We hope things have been well. Attached to this email is our usability study report on TripAdvisor.

The purpose of this report is to present our findings from our usability study on TripAdvisor. This report will first introduce the goals of our study, introduce our team, and go over the goals of this study. Next, this report will cover our methods used during our research study and the metrics our team used. Then, this report will go over our key findings from our usability study.

We hope that the findings of this study can be used in order to help solve the problem of increasing purchases from TripAdvisor. If you have any questions, comments, or concerns about our report, please feel free to contact us.

Thank you,

Madeleine Le Tre Paolini Gabby Bilka Aylee Neff

TripAdvisor 1

Usability Study on TripAdvisor

December 11th, 2018 [email protected]

Team 9 TripAdvisor 2

Table of Contents

Author Contact Information 3

Executive Summary 4

Introduction 5

Methodology 6

Metrics 7

Results 8

Conclusions 15

Back Matter 16 Screening Questionnaire 17 Consent Form 20 Scripts used during facilitation 21 Task Scenarios 25 Post Test Interview 28 Data logging forms 29

Final Reflection 45

TripAdvisor 3

Author Contact Information

For any questions about this study, please contact the authors of this study using the emails below.

Team Email [email protected]

Gabby Bilka [email protected]

Tre Paolini [email protected]

Madeleine Le [email protected]

Aylee Neff [email protected]

TripAdvisor 4

Executive Summary

Background TripAdvisor is an online travel platform intended to help users plan future trips by providing information about different countries and cities. TripAdvisor allows users to book hotel rooms, flights, restaurant reservations, attractions, and guided tours off of their platform. TripAdvisor also allows users to view curated guides created by other users and look up additional information about the location that they are traveling to. As of late, TripAdvisor’s marketing and design team are trying to figure out how they can increase sales conversions in order to meet higher sales goals, by encourage users to buy through the website.

Purpose The purpose of this study was to determine if there were any specific interactions that were preventing users from making purchases on TripAdvisor and to evaluate how efficiently a user can find appealing experiences and purchase them. The research questions that were focused on in this study, are “Are users able to efficiently find experiences that appeal to them and purchase them smoothly?” and “Which interactions are causing the most frustration for users?”

Major Findings Participants enjoyed using TripAdvisor to find information about their destination, due to the amount of filters and the high amount of aggregated information on the website. One participant said that they liked how TripAdvisor provided them with a wide variety of options, even though the participants had to go to another website. Participants also liked that the site would remember previously entered information like travel dates, location, and number of people travelling, which made completing tasks easier for them.

One finding that we found was that participants did not like it when TripAdvisor redirected them to another website, such as Expedia. When redirected to another website, they would sometimes lose information previously inputted, and felt like they had to do tasks twice. Another finding we found was that participants reacted negatively to pop­ups, often closing them immediately. Participants preferred that there were little to no pop­ups that appeared on the page, as they felt it detracted from their experience using the website.

One other finding that we found in our usability study, was that TripAdvisor had an unclear navigation. During one of our tasks, participants were unable to find the guides tab, due to the fact that the sidebar, main cards, information in the cards, pop­ups, and top navigation bar were all fighting for the same amount of attention. Finally, one of the biggest findings was that we found in our usability study was that participants were overwhelmed with the layout of the information on some pages, such as the flights page. However, participants did like that when TripAdvisor provided them with all information on one page, specifically on the travel guides and things to do pages. TripAdvisor 5

Introduction

TripAdvisor is a online travel platform that is intended to help users plan their next trip, by providing information about their intended destination. TripAdvisor allows users to browse and book flights, hotels, tickets to attractions, tour package tickets, and restaurant reservations though it’s website. TripAdvisor also allows users to view guides made by other users, look up topics about their intended location on their travel forums. TripAdvisor’s main source of revenue is selling tour packages tickets, tickets to attractions, hotel reservations, and flights on their platforms. Currently, TripAdvisor’s marketing and design team wants to determine how to increase purchases on TripAdvisor’s platform to meet higher sales goals at the end of the year.

The goals of our study was to see if there were any specific interactions that were preventing users from making purchases on TripAdvisor and to evaluate how efficiently a user can find appealing experiences and purchase them. This study intends to help solve the problem of increasing purchases on TripAdvisor’s platform by answering the research questions, “Are users able to efficiently find experiences that appeal to them and purchase them smoothly?” and “Which interactions are causing the most frustration for users?”

Our testing team is composed of four UX researchers, Madeleine Le, Aylee Neff, Tre Paolini, and Gabby Bilka. All four UX researchers were in charge of creating the methodology and the materials needed for this usability test. Each of our team members took turns being the moderator and a note taker during each test, moderating at least one study, and taking notes during at least one test.

Some unique terms that are in this study is the term, “traveled”, “travel”, and “itinerary”. In the context of this study, the terms “traveled” and “travel” are defined as going to a location that requires users to fly on a plane to get there. The term “itinerary” is defined as a list of attractions and/or tours that users want to go to.

This document will first go over the methods our team used during our study, which includes participant recruitment, the study procedure, and the list of tasks done. Next, this document will go over the metrics that our team used. Then, this document will cover the results of our study, which includes design recommendations. Finally, this document will conclude with next steps for TripAdvisor’s marketing and design team and final recommendations that go beyond what this study covers.

TripAdvisor 6

Methodology

Participants were recruited via an online survey posted in social media outlets; such as, Facebook. To qualify for this study participants had to be over the age of 18, did not work in the travel industry, and did not traveled in the past month. Our study included 10 participants between the ages of 18 and 23. Six were male and 4 were female. A table with the questions from our recruitment survey and the answers of the participants we reached out to can be found in the appendices.

Our sessions took place in private study rooms in the Odegaard Library and Foster Business Library. Each session consisted of a participant, a moderator, and a notetaker. The moderator sat across from the participant and the notetaker sat beside them. Sessions were conducted on laptops provided by the research team to prevent any personal participant information from being accidentally shared. The moderator was responsible for guiding the participant through the session. They reminded the participant to think aloud when necessary, provided hints if the participant was struggling, and answered questions. The notetaker recorded participant behaviors such as what they clicked on and responses to interview questions. Notes were taken free form to allow flexibility. We used the UserZoom software to record our sessions so we could refer back to them after the session. UserZoom allowed us to record both the participant's and moderator’s screens, facial expressions, and audio feedback.

Participants were greeted upon their arrival and invited to make themselves comfortable. The moderator then read the consent form aloud and passed it to the participant to look over and sign. We then answered questions if there were any before briefing the participant on the session to follow. If there were no questions regarding the session then we proceeded to the pretest interview. Participants were first asked to describe the last time they traveled and how they made those arrangements before diving into the tasks. They were asked to compete five tasks including: booking a flight, booking a hotel room, finding three activities to do in a day, finding a three day premade itinerary, and making a reservation at a restaurant with good service. After each task, participants were asked to rate statements on a 1­5 scale which can be seen in the test kit in the appendices. Once the test was finished participants filled out a short post test questionnaire and engaged in a post test interview to review their overall experience using TripAdvisor. We conducted this test to explore the following research questions in table

TripAdvisor 7

Metrics

We collected several types of data during each usability test to get a full picture on our participants’ experiences using TripAdvisor.

We collected both qualitative and quantitative data throughout the process of our usability tests. We compiled the qualitative data into an affinity diagram and our quantitative data into a table with calculated averages. Both of these can be found in the appendices. The results of this data is discussed in the result section of this report.

The qualitative data was collected from each participant during a pre­test interview, note­taking during each task, five post­task interviews, and a post­test interview. The exact questions asked can be found in the test kit in the appendices. In the pre­test interview, we tried to learn more about the participant by asking them about their most recent traveling experience and their experience using online services to make accommodations while traveling. During each task, we took tons of qualitative notes about what the participant was “thinking­aloud”, what they were doing during the task, and what types of errors they encountered. After the participant completed each task, we asked them about how they felt during the task and how it compared to how they would normally perform the task. After the test was finished, we asked the participant some overall question about their experience using TripAdvisor, like question about which task was easier or harder and which features they would like to see changed.

The quantitative data was collected from each participant via five post­task questionnaires and a post­test questionnaire. After each task, we asked participants to rate their satisfaction with the ease of use, length of task, and available information to complete the task on a 1­5 scale. After the test was completed, participants were asked to rate several statements about their experience, such as if they would use TripAdvisor again, on a 1­5 scale. These statements can be found in the test kit in the appendices. Error counts during each task were also recorded to measure which tasks caused the most difficulty to participants. Errors included wrong button clicks and backtracking.

TripAdvisor 8

Results

Analysis After finishing our usability tests, we cleaned up and compiled the notes from each session. These notes can be found in the appendices. We broke these notes down even further by doing two separate analyses for the qualitative and quantitative data we gathered. The combination of these analyses led us to the main findings of this study.

For the qualitative data, we performed an affinity analysis to group similar into more understandable categories. To start this analysis, we used a program called R eal Time Board t o create many virtual sticky notes based on all of our qualitative notes. With all the different notes together, we started matching quotes, actions, and problems that all addressed the same feature/issue together. Soon, we had several groups that each had a different label and our affinity diagram was complete. The full diagram can be found in the appendices

For the quantitative data, we compiled the data we collected based and moved it all into a spreadsheet to analyze it. For each task, we took the average of each participants answer in the post­task questionnaire to see which tasks were the easiest to use, most efficient to perform, and held the most valuable information. We also took the average of all each post­test question to see how participant felts about different aspects of the overall Tripadvisor experience. Lastly, we took the average of the amount of errors each participant had for each task to see which tasks caused the most issues for participants. The results of this analysis and the affinity analysis were used to discover our key findings and recommendations.

Detailed Findings The following table presents our usability findings synthesized from our qualitative and quantitative data. It is organized and ranked by severity. Each finding is supported by participant quotes, reactions, and ratings in addition to our heuristic evaluation outcomes.

Severity Scale N/A – Positive/neutral finding 1 – Users are unable to complete the task and are completely stuck 2 – Users are confused and struggling, breaking the flow of the task 3 – Users may still be struggling and the task took longer than should have 4 – Cosmetic issues, didn’t necessarily inhibit the task but caused participants to hesitate or express their discomfort

TripAdvisor 9

Positive/Neutral Results Finding & Evidence Recommendation Severity

Use of Filters to Specify Search Criteria We recommend to keep N/A Throughout the website, when finding specific hotels, the amount of filters and flights, things to do, or restaurants (Tasks 1­3, 5), users are make them more given the option to filter their search results by numerous noticeable. They are in criteria. When the filters were noticed and applied, users the proper location on were pleased with how effective and comprehensive they the page however the were in narrowing down their results to exactly what they sheer number of them requested. can be overwhelming and cause users to Evidence overlook them. When completing Task 2, booking a hotel, in every instance, if a participant didn’t notice the top result matching their hotel criteria, they would select the sidebar filters to find a suitable room. Additionally, Tasks 1­3 and 5 had similar ease of use ratings in the post task interview ranging from 3.5 ­ 4.15 while Task 4 only had a 3 average, potentially influenced by the use of filters to find what they needed.

Influence of Photos We recommend to N/A Throughout the website, there are photo feeds and continue focusing on carousels showcasing the features of that specific attraction pictures to showcase ­­hotel, thing to do, restaurant. With their new redesign the attractions and their front page is focused on user submitted photos. appeal to draw in Participants were either neutral or positive towards all of younger, social media these photos throughout the site and they influenced their fueled audiences to decision on booking or ignoring an attraction. TripAdvisor.

Evidence Throughout P3’s usability test, they often remarked how the pictures are what made them choose the restaurant or hotel to go to. Additionally, they use sites like Instagram often and want to go to places to take pictures and share them. Additionally, tasks 2, 3, and 5 were ranked much higher than tasks 1 and 2 when asked about satisfaction of amount of information to make decisions, 4­4.5 average instead of a ~3.7. This could have been due to the fact that these were the tasks where pictures accompanied information when booking.

Informative Reviews We recommend to N/A During tasks 3 and 5, things to do and restaurants expand the reviews respectively, there was a similar UI when displaying functionality across attraction or restaurant information. In this style of interface hotels and flights as TripAdvisor users are encouraged to leave reviews and well. Especially for photos describing their experience at that location. hotels, other travelers’ Generally, these reviews were deemed helpful and individual experiences informative by participants when they were booking events. are influential in the decision making TripAdvisor 10

Evidence process. The UI in the When completing task 5, participants were instructed to things to do and check to see if the restaurant had good service before restaurants section is booking it. To complete this part of the task participants sufficient for this looked at either the service ranking overview at the purpose. beginning of the reviews section or at the first few individual reviews listed in the section. They trusted the comments displayed on the site and no participant looked past the first page of reviews even if there were numerous.

Remembering information across site, continuity We recommend N/A While completing tasks, participants would enter in their focusing on continuity travel information in one aspect of the site, say the not only on the destination and travel dates when booking flights, and TripAdvisor website but TripAdvisor would remember that information when they also when transitioning went to book a hotel or find things to do. While in users onto different TripAdvisor, the site was relatively consistent in holding booking sites. While on onto relevant data and showing suggestions for events TripAdvisor users are based on that. For our scenario, our participants were accustomed to only travelling to Prague from Dec. 17 to Dec. 21 and while they entering their were completing the tasks after the first ­­ hotel information once and reservations, things to do, and restaurant reservations ­­ became frustrated when the site would remember that information. having to reenter everything when going Evidence to a different hotel or Our participants were satisfied with this feature (if they flights website. noticed it and commented on it) as it reduced the effort it took to complete the tasks and go through the booking process. When evaluating this with Nielsen usability heuristics in mind, this feature is a great example of recognition rather than recall in that users wouldn’t have to recall what information they’ve entered previously and redo it.

Trust of known sites, familiarity Talked about more in N/A When booking hotels or flights through TripAdvisor, the the “redirecting to actual booking process occurs on other partner sites like another site” finding, we Expedia or Kayak. TripAdvisor acts as an aggregation site would recommend to find the booking that makes the most sense for the when transitioning from users’ criteria. This allows users to compare prices and TripAdvisor to another offerings to make the best selection for them. site, users should be guided more Evidence transparently to One participant commented in the post interview that they decrease confusion and liked the detailed information there even if they didn’t look surprise. at it. Often in their pre and post interviews participants would talk about using websites that TripAdvisor connected to to make travel plans, explaining how they were already familiar with these sites. For participants, knowing that the information was there was reassuring, even if they didn’t explore it. TripAdvisor 11

Potential issues Finding & Evidence Recommendation Severity

Unclear Navigation We recommend to simplify the overall 1 When navigating through TripAdvisor there page design to reduce the amount of are many ways to get to the same page as navigation elements on each page. well as many navigation options in general. Design wise we would also recommend In the image below you can see the multiple constructing hierarchy in different navigation bars and dropdowns: elements on the page using contrast, color, and size. One participant mused about the simplicity of AirBNBs user interface in how there were clear call to actions and only one navigation bar. Below is a low fidelity wireframe showing a less cluttered interface: Figure of hotel navigation options

Throughout Task 4 in particular, finding a travel guide, many participants struggled with finding the correct section due how to it was the only task that didn’t have a high priority navigation button. In fact, Task 4 was the only one where participants failed completed, two out of the ten were unable to find the travel guides and gave up.

Additionally, during our heuristic evaluation Figure of less cluttered potential interface we noticed that the visual hierarchy on the pages was weak once in a specific page like hotels or flights. The sidebar, main cards, information in the cards, popups, and top nav bar were all fighting for the same amount of attention which aided in limiting navigation.

Disabled button We recommend that this issue be 1 P10 discovered a serious bug during Task immediately tested and fixed, as it can 1: On the first flights page, P10 chose to completely stop a user from continuing compare prices on every site and put in an a task or making purchase. If it is not email to receive price tracking updates. The easily fixed, we believe the “compare combination of using these two features prices on different sites” option should disabled the “search flights” button which is be removed as participants seldom required to move forward in this task. There used it correctly and some even said it was also no feedback when clicking the was not helpful to them (see disabled button, so P10 could not figure out Information Overload). what was wrong and searched for several other ways to move forward to no avail.

Redirecting to another site We recommend being very visible 2 In Tasks 1 and 2, participants were about the process of redirecting to a redirected to other websites after choosing new website. The flights page generally TripAdvisor 12

a flight/hotel to confirm and pay for their did this well by opening a transition bookings. Multiple participants, namely P2, page that told the user that they were P4, P5, P8, and P9, were vocal that they did being redirect to a partner site before not like being redirected to another site the actual redirection happened. The because they redo adding some information hotels page does not have this feature. or choosing flights: “I felt like I did We believe this could be added to the everything twice” (P2 during Task 1). hotels page as well, which would help reduce confusion in participants when they are inevitably redirected.

We would also recommend trying to make sure any information that users put in on TripAdvisor is automatically filled in on the website the user is redirected to. Some websites did this better than others, but it is likely that TripAdvisor cannot control which sites support this feature most effectively. To fix this, we believe that the user should be made aware of what information they are going to have to re­input on the new site via the transition page. This way, users are not confused when the information they already put in is missing on the new website.

Information Overload We recommend changing the layout of 2 Participants generally liked the way that the various pages on the “flights” TripAdvisor provided and displayed section of the website. Specifically, we information, with several quotes and higher believe looking at other comparable than average ratings that supporting this: website designs, such as google “liked that it gave me every option there is, flights, or looking at the design of the even through different websites” (P10). more highly rated sections of the However, the flights section of the site website, like the travel guides and proved to cause the most difficulties for things to do sections, for inspiration. participant with how it displayed information. We also recommended completely The “compare prices” page of the site is refactoring the design or outright opened in the background after searching removing the “compare prices” page, for flights, which caused most participants to since it was mostly ignored by ignore it completely and just use the main participants, except for when it was “flights” search page. The participants that criticized. We believe it could be fixed did notice it did not utilize it and stated that it by only showing two tabs of information was not helpful: “This page I’m looking at at once, so that the text and features of with the comparisons in not very helpful ... each tab are not stacked on top of It’s just information overload” (P4) (see each other so closely. Further tests Figure 1). would show if this feature is worth keeping or should be removed. TripAdvisor 13

Figure 1: Compare prices page

The main flights page also caused issues for participants, with P2 and P8 being vocal about how overwhelming the information is: “So there is a lot of information on the page that is a little difficult to take in all at once” (P8) (see Figure 2). The flights task had the lowest average rating (3.68/5) for having relevant information available and the lowest average ranking for reasonable length of time taken to complete (3.65/5).

Figure 2: Main flights page

Unexpected price hikes We recommend that TripAdvisor 2 choose to only partner with booking P1, P5, P6, P7, P9, and P10 all noticed sites that do not randomly increase prices on the partner sites being higher than their prices. This will increase user trust they were on TripAdvisor, some by a with TripAdvisor and cause less considerable amount and some by a small confusion for users of the site. At the amount: “Trip price increased by 99 cents” very least, partner sites that engage in (P6). this practice should not give TripAdvisor a price that is significantly Upon further inspection, it appears some lower than the price that they actually partner sites will have different prices for charge on their own site. certain flights and hotels whenever you visit them, even if you just open another window and put in the exact same url (even without changing any cookies/cache settings). This practice is likely used to squeeze a bit of extra money out of users without them noticing. TripAdvisor 14

Pop­ups We recommend utilizing design 3 Pop­ups appear throughout the with elements besides pop­ups to attract participants interacting with one as soon as users attention to certain parts of the they start doing tasks on the homepage. In website. Pop­ups were not useful to all cases that involved pop­ups, participants any participants and only resulted in would ignore any content that was on the negative reactions, so we believe that pop­up and close it quickly. P2, P4, P5, and any information that was conveyed P8 were all vocal about disliking the through pop­ups would be better pop­ups and not finding them useful: “I hate placed in a prominent way on the main pop­ups, don’t give me pop­ups.” (P4). website page.

Maps We recommend emphasizing and 3 Map features are available on various tasks distinguishing the map feature as an throughout the TripAdvisor website, but option when it is available. Currently, it were only utilized by a handful of is housed in a small box in the top right participants. Participants who did use the of the pages it is available in (see maps, P5, P6, and P9 found them to be Figure). We believe it the box size and important: “I liked having a map to see the text size of “View map” button where all the locations were”. However, P6 should be increased so users are more and P9 ran into issues using the map and likely to click on it. felt it could be improved: “How they do the map is terrible, location is the biggest thing in a different country” (P6).

Figure: Current placement of map button

We also recommend further testing of the map feature specifically to find any usability issue or bugs that could be fix to improve the feature’s overall experience.

Preferring saving landmark, not buying We recommend adding system to save 4 it activities without requiring users to When finding things to do, some create an account to do so. Since the participants would keep track of their “add to cart” option is available without chosen activities by “adding to cart”. For an account and proved to be useful for some activities, this was a useful way of participants, we believe it would be remembering what to do and having a beneficial to create a similar system for convenient way to pay for them eventually. activities that cannot be added to cart. But, some activities did not have an option Another option would be to create an to add to the cart, and users would have to “itinerary” system that would allow create a TripAdvisor account if they wanted users to add both purchasable and to keep track of that activity. P5 and P7 ran non­purchasable activities in one place. into this problem and decided to not create an account and pick something else instead.

TripAdvisor 15

Conclusions

In order to help increase TripAdvisor’s amount of conversions, our team recommends that TripAdvisor’s design and marketing team implement the design changes that we recommended for each finding that was covered in this document. One of our final recommendations that we recommend for TripAdvisor’s marketing team, is to restructure their marketing strategies around pop­ups, specifically to create another marketing strategy that is not as dependent on using pop­ups to gain users’ attention. By creating another marketing strategy, TripAdvisor could potentially increase their conversion rates and meet higher sales goals. This recommendation is based off of our finding that pop­ups elicited negative reactions from the participants. Another final recommendation that our team recommends for TripAdvisor’s design team is to work on updating the interface of the website’s screens and to do more research on what users prioritize the different types, so that the navigation is clearer and that only the information that pertains to the page that the user is on. These redesigns can lead TripAdvisor’s conversions potentially increasing, because of users being able to find attractions and tickets that they want to buy, leading to more purchases on the background. In conclusion, in order to help meet higher sales goals, TripAdvisor could implement the design recommendations listed, which could help lead to higher conversion rates.

TripAdvisor 16

Back Matter Appendix

Usability Test Kit Team 9 ­ Madeleine Le, Gabby Bilka, Aylee Neff, Tre Paolini ______

Description TripAdvisor is an online travel platform that allows users to book attraction tickets, restaurant reservations, hotel rooms, and flights to different countries and cities. TripAdvisor generates revenue by offering tickets for tour packages, attractions, hotel rooms and flights. Our team plans to evaluate how efficiently a user can find appealing experiences and purchase them. Our team also plans to evaluate if users are able to easily navigate the forums.

Content

Screening Questionnaire 1

Consent Form 4

Scripts used during facilitation 5

Task Scenarios 9

Post Task Questionnaire 10

Post Task Interview 10

Post Test Interview 11

Data logging forms 12 TripAdvisor 17

Screening Questionnaire Social Media Post for recruiting Title: Students needing help evaluating usability of a travel website. We are a team of students at the University of Washington seeking participants for a usability study. We will be evaluating a travel website. If you are interested in participating, please fill out the short survey below. Respondents will be contacted based on their compatibility with the intended user groups within 72 hours of submitting the survey.

Please note that this study is for educational purposes and that participation is voluntary. The study will be at the University of Washington ­ Seattle campus and will require a time commitment of one hour. Refreshments will be provided and participants will enter in a drawing to receive a $25 Amazon gift card. Responses will take no longer than five minutes to complete.

The test session will be recorded. All personal information is confidential.

Actual Questionnaire 1. What is your name? 2. What gender do you identify as? 3. What age range do you belong to? ○ <18 ○ 18­22 ○ 23­30 ○ 31­40 ○ 41­54 ○ 55+ 4. Do you work in the travel industry? ○ Yes ○ No 5. How often did you travel in the past month? ○ 0 times ○ 1­2 times ○ 3+ times 6. How often did you travel in the past year? ○ 0 times ○ 1­2 times TripAdvisor 18

○ 3+ times 7. What are some online resources you’ve used to plan your trips? Select all that apply. ○ TripAdvisor ○ Bookings.com ○ Expedia ○ Kayak ○ Yelp ○ Other: ______8. Have you ever purchased an experience (restaurant reservation, tickets to an attraction, etc.) online? ○ Yes ○ No 9. Please insert your contact information below. We will contact you by email in the next 72 hours if you are selected to participate in this study. ○ Email: ○ Phone (Optional): 10. Please indicate which days and time ranges you are available. All times are in Pacific Standard Time Zone. ○ Monday, November 12, 2018 between 12pm to 6pm ○ Tuesday, November 13, 2018 between 3pm and 6pm ○ Thursday, November 15, 2018 between 3pm and 6pm ○ Friday, November 16, 2018 between 1pm and 3pm

Termination script: Thank you for taking our survey. We appreciate your willingness to participate in our usability sessions. If you meet our recruitment criteria, we will contact you within the next 48 hours to confirm a time for your usability session.

Best, UW students

Qualifications for Study Entry Participants must be over 18 (Q3), do not work in the travel industry (Q4), and have not traveled in the past month (Q5). Question 2: Though this category is open­ended in order to account for any non­binary gender identities, we will ideally schedule participants in proportion to: ● 40% male ● 40% female ● 10% non­binary identities TripAdvisor 19

Question 3: There are six groups; ideally schedule participants in proportion to: ○ 20% 18­22 ○ 20% 23­30 ○ 20% 31­40 ○ 20% 41­54 ○ 20% 55+

Question 6: There are two groups, 50% for each group; ● Group 1: Has traveled less than twice in the past year ● Group 2: Has traveled three times or more in the past year

Question 7: There are two groups, 50% for each group: ● Group 1: Has used TripAdvisor in the past ● Group 2: Has not used TripAdvisor in the past

TripAdvisor 20

Consent Form I agree to participate in this study conducted by the TripAdvisor.com Usability Test Group at the University of Washington, Department of Human­Centered Design Engineering. During this study:

● I will be asked to perform certain tasks on a computer ● I will be interviewed regarding the tasks I’ve performed ● I will be recorded through audio and video during the session

I understand and consent to the use of the recording by the TripAdvisor.com Usability Test Group at the University of Washington, Department of Human­Centered Design Engineering. I understand that this information and recording was for research purposes only and that my name and image will not be used for other purposes. I relinquish any rights to the recording used by the TripAdvisor.com Usability Test Group at the University of Washington, Department of Human­Centered Design Engineering without further permission.

Participation in this study is voluntary. Any information that I provide will remain confidential and in no way linked to my identifying information, including the audio recordings. I may refuse to answer any questions or discontinue participation at any time.

Below is my signature indicating that I have read and understood the information on this form and that any questions that I might have about the session have been answered.

______Participants signature Today’s date

TripAdvisor 21

Scripts used during facilitation

Introductory Script ❏ Before the session starts, confirm that Userzoom is open and working on both your personal computer and the computer the participant will be using.

❏ Test that everything is working right with the participants computer and that the browser’s cache has been cleared.

❏ Make sure that Userzoom is not recording, and do not start recording until after the participant has made it clear that they are ok with being recorded and have signed the consent form.

When the participant arrives, take some time to let the participant settle down and relax before starting the session. Ask the participant about their day, offer refreshments and act like a human())(()

Session Start:

Start time: ______Participant number______

Thank you for coming out today and participating in our usability study. We really appreciate you taking time out of your day to help us. I’ll be using this script during our session to ensure that my instructions are the same for all the participants of this usability study. ____ will be taking notes during our session, Our objective today is to see how you use the TripAdvisor website. I want to make it clear that we are testing the usability of the site, not you. Anything you do during the session is okay, there are no wrong answers. As we start the testing, I’m going to ask you to think out loud, to say anything that you are thinking about while using the website. We ask that you please say what you are thinking and be honest about the good and bad aspects of the site. We had no part in the design of this website, so you will not be offending anyone if you have issues with any part of the site. Our objective is to improve the usability of the site, so your honesty and critiques are important to making the website better. If you have any questions during the session, please feel free to ask. I may not be able to answer every question during the study but I will answer all your questions once the session is complete. This is so we know you are getting a true unaltered experience using the website, and TripAdvisor 22 we would not want to guide you or prompt you use the site differently than you normally would. If you need a break at any point in the study, just let me know.

We have provided a computer for you to use during this session. This computer has a program called “Userzoom” on it that allows us to record the computer screen and you during the session. This recording will only be seen by the four members of our team (some of which may not be present) to help discover and design improvements to the website. This recording will be deleted once we have finished reporting the ways to best improve the website. Your information will not be attached to this recording, and any information that could link you to this study, such as your name or age, will be anonymized. If you have any question about the video or the recording process, I’d be happy to answer them now.

We now ask that you sign this consent form which goes into more detail about this session and confirms you understand everything I have told you so far. Please feel free to read over it again. ❏ Give the participant the consent form and a pen. ❏ Confirm that they signed the consent form and put the correct date. Now we will turn on the cameras and start recording. ❏ Open up the computer, turn on the recording, and confirm that it is working via your personal computer.

Ok, lets get started. This is how today’s session will work: ● We will start by doing a pre­test interview to learn more about you and your experiences traveling. ● I will then read out loud three scenarios that a traveler may find themselves in and ask you to complete tasks using TripAdvisor. ● After each task, I will give you a quick Post­Task Questionnaire and ask you a couple of questions about your experience during the task. ● After we go over every task, I will give you a longer questionnaire to complete and we will debrief about your overall experience using TripAdvisor.

Do you have any questions before we begin?

TripAdvisor 23

Study Script ❏ Read the Introductory Script. ❏ Do the Pre­Test Interview. As mentioned earlier, we want you to think aloud while you complete each task. “Thinking aloud” means what it sounds like, saying anything that comes into your head while going through the website. It can be tricky to remember to say everything you are thinking out loud, so we will do a quick “think aloud” exercise to give you some practice before we start doing tasks. While thinking aloud, could you please tell me how many windows are in your house/apartment? ❏ Confirm that they are “thinking aloud” properly and help them get it right if they are struggling. Okay, to start, please navigate to the TripAdvisor website from Google. Once you get to the homepage, take some time to inspect the page. You can scroll up and down, but please do not click anything yet. ❏ Allow participant to explore front page for at most two minutes, participant will likely finish before then. Ask them what they think they can do in order to encourage thinking aloud. Also ask them about the perceptions of the homepage. Okay, now that you have seen the front page, I am going to ask you to do a task based on a scenario. I’ll also have a paper copy with the task written down to help you just in case you get lost or forget.I’m going to read the first scenario out loud now: ❏ Read the first scenario, and remind them to think out loud once you finish reading the scenario. ❏ Allow the participant to proceed until they get stuck for a substantial amount of time or get frustrated. If 5 minutes have passed without completing a task or they stop doing the task, give them assistance through the form of a question or probing. If the participant is still frustrated, stop the task and move on to the next step. ❏ Pull up the participant a post­task questionnaire on the computer and interview after the task is complete. ❏ Remember to ask any questions that come up from observations during the post­task interview. ❏ Repeat the above steps for each task. We are all done with the tasks now, thank you so much for going through these with us. Now I will pull up a final questionnaire final questionnaire that asks about your overall experience using the TripAdvisor website. ❏ Give participant time to do final questionnaire. ❏ Do final interview with participant, with any extra questions that came up from the tasks. ❏ Ask the observer if they have any questions for the participant. Ok, we are all done. Thank you again for participating in our usability study, we really appreciate it. Do you have any final questions for us? TripAdvisor 24

❏ Explain any incentive logistics ❏ Stop the recording and save the file ❏ Say thanks again and escort the participant out of the premises.

Pre­test Interview ● Tell me about the last time you traveled. ○ When was the last time you traveled? ● How do you usually make your travel arrangements? ○ What resources do you use to plan your travels? ● Have you ever used TripAdvisor? ○ If yes, when was the last time you used it? ○ Can you tell me about your experience using it? ○ What did you use it for?

TripAdvisor 25

Task Scenarios Preliminary Briefing – You are are planning a vacation during winter break from Seattle to Prague with your significant other. As you are relatively thrifty, you are interested in accommodations with a budget of $100 a night with breakfast and wifi provided. You will only have t hree full days in Prague and would like to make the most of them by pre­planning your trip and making sure each activity is worth your time.

Goal Task Scenario Indication of when task is completed

Finding flight tickets Purchase round­trip Starting on the home Task completed when flight tickets page, start planning the user gets to the your dream vacation pay screen. to Prague by getting a roundtrip flight from Seattle for the week of December 17th to December 21st.

Finding a hotel room Find a hotel within After finding a flight, Task completed when the user’s budget that you also need to find the user gets to the has all of the a hotel room for two pay screen. amenities that they people, that includes want breakfast and wi­fi, under $100 per night during the week of December 17th to December 21st.

Finding things to do Create a list of fun After finding your Task completed when activities/events to do hotel, you’re the user has found interested in three activities/events exploring what there that they would book is to do in Prague. for that day. Try creating an itinerary of the day after you arrive with at least three activities or events. Some examples of TripAdvisor 26

these kinds of activities could be castle tours, concerts, or museums.

Exploring pre­made Find a structured You trust the Task completed when itinerary guide that has a community of user finds and recommendations TripAdvisor users’ travelers who have explores the 3­day travel visited Prague before itinerary in the travel recommendations for and want to see their forum section of the three days tips and travel site recommendations. Think about where you could find structured guides. In particular, you’re interested in a curated list of activities and recommendations for your t hree days in Prague.

Evaluating and Make a reservation to For your first night in Task completed when making reservations a highly rated Prague you are the user reacts the at a restaurant restaurant and to interested in eating “select time” screen check if that dinner at a fancy and has audibly said restaurant has good restaurant. Try whether the service making reservations restaurant has good for a highly rated service or not (drawn restaurant for dinner from user reviews) for your first night in Prague on December 18th and check to see if it has good service.

TripAdvisor 27

Post Task Interview ● On a 1 (strongly disagree) ­ 5 (strongly agree) scale, rate your opinion on the following statements: ○ I am satisfied with the ease of use to complete the task ○ I am satisfied with the amount of time it took to complete the task ○ I am satisfied with the amount of information provided to help me make decisions related to completing the task (i.e. whether individualized accommodations are available, what is included in a ticket, etc.) ● How would you go about finding this information for a trip usually? ○ How does it compare to this experience? ○ What was better? ○ What was worse?

TripAdvisor 28

Post Test Questionnaire

1 (strongly 2 3 4 5 (strongly disagree) agree)

I would use this site again

I would recommend this site to someone else

I found this site intuitive

I found this site consistent

I found the information on the site helpful

Post Test Interview ● Can you tell me about your overall experience using TripAdvisor? ● What did you like most about using the website as you completed tasks? What did you dislike most about using the website as you completed tasks? ● What is your prefered TripAdvisor feature? ○ Which task was easiest to complete? ○ Which task was hardest to complete? ● What would you change about your overall experience using TripAdvisor?

TripAdvisor 29

Data logging forms Note: Errors are described as anything that a participant does that is outside of the sequence of events to complete the task or anything that requires the participant to retrace their steps to move forward.

Task Description Task Start End Number of Types of Errors Success Time Time Errors

Finding flight tickets

Finding a hotel room

Finding things to do

Exploring pre­made itinerary recommendations

Evaluating and making reservations at a restaurant

TripAdvisor 30

Task Description Positive Participant Negative Participant Observer Comments Quotes Quotes

Finding flight tickets

Finding a hotel room

Finding things to do

Exploring pre­made itinerary recommendation s

Evaluating and making reservations at a restaurant

Recruitment Survey Results:

Have you ever purchased an What are experience some (restaurant online reservation, How often How many resources tickets to What What age Do you did you times did you’ve an Used What is gender do range do work in the travel in the you travel used to attraction, Accepted Tripadvisor your What is you identify you belong travel past this past plan your etc.) or not? ? Contacter Timestamp name? your email? as? to? industry? month? year? trips? online?

Expedia, 11/6/2018 XXXXXX X xxxxxx@xx cleartrip.co 17:18:21 XXXXX .xxx Male 18­22 No 1­2 times 3+ times m Yes

TripAdvisor, Booking.co xxxxxxxxxx m, 11/6/2018 Xxx ###@xxxxx Expedia, Yes Gabby 19:23:07 Xxxxxxx .xxx Female 18­22 No 0 times 3+ times Yelp Yes

Expedia, 11/6/2018 Xxxxx#@xx Google No Madeleine 20:21:02 Xxxx Xxxx .xxx Male 18­22 No 0 times 3+ times flights Yes TripAdvisor 31

11/7/2018 Xxxxx xxxxxxx@x Yes Yes Gabby 10:52:44 Xxxxxx x.xxx 18­22 No 0 times 1­2 times TripAdvisor Yes

TripAdvisor, Kayak, Google flights, Lonely Planet, 11/7/2018 xxxx@xxxx Secret Yes Yes Madeleine 13:49:52 Xxxx Xxxx xxxx.xxx Male 23­30 No 1­2 times 3+ times Flying Yes

xxxxxxxxx. TripAdvisor, 11/7/2018 Xxxxxxxxx xxxx@xxxx Expedia, 18:29:25 Xxxx x.xxx Female 18­22 No 1­2 times 3+ times Yelp Yes

TripAdvisor, Booking.co m, Expedia, Xxxxxx Xxxxxx.xxx Yelp, 11/7/2018 Xxxx­Xx­Xx xxxxxxx@x Google 21:31:04 xx xxxx.xxx Female 18­22 No 3+ times 3+ times flights Yes

xxxxxxxxxx 11/7/2018 Xxxx @xxxxx.xx Yes Yes Tre 21:36:52 Xxxxxx x Female 18­22 No 1­2 times 1­2 times TripAdvisor Yes

11/7/2018 Xxxxxx xxxxxx##@ Yes No Tre 21:37:48 Xxxxxxxxx xx.xxx Female 18­22 No 1­2 times 3+ times Kayak, Yelp Yes

11/7/2018 Xxxxx xxxxxxxx@ TripAdvisor, 21:40:02 Xxxxxx xx.xxx Female 18­22 No 1­2 times 3+ times Yelp Yes

11/7/2018 xxxxxx@xx 21:46:25 Xxxxx Xxx .xxx Female 18­22 No 1­2 times 1­2 times Expedia Yes

Xxxx#### TripAdvisor, 11/7/2018 @xxxxx.xx Expedia, Yes Yes Madeleine 21:47:18 Xxxxx x Female 23­30 No 0 times 1­2 times Yelp Yes

Expedia, Yelp, 11/7/2018 Xx­Xxx xxxxx##@x google, 22:32:09 Xxxx x.xxx <18 No 0 times 3+ times blogs Yes

11/7/2018 [email protected] Yes No Madeleine 22:59:22 Xxxxx Xx xx Male 18­22 No 0 times 3+ times Yelp Yes

Booking.co m, 11/8/2018 xxxxxxx@x Expedia, Yes No Madeleine 9:44:53 Xxxx Xxx x.xxx 18­22 No 1­2 times 3+ times Yelp Yes

##xxxxxxxx 11/8/2018 Xxxxxx xxxx@xxxx 11:38:33 Xxxxxx x.xxx Female 18­22 No 1­2 times 3+ times None Yes

TripAdvisor, 11/8/2018 xxxxxxx@x Expedia, 17:16:31 Xxx x.xxx female 18­22 No 1­2 times 3+ times Yelp Yes

11/8/2018 Xxxxxxxxxx xxxxxxx@x 17:43:58 x Xxxxxxx x.xxx Female <18 No 1­2 times 3+ times Expedia Yes

Expedia, 11/8/2018 Xxxxxx xxxxxxxx@ Cheapoair, Yes No Madeleine 18:05:10 Xxxxxxxx xx.xxx Female 23­30 No 0 times 3+ times Priceline Yes

TripAdvisor, 11/8/2018 xx#####@ Booking.co 19:35:58 Xxxxxxx Xx xx.xxx Female 18­22 No 1­2 times 3+ times m, Yelp Yes

Affinity Diagram: TripAdvisor 32

Quantitative Data Table with Averages:

Post Task q1 t1 t2 t3 t4 t5 Particpant # 1 4 4 4 4 5 2 3 4 4 1 5 3 3 4 5 2 3 4 2 3 3 2 2.5 5 4 5 4 3 4 6 5 4.5 3 3 5 7 5 2 5 5 5 8 3 4 2.5 1 4 9 3 4 3 4 4 10 3 4 3 5 4 avg 3.5 3.85 3.65 3 4.15 q2 t1 t2 t3 t4 t5 1 4 3 5 5 5 2 2 3 4 1 5 3 3 4 4 3 3 TripAdvisor 33

4 2 3 3 5 3 5 3 5 5 4 4 6 5 4 4 5 5 7 5 3 5 5 5 8 4.5 5 3.5 1 4 9 4 5 3 5 4 10 4 4 3 5 4 avg 3.65 3.9 3.95 3.9 4.2 q3 t1 t2 t3 t4 t5 1 5 5 5 5 5 2 2 4 4 1 5 3 4 5 5 5 4 4 5 5 2 5 3.5 5 3 5 4 3.5 4 6 4.8 4 5 5 5 7 4 3 5 5 5 8 3 5 2 1 4 9 4 4 4 4 5 10 2 4 4 4 5 avg 3.68 4.4 4 3.85 4.55 Post Test q1 q2 q3 q4 q5 1 5 5 4 5 5 2 3 3 2 3 3 3 3 3 4 4 5 4 4 4 3 4 5 5 3 3 4 4 4 6 5 3 4 4 5 7 4 4 5 4 4 8 4 4 4 4 4 9 4 3 2 3 4 10 4 4 4 4 5 avg 3.9 3.6 3.6 3.9 4.4

TripAdvisor 34

# of errors per task t1 t2 t3 t4 t5 1 1 1 2 2 4 3 1 1 4 1 1 5 6 2 2 1 7 1 2 1 8 1 3 3 9 1 1 2 10 2 2 avg 1.285714286 1.4 2 2 0

Cleaned Up and Compiled Notes

Participant 1 ­ Madeleine (Hugo)

Pre­test Interview

1. This June, used a little bit of TripAdvisor ­ Google flights ­ Kayak ­ Use google, Kayak, and Priceline

2. Used TripAdvisor, not for booking stuff but to find an 8 day tour ­ Thought it was pretty good ­ Does not remember having any issues

Hotel Room ● Used filters to find a room that includes wi­fi and breakfast

Task 1 ● Post Task Questionnaire: 4,4,5 ● Usually would use TripAdvisor ● Basically the same: liked how it offered them to check through expedia, kayak, etc. ● Filters might not be placed in the best spot

Task 2 ● Post Task Questionnaire: 4,3,5 ● Usually would use Priceline or Google ● Default sort was confusing, sort was confusing with tabs ● Showed rooms outside of budget

Task 3 ● Post Task Questionnaire: 4,5,5 ● Usually would use TripAdvisor ● Liked categories ● Unclear whether or not experiences would conflict with each other

Task 4 ● Post Task Questionnaire: 4,5,5 ● Probably google “3 days in Prague” and end up at a Buzzfeed post ● It’s pretty direct, seems like it’s a good primary source of information ● Not super obvious, required a little hunting

TripAdvisor 35

Task 5 ● Post Task Questionnaire: 5,5,5 ● Google search “best restaurants in Prague” ● Google reviews ● Yelp reviews ● Search best places to eat within a price range ● Made all information that came to mind obvious ● 5,000 restaurants: perhaps add some different filters

Post Test Interview ● Pretty good, never felt completely lost ● Knew where to get started, then it was pretty straight forward ● So much stuff ● Liked user itinerary and how people could rate it ­ Options for different itineraries ● Task 1 was the easiest ● Task 2 was the hardest, took longer than others

Participant 2 ­ Madeleine (Hannah)

Pre­test Interview 1. If I plan a trip for myself, I use expedia, priceline, looking for cheapest option with reasonable flights. 2. Has not used TripAdvior before

Task 1 ● Starts by clicking on flights ● Fills information into bar via calendar, chooses “prefer non­stop” ● Chooses to find expedia because she has experience with them ● Waits for new tab to load with information ● Decides against cheapest flight because long layover ● Is confused by flight page that connects to partner pages ● Choses to reserve a seat because she is flying with another person ● Takes a lot longer to book through airline website compared to other participants ● Post Task Questionnaire: 3,2,2 ● Post Task Interview ○ 1. I look at expedia and at sites where I have flexibility to choose where I sit ○ 2. Quotes ○ 3. Quotes

*** “ I expected to do it all through one site.”

Task 2 ● Clicks on “hotels” ● Searches “Prague” ● Inserts dates on calendar ● Moves filters for under $100 and free wifi and breakfast ● Scrolls through options and compares prices / amenities ● Uses photos to decide on hotel ● Gets redirected to booking.com ● Avoids popups and chooses hotel ● Chooses bed and non­refundable option ● Post Task Questionnaire: 4,3,4 ● Post Task Interview ○ 1. Use expedia, had similar experience with filters etc. ○ 2. I liked that it gave different prices through different websites ○ 3. Quotes

Task 3 ● Goes to homepage ● Chooses “things to do” and types in Prague ● “Usually I travel with friends, so I never have to find things on my own” ● Inserts correct date ● Clicks “see all” for top things to do ● Chooses activity without purchase ● Chooses 3 activities without purchase, but goes back to look at canal tour info ● After looking at description decides to purchase canal tour ● Post Task Questionnaire: 4,4,4 ● Post Task Interview ○ 1. Google, type in “things to do” and compare, ask friends who have been there what they would do ○ 2. “Has more information in one place than having to compare a bunch of websites.” ○ 3. “It’s nice to hear from people that I know personally and what they enjoyed.”

Task 4 ● tarts on home page and focuses on left tab with pictures of people ● Gets redirected to “prague hotels” and navigates back ● Goes thru “feed” and finds top contributors in prague, chose a profile ● See’s his reviews and photos and map, finds no valuable information TripAdvisor 36

● Goes thru forums to “prague” ● Misses small tab with “three days in prague” **** ● Chooses travel destination expert ● Decides to reach out to expert and finished

OVERALL: FAILS TASK

Likely reason for failure: “travel guides” option is NOT in “more” tab unless on a specific city page, therefore people who failed assumed the information is in the forums and get lost

● Post task: 1,1,1 ○ Why confusion? Could not click on thing on front page, I thought that would take me to the write section 1. Would reach out to family for recommendation 2. Nothing was better 3. I had no idea what I was doing. I could not figure out why I couldn’t click on the front page section

Task 5

● First click on “restaurants” ● Searches prague ● Looks at filters, clicks “online reservation” ● Scrolls around, then chooses “mid­priced” option ● Chooses restaurant and checks reviews, likes reviews ● Selects date and time for reservation ● Post Task: 5, 5, 5 1. Yelp, google for other sites 2. I liked all the information on different restaurants it gave me 3. Nothing ­ pretty good

Post Test Interview 1. Had the options for everything to go somewhere, flights were confusing and I would rather do it all through tripadvsior instead of doing everything twice, hotels was more straightforward, better than finding flights. Things to do and restaurants are easy. Overall: balanced between positives and negatives 2. Restaurants are easy 3. Flights and guide were hard 4. I want the flights and figuring out how to connect to people to be easier

Participant 3 ­ Gabby (Adelia) Pre­test Interview ● Need video :| Task 1 ● Quickly x’d out of front page popup ● Doesn’t use too many travel websites ­­ just Yelp of Indonesia ● Missed 2 people, only had herself ● Jst picked top one flight ● Wants just 1 deal ● Post task Q: 3, 3, 4 ● Post Task Interview ○ Use multiple flight websites and compare tickets ■ Idea of price range ○ Others easier, no pay now, need a partner ­> didn’t see it ○ More flight details than other websites before checkout ○ Finding pay screen is worse ­­ “pretty ok” Task 2 ● Not sure how to get back home after flights, just chose top 1 in price range and had all amenities needed ● Saw deal, cheaper than other times, realized its for two people when price was so cheap ● Post Task Q: 4 5 5 ● Post Task Interview ○ Why so quick? Very detailed, everything included in the card ○ Go directly to airBNB for booking, likes neighborhood map but likes the listing first then location ○ Likes comparing between partners Task 3 ● Scrolling to see what info is there for tour, if interested ● Not really checking availability, just seeing if it looks like something fun to do ● Skimming a couple of reviews ● Looking through diff categories ● Focused on what to do, logistics don’t matter ● First one that popped out ­­ trusting top things to do ● 5 4 5 ● Just googles everything, instagram → pics inspo, wants to take them ● Logistics are there like ticket prices, not usually so concerned, ● Too many recommendations

Task 4 ● Looks through things to do for a while ● Tries … clicks travel guides ● Hard to complete, like the guide (?) ● 2 3 5 ● Ask friends for recommendation, travel blogs TripAdvisor 37

● Saves time if I’m not familiar with city ● Likes tips ● Finding page is difficult — should be beside hotels, flights and restaurants

Task 5 ● Restaurants, chooses first for fine dining ● Checks out first couple of reviews ● Skipped reserve online filter ● Looking for reservation button online ● Goes to restaurants website, goes into too see if available on dates with number of people ● Likes service based on review ● 3 3 4 ● Yelp usually, or call restaurant ● Not as much info in comparison ● Easier in yelp as she wanted a reserve button, why go to another website? ● Likes photo slideshows, affects choice

Post interview ● Some easy to navigates, some not ● Lots of information, laid out nicely (except flight) hotel easy, flight hard, travel guide hard, restaurant easy/hard, liked hotel ● Website navigation, buttons we need are not there, want more obvious places to go to and direct to travel guide as main button ● Liked detailed info but didn’t look at it, more trustworthy knowing its there

Participant 4 ­ Gabby (Supo) Pre­test Interview ● Usually google flights, cheapest option ● Secret flying —> flights are focus, go to TA as yelp when abroad ● 2 years ago in Europe, language and cooking classes in Mexico ● Best alternative with yelp info wise, usually not totally satisfactory, more clutter, “not super sleek”, yelp not in Europe ● Homepage ○ No call to action (single) ○ No pop ups ○ Search bar ○ New feed, not really what he wants off TA

Task 1 ● Already from Seattle ● Selected all websites ● Alert might have you experience buyers remorse, pass ● Another pop up for alerts ● “Information overload” ● Too many options for filters ● 1st is always cheapest ● Then there’s more options ● Once redirected, so its searching again? ● Would go directly to Condor not through TA ● Skips thru extras —> “are we done yet?” ● 2 2 5 ● For aggregate sites would normally look for cheapest ● And also not red eye, punch in dates manually on other site — don’t trust ag sites to actually book the flight ● So many options, overwhelming information ● Everything on one page is both + and ­

Task 2 ● Hotel button in nav ● Likes they remembered dates and info already entered ● Goes up to $100 filter ● Not much different from top result, new tab still on TA ● Trip A cheapest but not on top ● Waste of space to have multiple options that are same ● 3 3 5 ● Scrolls thru website ● AirBNB —> doesn’t stay in hotels ● Would pick external sites to book thru for hotels ● Likes TA ratings of previous guests ● Builds trust w/ TA, extra info ● Lots of clicking, info not ordered (no hierarchy)

Task 3 ● Not sure why dates, doesn’t go for tourist attractions typically ● Would add to google maps — locations to go to ● Doesn’t want pics as he wants the surprise, wants own experience ● Clicks on map —> wants things close to each other other ● Wants to know once in an attraction what else is nearby ● 3 3 2 ● Asks friends for recommendations, TA to see what is out there ● Kinda complicated, would google TA Prague, too many clicks ● Likes update results, like a refresh —> festivals or free museum dates TripAdvisor 38

● Could find list anywhere, no extra value

Task 4 ● Travel forum —> travel guide … are hidden ● Which list to trust? 97% rating, how many voted? ● It’s ok, meh ● 2 5 5 ● NY Times list, lonelyplanet —> simple, more trustworthy, just one guide so its going to be good as the only one out there ● Hidden, lots of info in guide, map is cool but wants more interactivity ● Likes rating, of overall and each place, lotsa text

Task 5 ● Restaurants top bar ● Local cuisine = fine dining ● #1 has 218 reviews #2 has 5000 reviews —> trust more reviews ● Likes ratings in categories ● Looks for reservation button ● Found reservation filters, found top restaurant online ● Nice if it auto updated to dates in Prague ● 1 (reserve) / 4 (ratings, 3, 3.5 ● TA for restaurants, no other global place, list of potential places, 1 fancy rest cheap ● Didn’t know which places you could or could not make reservations for ● Split ratings liked ● No hierarchy of info, forced to scan whole page

Post Interview ● “They tried”expected more from website —> overwhelmed ● Surprised reservations was hidden, as that’s super needed ● Liked the dates popping up again ● Liked ratings and reviews ● Lots of info, lots to click, wishes for more guidance ● Loop back in(but no pop ups!) ● Conversion of users to customers ● Liked reviews for restaurants most, other websites do other things better ● Things to do easiest — already there sorted auto by most reviewed (?) ● Finding flights hardest ● Want to know exactly what I need to do on site, single text box, omnibox ● Too much information

Participant 5 ­ Madeleine (Elizabeth) Pre­test Interview ● Last traveled to Whistler in early september ● Uses Airbnb, Yelp, Tripadvisor ○ Previously used TripAdvisor three weeks ago to find something to do ● Local websites and Expedia

Task 1 ● Price and alert windows are pretty annoying ● Didn’t like that the expensive price was shown the first ● Did not like how there were so many options for prices from different partners ● Is redirected to another website ○ Does not like the price hikes on this website ● Post Task Q: 4, 3, 3 ● Post Task Interview ○ Normally would use Expedia or something similar ○ The experience is similar to what they’ve used in the past ○ Harder to see exact details for the different deals ○ Condor website was bad, due to having extra charges Task 2 ● Goes to hotel page ● Finds one ○ Moves the slider on the side to filter results >$100 ○ Clicked on filter boxes that allow her to see hotels with breakfast and wi­fi ● When she clicks on it, it redirects her to Expedia ○ Clicks on another hotel different than the one that she picked on TripAdvisor ● Post Task Q: 5, 5, 5 ● Post Task Interview ○ Would search for hotels again with this method, only doesn’t really pay attention to the wi­fi ■ Would do it on Airbnb though ○ Super easy to adjust the price and find filters (wi­fi and breakfast) ○ Saw that the room with no cancellation was cheaper Task 3 ● Finds list of top things do in Prague, by going to things to do page ● Finds something, decides it’s good based on the number of stars it has ● Tries to save each of the things/landmarks that she wants to do, but when she tries to save it, it asks her to create an account ● Decides that she’s going to save everything by opening up each thing/landmark into different tabs, as a way of getting it on the list ● Post Task Q: 4,5,4 TripAdvisor 39

● Post Task Interview ○ Likes how opening up multiple tabs was helpful to creating a list ○ Normally uses TripAdvisor ○ Would typically use city centric websites, gives less tourism oriented stuff if she goes to that website ○ Too touristy in terms of options given Task 4 ● Uses the general search to find guides at first, typing in “3 days in Prague” ○ Goes to search results, it at first only shows hotels and attractions ● Is able to find guides by clicking on the tab in the search results “Travel Guides”, which allows her to see only travel guides ● UserZoom prevents her from clicking the link to the guides ● Opens up two guides ○ Scrolls through only one of the guides, decides on that one ● Post Task Q: 3, 4, 3.5 ● Post­Task Interview ○ Normally just googles travel guides ○ Likes reading blog posts ○ TripAdvisor guides only provide locations, not much on the details ○ Liked having a map to see where all the locations were ○ Finding the guides itself was hard, even when she typed in the search bar ■ Initially had only hotels and landmarks seen, first hard to see ■ Was able to see the button that could see the option for her to filter eventually ■ Thinks the filters need to be more visible Task 5 ● Clicks on a restaurant tab, goes to the fine dining section ● Sorts all results, by clicking on ranking tab ● Looks for reviews with service based on typing in “service” in the search bar to find reviews that indicate good service ● Goes to website to make a reservation, can’t seem to find a way to make a reservation ● Post Task Q: 4, 4, 4 ● Post Task Interview: ○ Normally uses Yelp, works similarly ○ Yelp uses OpenTable to make reservations, which she did not see on TripAdvisor ○ Good option of restaurants, easy to navigate Post Test Interview ● Overall experience was pretty good, liked everything except flights ● Did not like pop­up tabs, very distracting ● Just want to use only TripAdvisor ○ Can’t book directly ○ Airline website was annoying ● Website was pretty, overall put together ● Likes the restaurant feature and things to do ● Flights was hard ● The things to do website was the easiest because of the list of things to do was high ● Overall experience was good ● Would prefer to do everything in TripAdvisor

Participant 6 ­ Gabby (Evan, no vid) Pre test interview ● Trip to Canada, airBNB for places ● Googled things to do in Vancouver ● Looks at search bar first ● Notices you might need an account, now just like Instagram for TA

Task 1 ● Flights —> find flights, ignores extra checkboxes ● Avoids entering email ● “Did I do this right?” ● “Trip price increased by 99 cents” ● Went to Expedia next ● 5 5 4.8 ● I think it has all the info, but the layout — don’t know the airlines ● Timing is confusing, doesn’t list airlines on the splash page ● Usually google flights ● Different in deals available, looks pretty similar ● Looks for price difference

Task 2 ● Where’s the home button — “small little title thing” ● “Nice recently viewed button that was easy” ● “And my dates, nice” ● Uses slider for < 100 and breakfast/WiFi included ● Now at Expedia, more deals ● Didn’t realize at first it wouldn’t be thru TA ● Let travelocity be the one to book through ● “Then a new tab…” ● Looks at description, then room type ● What is a double vs a twin? ● 4.5 4 4 ● Didn’t look at where it is in Prague TripAdvisor 40

● “Oh map seems useful” better than distance from ● Location almost as important as price ● Map button should be more obvious ● Lots of info but could be more clear ● AirBNB default might be a map, guarantees you know where it is ● Location > deals, map ● Specific amenities are better “more options to limit places”

Task 3 ● Straight to things to do ● Still in Prague which is convenient ● Concerned about price, looks at reviews ● More info on tix ● Trying to figure out if tix for specific day or just whenever, date select happens after add to cart — opens at 9am ok ● Back to things to do — diff one than last time, have to enter in Prague again ● “Now back to the original page” ● Again map isn’t obvious — what else if near the zoo? ● Options for deal to select ● Tour option not available — that’s unfortunate ● Back and forth from map to single page ● Just walk through ● Back button doesn’t take me to map ● 3 4 5 ● Really wants a map, especially for multiple activities in the day, wants to enter in availability ● “Kind of a hassle” map on availability ● Doesn’t save search and have to ● “Longer than necessary but not terrible” ● Show which activities are available ● Usually just google and might stumble upon TA, google maps is kinda easier ● Easier to find but not as easy to book

Task 4 ● Starts in things to do ● Useless thing that has every other thing ● Tried clicking buttons … ● “Maybe this is exactly what I needed” … ● 3 days in Prague for budgeted minded ● Tips and a list of restaurants —> “don’t think this guy is great at it” ● Chooses the top one ● “Oh ok so this one has a plan for each day” ● “Day laid out which is p nice —> this seems useful” ● 3 5 5 ● Once I found the section it is easy ● I wouldn’t ever press this button normally ● Once I found it I think its useful ● Parents would find a pro to do it, but it costs money, TA is more cost effective, would choose

Task 5 ● Online reservation as filter ● Only like 5 ● Looks to see if we can filter by good service ● Has date saved ● Seems fancy, good reviews, looks at pics ● Rating for service ● Aren’t available for 18th, filter for date for reservation? ● 5 5 5 ● Add a date reservation filter ● Google places to eat, don’t generally book ahead ● TA is more useful, faster online booking ● Hopefully they have all the restaurants in the area

Post interview ● How they do the map is terrible, location is biggest thing in diff country ● Liked all the combo of it all in one ● Tasks were all similar, community travel thing was kinda confusing as its in bubbles ● Wouldn’t do it normally but only if needed ● Why cruises over travel guides?

Participant 7 (Noah, no vid) Pre­test Interview ● I use mostly searching on google to look up flights, hotels, hostels

Task 1 ● Ignores compare prices, doesn’t want to put in an email for alerts, ignores popups ● Sees cheapest option and chooses it, includes ● Decides hostel + flight deal is the best deal ● Post Task Questionnaire: 5,5,4 TripAdvisor 41

● Post Task Interview ○ 1. Google flights I wanted and looking at some airlines, would weigh price of flights vs reasonability ○ 2. The deal with the fight and hostel was pretty good and I don’t think I would have found it on my own. ○ 3. Not a lot of different flight information

Task 2 ● Goes to hotel tab, likes that it knows he is looking in Prague ○ Opens a tab by accident ● Best value vs best price ● Chooses hotel based on hitting requirements and based on photos ● Looks at reviews, notices amenities like terrace and public tram access ● Confused about price difference ● Starts again on booking.com ● Goes back to TripAdvisor and chooses another option ● Post Task Questionnaire: 2,3,3 ● Post Task Interview ○ 1. Use “hotel” search function on google ○ 2. TripAdvisor gives more details ○ 3. Price listed was not same as listed on TripAdvisor

Task 3 ● Chooses “things to do” ● Amused by interesting choices, scrolls around to see various options ● Chooses food tour but it is too expensive, goes back ● Would have saved it but did not want to sign up ● Looks for more details on Prague zoo’s ● Checks review to confirm decision ● Decides on zoo and Prague Castle based on being top 2 tourist destinations in Prague ● Checks reviews to confirm ghost tour decision ● Post Task Questionnaire: 5,5,5 ● Post Task Interview ○ 1. Wing it, spend 5 min on google ○ 2. Well organized, a lot of user ratings and feedback, liked filtered categories ○ 3. N/A

Task 4 ● Goes back to homepage ● Chooses “more tabs,” finds travel guide ● Looking at 3 day guide, reads description ● Looks at each days activities, excited about jazz ● Wonders why she made it, thinks about her life ● Looks specifically at jazz boat, looks at reviews ● Decides to do everything in guide ● Post Task Questionnaire: 5,5,5 ● Post Task Interview ○ I don’t think I would look this up ○ Looks nice, detailed and user organized, very informative ○ N/A

Task 5 ● Goes to restaurants tab ● Chooses food type based on categories ● Decides on expensive restaurant after scrolling ● Doesn’t see reservation option on TripAdvisor ● Goes to website to make reservation ● Post Task Questionnaire: 5,5,5 ● Post Task Interview ○ Yelp or just google, google maps ○ About the same as using other services ○ Nothing really

Post Test Interview

1. Pretty good, flight and hotel options were not as strong as things to do and travel guides would need to compare TripAdvisor with other flights and hotels 2. Travel guide was cool and well designed, never thought about this option. Price difference between TripAdvisor and partner site also need to double check flights with other sources. 3. Liked travel guides, had a lot of information 4. Reservation was easy. Would take more time to do flight 5. Would have spent more time looking at flights and hotels. Would change flights page to fit more flight information.

Participant 8 (David)

Pre­test Interview ● No experience in making travel bookings online. But travels somewhat frequently.

*** BIG NOTE: Participants who use sites that ask a lot of questions get annoyed***

TripAdvisor 42

Task 1 ● Takes as second to look over front page, closes popup ● Puts information into the dates ● Chooses each of the options to compare flights ● Thinks there is a lot of information on the page but can be pieced through somewhat well. ● Sees a flight with one stop and chooses it ● Chooses a reasonable flight and clicks view deal ● Cannot discern between different partners, chooses cheapest option ● Doesn’t like big change in UI, extra options causes confusion ● Wonders about several options that are not laid out to him ● Post Task Questionnaire: 3, 4­4.5, 3 (felt like too much information) ● Post Task Interview ○ 3. A lot of options and dealing with that option

Task 2 ● Notices some information is added and some isn’t ● Likes that accommodations are included in listings ● Notices left side and adds filters ● Goes to booking site for hotel ● Looks at photos to determine if he wants hotels ● Post Task Questionnaire: 4,5,5 ● Post Task Interview ○ 2. Liked having control ○ 3. N/A

*** Thinks flights should be first

Task 3 ● Likes that it kept his date information from previous tasks ● Realizes the correct day to book activities and sets day to that day on things to do ● Scrolls through categories and inspects them ● Chooses first tour option ● Look at errors for more information ● Goes back to homepage after adding item to cart ● Concerned with time overlaps during the day, continues to choose item anyway ● Looks back at other activity to confirm timing works ok ● Finds a ghost tour based on timing (night) ● Sees that event is not available on everyday of the week ● Chooses final tour by adding it to cart ● Post Task Questionnaire: 2.5,3.5,2 ● Post Task Interview ○ Asking people at hotel/ friends ○ It gives a lot of options compared to talking to people ○ The burden of having more options makes it hard to decide. Pros and cons to both.

Task 4 ● Goes back to homepage ● Looks at extra options and looks at travel forums ● Searches “Pragu”on forums, so search results come up (no autocorrect suggestions) ● Realizes how to spell and redoes search ● Does not think forum topics apply to his situation ● Looks at off topic forum post, goes back to forums ● Chooses to make forum post, realizes its a bad way to get recommendations ● Post Task Questionnaire: 1,1,1 ● Post Task Interview ○ Looking up stuff online and talking to people and based on what I’ve done before ○ Nothing ○ Maybe have too much, lack of experience, thinks making a post would take too much time and does not trust random people

Task 5 ● Assume his first day is at a reasonable time ● Goes to “restaurants” does not like to trust “sponsored” content, still clicks it ● Looks at reviews, thinks looks good ● Cannot find any online reservation option ● Looks at “Q&A” and sees people make reservations through store website ● Finds website on page and goes to make reservation from there ● Post Task Questionnaire: 4,4,4 ● Post Task Interview ○ Google search or some other restaurant review site, maybe call ○ Good because the information is in english and avoids language differences ○ Could not get all information immediately, no reserve button

Post Test Interview 1. Fell short in planning out entire day, but good for other things 2. Top bar with big options like hotels makes it easy, only seems like a middle man and keeps everything standard, they have different UIs etc. 3. Clear about pricing, details were easy to see 4. Finding hotel room ­ easy; planning things to do hard to completely plan out entire day TripAdvisor 43

5. Do not have expectations to rely on other people for trip information

Participant 9 (Olivia, no video)

Pre Test 1. I went to Maine and Ohio and Boston 2. Google flights 3. No experience with TA

Task 1

● Starts by searching prague ● Clicks on “flights” ● Chooses extra days to accomodate for flight time ● Chooses 17­21 after realizing error ● Clicks out of pop up ● Decides to choose cheapest flight available ● Scrolls through “view deal” page, thinks its a lot ● Clicks on airline link at bottom to find more information ● Chooses to book with airliner over travel sites ● Sees more flight options on website, chooses flights ● Ignores extras when making reservations ● Post Task: 3, 4, 4 1. Google flights, it’s very uncluttered with its info 2. I liked the list of potential airlines vs. google flights only giving top 3 3. Did not like layout, thought it was weird that you had to pick your flights twice and that price was different

Task 2

● Goes back to homepage ● Clicks on hotel, searches prague ● Notices that the site remembers her date ● Changes price range filter ● Also filter for breakfast and wifi ● Likes choices and chooses first option, which is the cheapest and has solid reviews ● Notices price difference between TA and partner site, decides to book anyways because it's still in budget ● Changes dates on partner site because she would not need a hotel that night ● Post Task: 4, 5, 4

1. Just google different hotels I’m familiar with. Not sure if TA has that option, location is a big factor and I would choose something centrally located 2. It was easy to add filters and price. Price comparison and reviews are key info 3. Maybe have more additions to put location and distance from another location

Task 3

● Chooses “things to do”, searches prague ● Likes “tops things to do”, notices that these options have a lot more reviews than others ● Scrolls through “top things” ● Only sees how to add day range, not specific day ● “When I click on ‘see all’, it shows the top 3 options before going back to showing categories in the same list ● Looks through reviews 2nd top rated item, notices it's time range ● Can’t save things, chooses to make itinerary in her head ● Thinks map feature is glitchy (may have been in part due to some operator error using computer) ● Looks through descriptions and reviews of three chosen itineraries ● Chooses to buy tickets in advance for last chosen activity, prague zoo ● Post Task: 3, 3, 4 1. Would likely google “things to do” in prague 2. Liked the reviews, like being able to sort by time of the year and keywords 3. The screens are cluttered, there's a lot of info and it was not smooth to use/go through, could not get all popular things on one list

Task 4

● Goes back to homepage ● Clicks “more”, decides to try “travel forum” ● Finds “three days in prague” not sure if its a coincidence or if the site remembered the amount of time she gave as a date range ● Notices guide give 3­day itenerary and tips, thinks that the guide is from a local ● Goes back to other forum posts, notices specific FAQs and moves on, ● Goes to all travel guides, likes the guides and informational titles ● Looks for sorting buttons, find none, notices only 19 guides ● Chooses “3 days in prague for the budget minded” ● Post: 4, 5, 4 1. Look up “first trip to prague” and come across random blog posts and take tips from multiple ones 2. 3. Did not like that you couldn't sort guides by things like food type, etc

Task 5

TripAdvisor 44

● Clicks “restaurant” ● Likes “local” option ● Scrolls through categories ● Tries to sort by date, does not find options ● Tries to sort by reviews, finds the option ● Adds filters: dinner, reviews ● Can’t sort by “service”, but likes that you can sort by dietary restrictions ● Chooses a restaurant and likes the reviews ● Cannot find a reserve button on TripAdvisor, goes to website ● Uses website to put in date, slight confused by military time ● Post Task: 4, 5, 5 1. Looking up “best restaurants in Prague” 2. Liked the different checkboxes/filters 3. The option to boo through TA would be nice to have, so everything is in one place, but still liked the website link

POST TEST TO NOTE: weird layout list in things to do* 1. Pretty useful, see why people like it, it would be good for older people. Does NOT like user interface, it was pretty cluttered and had ads, also repeated itself a lot throughout the site 2. I liked all the information it gathered, it was nice info to have at a glance while searching. Site was pretty clunky, wasn’t super streamlined, did not like separate pop up page during flights 3. Liked restaurants, would probly use that feature while traveling, liked that you could filter by dietary restrictions, google does not have that 4. Exploring itenerary was easy ­4, finding flights was hard ­1 5. Make the main tabs more clea, the content was good but interface could be improved. A lot going on in “things to do” that user may not be interested in.

Participant 10 (Nick)

Pre Test: 1. Went to vancouver 2. Expedia to see the best deals/times, driving using google maps, use google to plan events 3. No experience with tripadvisor

Task 1: ● Chooses flights, types in prague ● Clicks all tabs to compare prices, realizes there is a select all button ● INCREDIBLY WEIRD FAILURE, it appears selecting the tabs disabled the flights button, causes a lot of confusion and makes him think he is doing something really wrong** ● Eventually restarts and gets on to the correct page ● Scrolls around page looking at different deals ● Chooses top cheap flight, realizes there is a long layover and decides to look back at more flights ● Reads info on cheapair and scrolls around ● Post Task: 3, 4, 2 1. Normally use expedia because I trust it and it has a better interface 2. I don’t know 3. Slightly confusing, did not like the interface, not user friendly (maybe?) a. Felt under pressure to perform b. One option was cheaper with a hotel than those without hotels, which was weird

Task 2: ● “I don’t like that there’s so much stuff” ● Clicks “hotel” tab, from flights page ● Finds a hotel with everything he needs at the top, likes that it uses expedia ○ Realizes that it doesn’t have breakfast and moves on ● Moves on to different hotel, clicks it ● Looks at listing on booking.com ● Looks through photos, decides its good ● See’s a lot of “BS” deals (lack of trust?) ● Realizes he is on a different site that he does not like, continues ● Post Test: 4, 4, 4 1. Expedia or hotels.com 2. It gave me every option there is, even through multiple websites 3. “No Gripes”

Task 3

● Clicks on “things to do” ● “Userzoom sucks” (in response to moving the facecam around the screen) ● Chooses “nightlife” ● Chooses “couples” ● Scrolls and finds a tour, is ok with reviews ● Makes sure dates work and adds item to cart ● Goes to checkout but decides to book more TripAdvisor 45

● Chooses a river cruise and enjoys ratings, chooses it since it works on his date and is in the morning ● Chooses segway tour and looks at rating, photos, and reviews to decide on it ● Realizes he has to do everything on one day and changes the dates and times of activities ● Puts in name for reservations ● Post Task: 3, 3, 4 1. Google “things to do” in prague 2. It had filters I could put on, and sort by day and price 3. Felt like everything was expensive and forced me to pay and open new tabs: “I felt lost”

Task 4

● Chooses “travel guides” ● Finds “3 days in prague” quickly ● “I’m intrigued about what suzana has to say” ● Likes city tips, said he would read all these ● Reads tips and reviews, decides to use this as his Itinerary ● Post Task: 5, 5, 4 ● “I feel like I got lucky” 1. Use google 2. I found it quickly 3. Nothing

Task 5 ● Clicks restaurants ● Likes categories ● Clicks “see all” on fine dining ● Chooses “bellevue” ● Looks through photos to decide on it ● “I’ll go to the website so I don’t have to call in ● Puts down date and time ● Post Task : 4, 5, 5 1. “I’d use the google” 2. It gave me a lot of restaurants and a lot of filters 3. “Do they have ALL the restaurants­ I guess so”

Post Test 1. It had all the things I needed to plan my trip, at first it was unfamiliar but it got better 2. I liked the variety of things I could plan. (anything bad?) No, it had everything I needed 3. I liked hotel and things to do and forum, “I liked it all!” 4. Easiest was the forum/guide because it was laid out easily; hardest was booking the flight 5. “ I wouldn’t change a thing”

Final Reflection

We learned a lot through the process of planning, creating, and running this usability study. If we were to do it all over again knowing what we know now, we would change our strategy for participant recruitment, run more pilot studies, and use a different video recording program than UserZoom.

We wanted to do our usability test with 8 different participants, so we stopped recruiting participants once we had booked 8 appointments. Unfortunately, 3 of the participants never showed up to their appointments, so we had to scramble to post more recruitment ads and get more participants last minute. It would have been better to book 10 to 12 participants if possible, expecting that some participants would likely not show up. At worst, we would have to run extra TripAdvisor 46 tests or cancel on the extra participants, which are both easier than trying to recruit more participants last minute.

We ran one pilot study as a group to make sure we were all on the same page of how to set up and run our usability tests. However, looking back I think it would have been better to run 4 quick pilot studies, so each person could have a chance to set up as a moderator. This would have saved us time in the future, as we usually had to spend extra time setting up the right programs and computer. Also, not all of us understood how to properly set up UserZoom, which cause us to lose some of the videos from our usability tests. If we had run more pilot studies, this fate could have been avoided, which would have saved us a lot of time and stress.

We used UserZoom to record the screens, faces, and audio of our participants during each test session. UserZoom ended up being more trouble than it was worth, and we would likely use a simpler recording tool like google hangouts or quicktime if we were to do this again. Like stated previously, setting up a test session incorrectly cause UserZoom to not save your video recordings, even if it seemed very clear that it was saving your video, even giving feedback like “your video will be saved to the cloud library”. We also were unable to figure out how to use some of the more interesting features of UserZoom, such as the ability to mark when a user has started and finished a task to make looking back at recordings easier. Since we really did not utilize UserZoom well, we believe that it would be easier to use programs with more intuitive interfaces and more visible systems.