Can Trailers Help to Alleviate Popularity Bias in Choice-Based Preference Elicitation?
Total Page:16
File Type:pdf, Size:1020Kb
Can Trailers Help to Alleviate Popularity Bias in Choice-Based Preference Elicitation? Mark P. Graus Martijn C. Willemsen Eindhoven University of Technology Eindhoven University of Technology IPO 0.20 IPO 0.17 PB 513, 5600 MB, Eindhoven, Netherlands PB 513, 5600 MB, Eindhoven, Netherlands [email protected] [email protected] ABSTRACT quire them to rely on memory. This can lead to unreliable Previous research showed that choice-based preference elic- feedback [2]. itation can be successfully used to reduce effort during user cold start, resulting in an improved user satisfaction with 1.2 Choice-Based Preference Elicitation the recommender system. However, it has also been shown One way to reduce the effort can be found in choice-based to result in highly popular recommendations. In the present preference elicitation. Where most recommender systems study we investigate if trailers reduce this bias to popular ask users to provide a number of ratings on items (explicit recommendations by informing the user and enticing her feedback), recommender systems applying choice-based pref- to choose less popular movies. In a user study we show erence elicitation ask the user to make a number of choices that users that watched trailers chose relatively less popular (implicit feedback). Using implicit feedback to produce per- movies and how trailers affected the overall user experience sonalized ranking has been shown to provide better fitting with the recommender system. prediction models than using explicit feedback [8]. In recent user studies users of collaborative filtering systems were pro- CCS Concepts vided with choice-based preference elicitation[4, 6]. Where in the more standard rating-based preference elicitation peo- •Information systems ! Recommender systems; •Human-ple are asked to rate the items they know, in choice-based centered computing ! Human computer interaction preference elicitation they are asked to choose the item that (HCI); User studies; User models; best matches there preference from a list. In our own work, this alternative has been shown to require less effort than Keywords more standard rating-based preference elicitation, while al- lowing for more control, resulting in more novel and accurate Recommender Systems; Choice-based Preference Elicitation; recommendations [4]. User Experience; Trailers; Information Other work compared a recommender system using rat- ings against a recommender system using pair-wise compar- 1. INTRODUCTION isons (i.e. choices between two alternatives)[1]. The sys- tem using comparisons provided better recommendations in 1.1 Cold Start terms of objective performance metrics (nDCG and preci- sion). In addition, users preferred the system using pair-wise New user cold start is one of the central problems in rec- comparisons as it made them more aware of their choice op- ommender systems. It occurs when a user starts using a tions and provided a nicer user experience. recommender system. As there is no information for this One observation in [4] was that providing users with the user to base recommendations on, the recommender system possibility to indicate their preferences through choices re- requires her to provide feedback in order to receive recom- sulted in a bias towards more popular movies, and subse- mendations. This requires quite often significant effort of quently users received more popular recommendations. Al- the user. though this experiment showed that popularity leads to higher In addition, as users watch only a certain amount of movies satisfaction at that moment in the lab setting of the study, over any time period, asking users to provide a set amount such popular recommendations may not provide sufficient of feedback may require them to provide feedback on items value in normal, long term usage scenarios. that they have experienced a longer time ago which will re- 1.3 Memory Effects in Recommender Systems Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed Memory effects could be a possible explanation for this for profit or commercial advantage and that copies bear this notice and the full cita- bias towards popular movies that results in people receiv- tion on the first page. Copyrights for components of this work owned by others than ing overly popular recommendations. In rating-based rec- ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- publish, to post on servers or to redistribute to lists, requires prior specific permission ommender systems memory effects have been shown to in- and/or a fee. Request permissions from [email protected]. fluence how users provide feedback. Bollen et al.[2] have IntRS 2016, September 16, 2016, Boston, MA, USA. demonstrated that ratings given closer to the time the movie c 2016 ACM. ISBN 123-4567-24-567/08/06. $15.00 was actually watched tend to be more extreme than ratings DOI: 10.475/123 4 for movies that have been watched a longer time ago. They Figure 1: User Interface used for the study. The interface to the left is used for participants in the trailer condition, the interface to the right for participants in the non-trailer condition. Within the interface the list of items to choose from is shown below, the trailer and additional metadata is above. argue that this is because of people forgetting information larity of the items a user chooses. about the movies required to rate them, which has conse- In terms of user experience we expect trailers to provide quences for the reliability of the input provided. This same the user with more information, which is expected to be re- effect could result in users choosing items that they recog- flected in the perceived informativeness of the system. As nize in a choice-based preference elicitation task: it is more we expect trailers to motivate users to select less popular likely that people remember more popular movies than less movies, we expect perceived recommendation novelty (the popular movies. opposite of popularity) and diversity to increase. Both nov- elty and diversity may affect system and choice satisfaction. 1.4 Trailers as source of extra information It is hard to formulate expectations about the direction of The current study tries to investigate if this bias towards the effect of trailers on user satisfaction. We expect user sat- picking popular movies can be alleviated by giving users isfaction in this setting to consist of system satisfaction (i.e. additional information to make more informed choices. \how well does this system help me") and choice satisfaction In order to both minimize the effort required and maxi- (\how happy am I with the item that I choose based on this mize the reliability of the input given during the new user system"). In previous research novelty and system satisfac- cold start situation, we propose to use choice-based pref- tion were shown to be negatively correlated [9, 4]. On the erence elicitation and provide the user with additional in- other hand, trailers might make users open to less popu- formation to give her the means to make more informed lar movies and as such novelty could have a positive effect choices. on choice satisfaction. Additionally previous studies have In most recommender systems users can already rely on shown that system satisfaction positively influences choice meta-information like for example genre, cast and a syn- satisfaction[9]. Having the possibility to watch trailers may opsis. A possible additional source of information about a result in an increased system satisfaction and thus choice movie can be found in trailers. Trailers may help the user in satisfaction. Considering all these effects it is hard to fore- two ways. Firstly, trailers can help a user in refreshing the see in what way trailers will affect user experience. memory to provide reliable feedback, alleviating potential The expected effects are shown in Fig. 2 below, with memory problems described in the previous section. Sec- where possible the directions of the hypothesized effect. ondly, even for movies that a user has not seen yet, a trailer can be used to evaluate whether or not a movie is worth watching. This is an advantage of choice-based preference elicitation over rating-based preference elicitation, because in rating-based users typically only rate (and provide infor- mation on) movies they have actually watched. 1.5 Research Question and hypotheses The present research aims to investigate how providing additional information in the form of trailers during choice- based preference elicitation affects the interaction in terms of both objective behavior and subjective user experience. In terms of objective behavior we hypothesize that trailers allow users to make more informed choices and rely less on Figure 2: Path diagram of expected effects. popularity when making these choices. In other words, we expect the possibility to watch trailers to reduce the popu- Table 1: Texts used for the items in the survey, with item factor loadings and factor robustness per aspect of user experience. Considered As- Item Factor Loading pect I got sufficient information on each movie to make a choice. Informativeness Visual information is more important to me for making a choice than written AVE = 0.587 information. α = 0:71 I like the way information about the movies is provided to me in this system. 0.936 The system provided too much information for each movie. I would rather have different information about the movies than what I got -0.647 from the system to make a choice The recommendations contained a lot of variety. Diversity The recommendations covered many movie genres. AVE: 0.655 All the recommended movies were similar to each other. 0.822 α = 0:80 Most movies were from the same genre.