0 Time-Aware Click Model
Total Page:16
File Type:pdf, Size:1020Kb
0 Time-Aware Click Model YIQUN LIU, Tsinghua University XIAOHUI XIE, Tsinghua University CHAO WANG, Tsinghua University JIAN-YUN NIE, Universite´ de Montreal´ MIN ZHANG, Tsinghua University SHAOPING MA, Tsinghua University Click-through information is considered as a valuable source of users’ implicit relevance feedback for commercial search engines. As existing studies have shown that search result position in search engine result page(SERP) has a very strong influence on users’ examination behavior, most existing click models are position-based, assuming that users examine results from top to bottom in a linear fashion. While these click models have been successful, most do not take temporal information into account. As many existing studies have shown, click dwell time and click sequence information are strongly correlated with users’ perceived relevance and search satisfaction. Incorporating temporal information may be important to improve performance of user click models for Web searches. In this paper, we investigate the problem of properly incorporating temporal information into click models. We firstly carry out a laboratory eye-tracking study to analyze users’ examination behavior in different click sequences and find that user common examination path among adjacent clicks is linear. Afterwards, we analyze user dwell time distribution in different search logs and find that we cannot simply use a click dwell time threshold (e.g. 30s) to distinguish relevant/irrelevant results. Finally, we propose a novel click model named Time-Aware Click Model (TACM) that captures the temporal information of user behavior. We compare TACM with a number of existing click models using two real-world search engine logs. Experimental results show that TACM outperforms other click models in terms of both predicting click behavior (perplexity) and estimating result relevance (NDCG). CCS Concepts: •Information systems ! Web searching and information discovery; Retrieval models and ranking; Additional Key Words and Phrases: Click model, click sequence, click dwell time 1. INTRODUCTION Modern search engines record user interactions and use them to improve search quality. In particular, users’ click-through has been successfully used to improve click-through rates (CTR), Web search ranking, query recommendation and suggestions, and so on. Although click-through logs can provide implicit feedback of users’ click preferences [Agichtein et al. 2006b], it is difficult to derive accurate absolute relevance judgments This paper is an extension of [Wang et al. 2015]. Compared with the previous conference version, it introduces a new time-aware click model (TACM) that incorporates click dwell time information. It also includes an extensive experimental assessment of the new model and compares the performance with a number of existing models including PSCM. This work was supported by Tsinghua University Initiative Scientific Research Program (2014Z21032), National Key Basic Research Program (2015CB358700) and Natural Science Foundation (61532011, 61472206) of China. Author’s addresses: State Key Laboratory of Intelligent Technology and Systems, Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2016 ACM. 1539-9087/2016/-ART0 $15.00 DOI: 0000001.0000001 ACM Transactions on Information Systems, Vol. 9, No. 4, Article 0, Publication date: 2016. 0:2 Y. LIU et al. owing to the existence of click noises and behavior biases. Joachims et al. [2005] worked on extracting reliable implicit feedback from user behaviors and concluded that click logs are informative yet biased. Previous studies showed that users’ clicking behaviors are biased towards many aspects such as “position” [Craswell et al. 2008; Joachims et al. 2005], “trust” [Yue et al. 2010], “presentation” [Wang et al. 2013], and so on. To address these problems, researchers have proposed a number of click models to describe users’ practical browsing behavior and to obtain an unbiased estimation of result relevance [Chapelle and Zhang 2009; Dupret and Piwowarski 2008; Guo et al. 2009a]. Most click models follow the findings from Craswell et al. [2008]; Joachims et al. [2005] that users’ attention decreases from top to bottom and assume that users’ potential examination/click paths are unique: the examination/click sequence is consistent with result position. Therefore, these models do not actually take practical temporal information into account. As modern search logs contain a time stamp for each user interaction (e.g., querying, clicking, etc.), we can obtain two important messages from this time stamp: the sequence of user clicks and the dwell time after each user click. For the click sequence information, eye-tracking experiments [Lorigo et al. 2006] showed that only 34% of search users’ scan paths are linear, while over 50% of sessions contain revisiting behaviors (i.e., given a search engine result page (SERP), the user first clicks the result at position i and then clicks the one at position j; j ≤ i) or skipping behaviors. We counted the non-sequential click proportion of multi-click query sessions (when a user clicked two or more results on one SERP) from two commercial search engine logs (Sogou and Yandex; see details in in Table I). We find that nearly one-third (27.9% for Sogou and 30.4% for Yandex) of multi-click sessions contain non-sequential click actions. While most existing click models are based on ranking positions rather than action sequences, the click sequence information is usually ignored, and non- sequential clicking behaviors are not considered, either. Dupret and Liao [2010]; Guo et al. [2012] already showed that the last click in a search session may be more reliable than other clicks. However, the last click performed by a user is not necessarily the one at the lowest position, but the last one in the sequence of clicks. It is thus necessary to take the click sequence information into account. As for click dwell time information, existing studies [Fox et al. 2005; Kim et al. 2014] showed that dwell time on the landing page led by user clicks (click dwell time) is a very strong indicator for user-perceived result relevance and user-perceived search satisfaction. Fox et al. [2005] showed that users are more willing to spend longer durations of time on those pages which are interesting and relevant. Kim et al. [2014] also showed that the longer the dwell time, the more satisfied the user will be, and the more relevant the search result tends to be. Therefore, click dwell time information will be very helpful for us to better understand users’ click behavior and make an accurate relevance estimation. Some existing click models [Wang et al. 2010; Xu et al. 2012, 2010] have tried to cope with click sequence information. These models relax the restrictions on users’ examination sequences (e.g., Wang et al. [2010] assumes that examination sequences can be arbitrary) to increase models’ descriptive power. However, most of these methods abandon the prior knowledge of user examination preference generated from other user behavior studies, which has been found useful. In practice, these models cannot achieve performance that is comparable to other popular click models according to our experimental results. To better understand users’ search interaction processes, we design a laboratory study to analyze users’ practical examination patterns. Our observations confirm clearly that many click behaviors are non-sequential. On the other hand, the ACM Transactions on Information Systems, Vol. 9, No. 4, Article 0, Publication date: 2016. Time-Aware Click Model 0:3 examinations of documents between two clicks usually follow one direction, but with possible skips. This observation shows that some of the assumptions used in the previous position-based models (e.g., the sequential examination assumption) are reasonable in local contexts (i.e., between two clicks). It is thus possible to build a new model upon the existing position-based models by adding new hypotheses. By this means, we not only inherit a framework which has already proved to be effective, but also combine sequential information to better capture users’ preferences for different search results. To better use click dwell time information, we analyze the dwell time distribution for different search logs (Sogou and Yandex). We verified the previous findings in Agichtein et al. [2006a] that clicks with dwell time longer than a certain threshold (e.g., 30 seconds) are good indicators of users’ perceived relevance. We also find that the dwell time distribution in different search engines may be rather different, which means that we must take the distribution factor into consideration to better model user behavior. Combining our findings with the previous conclusions from Kim et al. [2014], we design different mapping functions to model user satisfaction based on click dwell