Learning to Rewrite Queries
Yunlong He Jiliang Tang Hua Ouyang Yahoo! Reseach Michigan State University Yahoo! Reseach Sunnyvale, CA East Lansing, MI Sunnyvale, CA [email protected] [email protected] [email protected] Changsung Kang Dawei Yin Yi Chang Yahoo! Reseach Yahoo! Reseach Yahoo! Reseach Sunnyvale, CA Sunnyvale, CA Sunnyvale, CA [email protected] [email protected] [email protected]
ABSTRACT isfy users’ information needs by understanding the intent from their queries. It is widely known that there exists a semantic gap be- Query rewriting (QRW), which targets to alter a given tween web documents and user queries and bridging this query to alternative queries that can improve relevance per- gap is crucial to advance information retrieval systems. The formance by reducing the mismatches, is a critical task in task of query rewriting, aiming to alter a given query to a modern search engines and has attracted increasing atten- rewrite query that can close the gap and improve informa- tion in recent years [6, 14, 18, 8]. For example, authors tion retrieval performance, has attracted increasing atten- in [14] propose to modify the search queries based on typical tion in recent years. However, the majority of existing query substitutions that web searchers make to their queries; and rewriters are not designed to boost search performance and query rewriting is treated as a machine translation prob- consequently their rewrite queries could be sub-optimal. In lem and statistical machine translation (SMT) models are this paper, we propose a learning to rewrite framework that trained based on query-snippet pairs [18, 8]. However, the consists of a candidate generating phase and a candidate vast majority of existing algorithms focus on either correla- ranking phase. The candidate generating phase provides tions between queries and rewrites or bridging the language us the flexibility to reuse most of existing query rewriters; gap between user queries and web documents, which could while the candidate ranking phase allows us to explicitly be sub-optimal for the goal of query rewriting – improv- optimize search relevance. Experimental results on a com- ing search relevance performance. For example, since the mercial search engine demonstrate the effectiveness of the SMT model aims at translating a sentence from a source proposed framework. Further experiments are conducted language to a fluent and grammatically correct sentence in to understand the important components of the proposed a target language, the SMT model prefers to rewrite the framework. query “how much tesla” as “how much is the tesla” instead of “price tesla”. Therefore it is necessary to explicitly consider 1 Introduction ranking relevance when developing query rewriting methods. In this paper, we propose a learning to rewrite framework Users of the Internet typically play two roles in commercial that consists of (1) candidate generating and (2) candidate search engines. They are information creators that generate ranking as shown in Figure 1. Given a query, we create possi- web documents and they are also information consumers ble candidates via a set of candidate generators in the candi- that retrieve documents for their information needs. It is date generating phase; while given a learning target, we train well known, however, that there is a “lexical chasm” [18] be- a scoring function to rank these candidates from the candi- tween user queries and web documents because they use dif- date generating phase in the candidate ranking phase. The ferent language styles and vocabularies. As a consequence, advantages of this framework are two-fold. First, the can- search engines could be unable to retrieve relevant docu- didate generating phase not only allows us to reuse most of ments even when the issued queries are perfectly describing existing query rewriting algorithms as candidate generators users’ information needs. For example, a user forms a query but also enables us to explore recent advanced techniques “how much tesla”, but web documents in search engines use such as deep learning to build new candidate generators. the expression “price tesla”. Therefore it has become in- Second, the candidate ranking phase gives us flexibility to creasingly important for search engines to intelligently sat- choose the target to optimize for query rewriting; for exam- ple, in this work, we desire to boost relevance performance thus we can choose the relevance target to rank candidates. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed Our contributions are summarized as below: for profit or commercial advantage and that copies bear this notice and the full cita- tion on the first page. Copyrights for components of this work owned by others than • Define the problem of learning to rewrite formally; ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- publish, to post on servers or to redistribute to lists, requires prior specific permission • Propose a learning to rewrite framework that is flexible and/or a fee. Request permissions from [email protected]. to incorporate existing query rewriting algorithms and CIKM’16 , October 24-28, 2016, Indianapolis, IN, USA optimize the relevance performance explicitly; and c 2016 ACM. ISBN 978-1-4503-4073-1/16/10. . . $15.00 DOI: http://dx.doi.org/10.1145/2983323.2983835 • Conduct experiments on a commercial search engine to Figure 1: Flow chart of the proposed learning to rewrite framework
demonstrate the effectiveness of the proposed frame- pure counts of term frequencies. However, the SMT system work. is used as a black box instead of fully tuned for the task of query rewriting. The rest of the paper is organized as follows. In Section 2, we briefly review the related work. In Section 3, we for- Another related research topic is query recommendation. mally define the problem and introduce the proposed frame- Authors in [12] make use of search session data to find query work in detail. In Section 4, we present empirical evaluation pairs frequently co-occurring in the same sessions, which are with discussion. In Section 5, we give conclusion with future used as suggestions for each other. Baeza-Yates et al. [2] work. propose to rank the clustered queries according to two prin- ciples: i) the similarity of the queries to the input query, and 2 Related Work ii) the support of the suggested query, which measures the Query expansion and rewriting have long been important magnitude of answers returned in the past to this query that research topics in information retrieval [3]. Xu and Croft have attracted the attention of users. A follow-up work [4] [23] studied using the top ranked documents retrieved by combines various click-based, topic-based and session based the original query to expand the query. This method suffers ranking strategies and uses supervised learning in order to from the sensitivity to initial ranking results and does not maximize the semantic similarity between the query and the learn from user generated data. Later approaches [6, 7, 14] recommendations. Cao et.al [5] addressed the data sparse- focus on using user query logs to generate query expansions ness issue by summarizing queries into concepts by clus- by collecting signals such as clickthrough rate [6, 24], cooc- tering a click-through bipartite. Then, from session data currence in search sessions [14] or query similarity based on a concept sequence suffix tree is constructed as the query click graphs [7, 1]. Since search logs contain query docu- suggestion model. ment pairs clicked by millions of users, the term correlations Recently, deep learning techniques [16, 21] have been ap- reflect the preference of the majority of users. However, plied on query processing and machine translation tasks. the correlation-based method, as pointed out by [19], suf- For example, authors in [11] generate distributed language fers low precision partly because the correlation model does models for queries to improve the relevance in sponsored not explicitly capture context information and is suscepti- search. Authors in [21] applied recurrent neural networks ble to noise. More recently, natural language technology in on machine translation tasks and achieved state-of-the-art form of statistical machine translation (SMT) [19, 18, 8, 9] performance compared to traditional SMT systems. A hi- has been introduced for the query expansion and rewriting erarchical recurrent encoder-decoder method is proposed in problems. In the SMT system all component models are [20] for the task of query auto-completion. properly smoothed using sophisticated techniques to avoid In this work, our approach is formulated as a learning sparse data problems while the correlation model relies on to rewrite framework and its key component is candidate ranking. Therefore the candidate ranking phase is similar with traditional techniques. Therefore a candidate gener- to the learning to rank framework in terms of many aspects ator based on deep learning techniques could be comple- such as loss functions [25, 22] and ranking features [15]. mentary to traditional ones and provides potentially better candidates. Recurrent Neural Network (RNN) is neural se- 3 Learning to Rewrite Framework quence model that achieves state of the art performance on The query rewriting problem aims to find the query rewrites many important sequential learning tasks. The long short- of a given query for the purpose of improving the relevance term memory (LSTM) is a one of the most popular RNN of the information retrieval system. The proposed frame- instance. It can learn long range temporal dependencies and work formulates the query rewriting problem as an opti- mitigate the vanishing gradient problem. We propose to use mization problem of finding a scoring function F (q, r) which the Sequence-to-Sequence LSTM model [21] to build a new assigns a score for any pair of query q and its rewrite can- candidate generator. In model training, we treat the original query as input sequence, and use its rewrite queries as target didate r. The framework assumes that G = {g1, g2, . . . , gM } is a set of M candidate generators. Candidate generators output sequences. In prediction, the most probable output could be any existing query rewriting techniques and we sequences are obtained by a beam-search method elaborated will introduce more details in Subsection 3.1. In the can- at the end of this section and are used as the query rewrite didate generating phase, we use candidate generators in G candidates. J to generate a set of rewrite candidates for a given query q Given an input sequence x1 = x1, ··· , xJ , a standard J as R = {r1, ··· , rn} where n is the number of generated RNN computes the hidden vector sequence h1 = h1, ··· , hJ J rewrite candidates. Each pair of query q and its rewrite and output vector sequence y1 = y1, ··· , yJ by iterating the candidate ri, i.e., (q, ri), is scored by the function F (q, ri). following equations from j = 1 to J: The rewrite candidates from R are then ranked based on hj = H(Wxhxj + Whhhj−1 + bh) the scores {F (q, r1),F (q, r2),...,F (q, rn)} in the candidate ranking phase. A key step of the learning to rewrite problem yj = Whyhj + by is how to obtain the scoring function F . where the W·,· terms denote weight matrices, the b· terms Let F = {f :(q, r) 7→ f(q, r) ∈ R} be the functional space denote bias vectors and H is the hidden layer function. of the scoring functions for any pair of query and rewrite can- For the version of LSTM used in the sequence to sequence didate and Q = {q1, ··· , qm} be a set of m queries. We use R = {r , ··· , r } to denote the set of rewrite candidates model, the gates and cells are implemented by the following i i,1 i,ni composite functions, where we followed [10]: of query qi generated by G where ni is the number of rewrite candidates for qi. For each query qi, we further assume that ij = σ(Wxixj + Whihj−1 + bi) Ii is the learning target that encodes the observed informa- fj = σ(Wxf xj + Whf hj−1 + Wcf cj−1 + bf tion about the quality of rewrite candidates in Ri. Note c = f c + i tanh(W x + W h + b ) that the forms of Ii are problem-dependent that could be j j j−1 j xc j hc j−1 c the label for each pair (q, ri) or the preferences among Ri oj = σ(Wxoxj + Whohj−1 + Wcocj + bo) for qi. With aforementioned notations and definitions, the hj = oj tanh(cj ). problem of searching in F for a scoring function F (q, r) is formally stated as the following minimization problem: In a sequence to sequence LSTM, we want to estimate m the conditional probability p(y1, ··· , yI |x1, ··· , xJ ) where X F = arg min L(f, qi, Ri, Ii). (1) x1, ··· , xJ is an input sequence and y1, ··· , yI is its cor- f∈F i=1 responding output sequence whose length I may differ from J. The LSTM computes this conditional probability by first The exact forms of the loss function L(f, q , R , I ) depends i i i obtaining the fixed dimensional representation v of the in- on the learning target I and we will further discuss these i put sequence x , ··· , x given by the last hidden state of in Subsection 3.2. As illustrated in Figure 1, the proposed 1 J the LSTM, and then computing the probability of y , ··· , y framework consists of two key steps – candidate generat- 1 I with a standard LSTM-LM formulation whose initial hidden ing and candidate ranking. In the following subsections, we state is set as the representation v of x , ··· , x : elaborate them with details. 1 J I 3.1 Candidate Generating Y p(y , ··· , y |x , ··· , x ) = p(y |v, y , ··· , y ), (2) We have two ways to obtain candidate generators for can- 1 I 1 J i 1 i−1 i=1 didate generating. One is to treat existing query rewriters as candidate generators. The other is to explore advanced where p(yi|v, y1, ··· , yi−1) is represented with a softmax over techniques to build new candidate generators. Two candi- all the words in the vocabulary. Note that we require that date generators used in the paper include one query rewriter each query ends with a special end-of-query symbol“
query r = rt1, rt2, ··· , rtn, < EOQ > given the query q = has the following form: qt1, qt2, ··· , qtm, < EOQ >, thus the training objective is L(f, qi, Ri, Ii) = 1 X (q, r) ∈ D log p(r|q), X |D| l(f(qi, ri,j ) − f(qi, ri,k)), ri,j ri,k where D is the training data set and p(r|q) is calculated ac- cording to Eq. (2). Once training is complete, we feed origi- where a typically used l includes the squared hinge loss func- tion l(t) = max(0, 1 − t)2 and the logistic function l(t) = nal queries to the model and produce rewrite candidates by −t finding the most likely rewrites according to the LSTM – We log(1 + e ). search for the most likely query rewrites using a simple left- List-wise loss If the learning target Ii includes a complete to-right beam search decoder instead of an exact decoder. order of the rewrite candidates Ri for qi in terms of rewriting It maintains a small number B of partial rewrites, where quality, the loss function can then be formulated to calculate partial rewrites are prefixes of some query rewrite candi- the cost of the difference between the ranking order gener- dates. At each time-step, we extend each partial rewrite in ated by the scores f(qi, ri,1), ··· , f(qi, ri,ni ) and the ground- the beam with every possible word in the vocabulary. We truth ranking order. For a more concrete example, let Ii be discard all but the B most likely rewrites according to the given in the form of (yi(1), yi(2), ·, yi(ni)) where yi(j) is the model’s log probability. As soon as the “
Point-wise loss Assuming that for each pair (qi, ri,j ), a candidates. One straightforward way is to use human label- grade label yi,j is available which indicates the quality of ri,j ing. However, it is not practical, if not impossible, to achieve as a query rewrite of qi, the loss function that is appropriate this for a web-scale query rewriting application. First, it is for point-wise learning target Ii is: very time and effort consuming to label millions of query rewriting pairs. Second, the relevance performance depends ni X on many components of a search engine such as relevance L(f, q , R , I ) = l(f(q , r ), y ), i i i i i,j i,j ranking algorithms, thus it is extremely difficult for human j=1 editors to compare the quality of rewrite candidates. Third, where l can be a squared error loss function, i.e., l(x, y) = for a commercial search engine, its components are typically kx−yk2 if y takes numerical values, or a logistic loss function, evolved rapidly and in order to adapt to these changes, hu- i.e., l(x, y) = log(1 + e−xy) if y ∈ {−1, +1}. man labels are consistently and continuously needed. There- fore it is desirable for an automatic approach to generate Pair-wise loss When the preference between a pair of rewrite learning target. queries ri,j and ri,k is available, the loss function for the In this work, we specifically focus on boosting the rel- pair-wise learning target can be constructed as the sum over evance performance via query rewriting, thus the learning pair-wise loss function. For example, let ri,j ri,k denote target should indicate the quality of the rewrite candidates the learning target that ri,j is a preferred query rewrite to from the perspective of search relevance. Intuitively a better ri,k given the original query qi. The pair-wise loss function rewrite candidate could attract more clicks to its retrieved documents. In other words, the number of clicks on the re- lower positions. With these intuitions, the discounted turned document from a rewrite candidate could be a good clicknum strategy defines yq,r from Cr as: indicator about its quality in terms of relevance. These in- k tuitions pave us a way to develop an automatic approach to X cr,u y := i , generate learning target based on the query-document click q,r i + 1 i=1 graph that is extracted from search logs. A click graph is a query-document bipartite graph which where the contribution of cr,ui in yq,r is penalized by its position in the ranking list. consists of the triplets (q, u, nq,u), where q denotes a query, u denotes a document and the edges in the bipartite graph • Discounted log clicknum: Click numbers in Cr could indicate the co-click between queries and documents and the vary dramatically. Some have an extremely large num- weights are co-click numbers nq,u, i.e., the number of clicks ber of clicks; while others have a few. In this scenario, (accumulated through a time period and a population of the documents with large numbers of clicks could dom- users) on u when the issued query is q. The click graph inate the learning target. Therefore the discounted log is constructed from user search logs, thus it contains rich clicknum strategy applies a log function to click num- information of users’ understandings and behaviors about bers as: queries and documents. There are many successful appli- k cations based on the query-document graph such as query X log2(cr,u ) y := i . suggestion [5] and query clustering [2]. Next we introduce q,r i + 1 i=1 our approach to generate learning targets from the query- document graph. • Logdiscounted log clicknum: The Discounted clicknum For each query and query rewrite pair (q, r), we estimate strategy could over-penalize the click numbers. Similar the click numbers if we alter q to r from the query-document to Discounted log clicknum, Logdiscounted log click- click graph as illustrated in Figure 3: num also applies a log function to positions as:
k • We use the query r to retrieve top k documents from X log2(cr,ui ) yq,r := . the search engine as U = {u1, u2, ··· , uk} where ui log(i + 1) indicates the ranked i-th document in the returned i=1 list. For example, we retrieve k = 4 documents for In some cases, we might not need to rewrite the origi- the rewrite candidate r1. nal query q. For example, the quality of q could be better than that of all available rewrite candidates. Therefore in • The click number of the document ui in U with the in- practice, we also assign the original query q as one rewrite formation intent of q c is estimated as n if there r,ui q,ui candidate. Following the same procedure to generate the are clicks between query q and document ui in the point-wise learning target for the query rewrite r, we can query-document graph (e.g., c = n in Figure 3) r1,d2 q,d2 generate one for the original query as y . If r is better than and 0 ( e.g., c = 0 in Figure 3 ) otherwise. The ra- q,q r2,d2 q, the documents retrieved by r should be more relevant and tionale of using (q, ui) instead of (r, ui) is that r could the number yq,r should be higher than yq,q. Similarly, if r drift the intent of q and we want to maintain the origi- is worse than the original query or even changes the query nal intent. For example, if r changes the intent of q, ui intent, the documents retrieved by r should be less relevant could be completely irrelevant to q and it is reasonable and the number y should be lower than y . Therefore to estimate c = 0. Let C = {c , c , . . . , c } q,r q,q r,ui r r,u1 r,u2 r,uk by treating the original query q as a rewrite candidate, the be the estimated click numbers for its top k retrieved proposed framework allows rewriting as the original query. documents. We maintain the intent of the original query q by using (q, ui) instead of (r, ui); thus we can directly obtain pair-wise With the estimated click numbers Cr and document po- sitions as shown in Figure 3, we can generate the learning and list-wise learning targets from the point-wise targets. target. Next we illustrate how to generate the point-wise For example, the pair-wise labels of two rewrite candidates ri and rj can be obtained by comparing yq,r and yq,r ; and learning target yq,r from Cr. Our basic idea is to aggre- i j similarly we can get an order of the quality of all rewrite gate the estimated click numbers in Cr to a unified target candidates of q. yq,r. In this paper, we investigate the following aggregating strategies: 3.2.3 Feature Functions To find the best scoring function, we need to solve the opti- • Clicknum: Intuitively the total click numbers can in- mization problem (1) in the functional space F, which is dicate the quality of a rewrite candidate in terms of usually very challenging. A practical way to reduce the relevance. Therefore the Clicknum strategy is to sum complexity of the optimization problem is to assume that click numbers in C as y : r q,r the scoring functions in F have some special structures and k X then search for the optimal F in the constrained space. A yq,r := cr,ui common approach is to assume that every f ∈ F is a linear i=1 combination of m feature functions, namely: m • Discounted clicknum: In addition to click numbers, X the positions of documents in the ranking list are also f(q, r) = λihi(q, r), (3) important. Ideally we should rank documents with a i=1 large number of clicks higher; hence we need to penal- where each hi(q, r) is the i-th feature function and λi con- ize these documents with a large number of clicks but trols the contribution of hi(q, r). There are also other ap- Figure 3: Generating click numbers Cr from the query-document click graph proaches to build the scoring functions. For example we can 4.1 Experimental Settings assume the scoring functions in F are ensembles of decision trees such that the search space is constrained by the num- ber of trees and the depth of the decision trees; and a gra- dient boosting algorithm can be applied to find the optimal In the candidate generating phase, we choose two candidate scoring function. In this paper, we focus on the approach generators – (1) the statistical machine translation (SMT) in Eq. (3) while we would like to leave the investigation of based query rewriter [18, 8] and (2) the LSTM based query other approaches as one future work. rewriter that is introduced in Subsection 3.1. We need a large parallel training data with queries and the correspond- For each pair of query and rewrite candidate (q, r), we can ing written queries to train both SMT and LSTM. Follow- build three groups of feature functions – (1) query features: ing the common practice in [18, 8], we prepare the training features are extracted from only the original query q; (2) data for candidate generators from the query-document click rewrite features: features are extracted from only the rewrite graph that is collected from the query log of a commercial candidate r and (3) pair features: features are extracted search engine. We first get query and document pairs that from both q and r. Before introducing the details about have co-clicks from the query-document click graph. Typ- these features, we first introduce notations we use in their ically titles of documents are very informative and repre- definitions. Let fq and fr be the query frequencies obtained sentative; hence we can consider the title of a document as from the search log. Let W = {W1,W2,...,WN } be the dic- an ideal query to retrieve this document. Therefore from tionary with N words. We use q = {wq,1, wq,2, . . . , wq,N } the extracted query and document pairs, we form a paral- to indicate the vector representation of q where wq,i is the lel training data with queries and titles. In this work, we frequency of Wi in q. Similarly we represent r as r = extract 800 millions of query and title pairs from the query- {wr,1, wr,2, . . . , wr,N }. We further assume that Uq and Ur document graph to train both SMT and LSTM models. To are the sets of URLs connecting to q and r in the query- obtain the score function, we use each of these two candi- document graph, respectively. The definitions and descrip- date generators to generate 10 rewrite candidates for a given tions of these features are summarized in Table 1. query and the learning targets are generated from the query- url graph with more than 2 billions of queries. Note that in this work, we only focus on the point-wise label since pair- 4 Experiment wise and list-wise labels can be obtained from the point-wise label as discussed in Subsection 3.2.
In this section, we conduct experiments to validate the effec- tiveness of the proposed framework via a commercial search In this work, we aim to use query rewriting to boost the engine. In particular, this section mainly aims to answer relevance performance. Meanwhile it is very challenging for the following three questions – (1) what is the quality of the human editors to judge the quality of rewriting candidates rewriting query created by the proposed framework? (2) in terms of relevance performance. For example, it is diffi- what is the impact of different learning targets on the per- cult for editors to reach an agreement about the quality of formance of the proposed framework and (3) what are the “how much tesla” and “price tesla” from the relevance per- effects of the feature functions? To answer the first question, spective since the relevance performance strongly depends we compare the proposed framework with the state-of-the- on many components of a given search engine. Hence we art baseline. Then we study the effects of different learning propose to evaluate the quality of query rewriting via rele- target generating strategies and feature functions on the per- vance performance. We further use Discounted cumulative formance of the proposed framework to answer the second gain (DCG) as the metric to assess the search relevance per- and third questions, respectively. We begin by introducing formance. DCG has been widely used to assess relevance in the experimental settings. the context of search engines [13]. For a ranked list of N Table 1: Summary of Group of Features for a Pair of Query and query rewrite (q, r)
Feature Group Feature PN h1 – Number of words in q as i=1 wq,i h2 – Number of stop words in q: Sq Query Features h3 – Language model score of the query q: Lq h4 – The query frequencies of the query q: fq h5 – The average length of words in q: Aq PN h6 – Number of words in r as i=1 wr,i h7 – Number of stop words in r: Sr Rewrite Features h8 – Language model score of the query rewrite r:Lr h9 – The query frequencies of the query rewrite r:fr h10 – The average length of words in r: Ar |Uq ∩Ur | h11 – Jaccard similarity of URLs as |Uq ∪Ur | h12 – Difference between the frequencies of the original query q and the rewrite candidate q: fq − fr PN i=1 wq,iwr,i h13 – Word-level cosine similarity between q and r: q q PN w2 PN w2 Pair Features i=1 q,i i=1 r,i PN PN h14 – Difference between the number of words between q and r: i=1 wq,i − i=1 wr,i h15 – Number of common words in q and r h16 – Difference of language model scores between q and r: Lq − Lr h17 – Difference between the number of stop words between q and r: Sq − Sr h18 – Difference between the average length of words in q and r: Aq − Ar documents, we use the following variation of DCG, all feature functions defined in Table 1. More details about
N the effects of the learning target generating strategies and X Gi DCG = feature functions on the proposed framework will be inves- N log (i + 1) tigated in the following subsections. i=1 2 The performance comparison over all 2407 queries is re- where Gi represents the weight assigned to the label of the ported in Table 2. We cannot report absolute DCG numbers document at position i, e.g. 10 for Perfect match, 7 for for confidentiality reasons; hence we report the DCG rela- Excellent match, 3 for Good match, etc. Higher degree of tive improvement compared to SMT. Note that it is evident relevance corresponds to higher value of the weight. We from t-test that all improvement is significant. use the symbol DCG to indicate the average of this value From Table 2, we make the following observations: over a set of testing queries in our experiments. To evaluate the search relevance performance, 2407 queries are sampled • With SMT as the only candidate generator, LQRW- from the first 75% traffic of the total query list (sorted by SMT obtains remarkable improvement compared to search frequency) of a commercial search engine and seg- SMT. SMT query rewriter does not consider relevance; ment queries into 959 top queries (0−40%), 751 torso queries hence rewrite candidates from SMT are not optimal for (40% − 60%) and 697 tail queries (60% − 75%). Web docu- the relevance performance. While LQRW-SMT explic- ments are retrieved from a multiple billion index and ranked itly incorporates the relevance performance into query by a fine-tuned commercial ranking system. For each pair rewriting; of original query q and retrieved document d, highly trained • By combining candidates generated by SMT and LSTM, human editors are requested to assign one of the five grades LQRW-SMT-LSTM outperforms LQRW-SMT. Although to determine the relevance of the document given the query. SMT and LSTM are trained with the same set of train- In this paper, we report the relevance performance in terms ing data, candidates from LSTM are complementary of DCG1, DCG3 and DCG5. to these from SMT. 4.2 Quality of Query Rewriting We can reuse most of existing query rewriters in the can- Table 2: Overall Search Performance Improvement didate generating phase and we choose the state-of-the-art (%) in terms of DCG Compared to the Query query rewriter, i.e., the SMT method proposed in [18] as Rewriter SMT. one of our candidate generators, hence we compare the pro- posed framework with the SMT method in terms of rele- Methods DCG DCG DCG vance performance. Note that we do not compare the pro- 1 3 5 posed framework with other query rewriters such as those LQRW-SMT +3.01% +2.65% +2.38% in [6, 14] since we only used the SMT method as the candi- LQRW-SMT-LSTM +3.03% +3.14% +2.95% date generator except the newly proposed LSTM candidate generator. For the proposed framework, we create two vari- The relevance performance of modern search engines is ants – (1) LQRW-SMT that only uses SMT to generate 20 strongly dependent on user behavior data. However, for rewrite candidates; and (2) LQRW-SMT-LSTM that uses torso and tail queries, the user behavior data is very sparse each of SMT and LSTM to generate 10 rewrite candidates, and noisy; hence the relevance performance may be not good respectively. In this experiment, we adopt the Logdiscounted for these queries. It is therefore interesting to show the log clicknum strategy to generate the learning target and use effects of query rewriting methods on different query traffic bands. The performance comparisons for top, torso and tail • Clicknum obtained the largest performance reduction. queries are shown in Table 3 and it can be observed that: This result suggests that in addition to click numbers, it is also necessary and important to consider docu- • Both LQRW-SMT and LQRW-SMT-LSTM consistently ment positions when generating the learning target outperform SMT for all traffic bands, i.e., top, torso and tail queries; • Discounted log clicknum outperforms Discounted click- num which indicates that the learning targets could be • The improvement of LQRW-SMT and LQRW-SMT- dominated by these documents with extremely large LSTM on torso and tail queries is higher than that numbers of clicks. on top queries. This property of the proposed frame- work has its practical significance since improving the • The performance of Discounted log clicknum degrades relevance of torso and tail queries are usually difficult. slightly compared to Logdiscounted log clicknum. Dis- counted log clicknum could over-penalize the click num- bers by positions. Table 3: Search Performance Improvement (%) in terms of DCG in Different Traffic Bands Compared to the Query Rewriter SMT. Table 5: The Relative Performance Changes Com- pared to Logdiscounted log clicknum. Note that “+” and Top “-” indicate performance improvement and decline, respectively. Methods DCG1 DCG3 DCG5 LQRW-SMT +0.72% +1.48% +1.51% Learning target DCG DCG DCG LQRW-SMT-LSTM +0.66% +1.62% +1.84% 1 3 5 Clicknum -2.04% -1.79% -1.58% Torso Discounted clicknum -0.95% -0.73% -0.77% LQRW-SMT +4.64% +3.43% +3.23% Discounted log clicknum -0.43% -0.34% -0.47% LQRW-SMT-LSTM +4.93% +4.31% +3.97% Tail LQRW-SMT +6.14% +4.13% +3.53% 4.4 The Impact of Feature Functions LQRW-SMT-LSTM +8.22% +5.46% +4.55% The parameter λi controls the contribution of the feature function hi. Therefore in this subsection, we study the im- To deeply understand above observations, we calculate pact of feature functions via model parameters λi. We first the coverage of different query rewriters in different traffic rank features by the absolute values of the t-statistic for bands and the results are shown in Table 4. Note that here model parameters and the top 10 features are shown as fol- the coverage of a query rewriter is defined as the percentage lows: of queries whose rewrite candidates generated by the query rewriter are different from themselves. We note that (1) all 1. h11: Jaccard similarity of URLs of q and r; query writers have higher coverage for torso and tail queries 2. h12: Difference between the frequencies of q and r; than top queries; (2) the coverage of LQRW-SMT-LSTM is consistently higher than LQRW-SMT; and (3) the cover- 3. h13: Word-level cosine similarity between q and r; age of both LQRW-SMT and LQRW-SMT-LSTM increases more for torso and tail queries than top queries compared 4. h9: The query frequencies of the query rewrite r; to SMT. 5. h8: Language model score of the query rewrite r; Table 4: Coverage of Query Writing in Different 6. h14: Difference between the number of words between Traffic Bands. q and r;
Methods Top Torso Tail 7. h15: Number of common words in q and r SMT 10.9% 20.7% 29.5% 8. h : Number of stop words in r LQRW-SMT 14.8% 30.4% 43.2% 7
LQRW-SMT-LSTM 16.1% 37.5% 51.9% 9. h17: Difference between the number of stop words be- tween q and r
4.3 The Impact of Learning Target Generat- 10. h3: Language model score of the query q ing Strategies Among top 10 features, there are 6 pair features, 3 rewrite In Subsection 3.2, an automatic approach with four strate- features and 1 query feature. This result indicates the im- gies is proposed to generate the learning targets. In this portance of pair features. We also investigate the perfor- subsection, we investigate the impact of these learning tar- mance of the proposed framework with only pair features get generating strategies on the performance of the proposed and top 10 features, respectively. The performance changes framework. We choose the Logdiscounted log clicknum as compared to the proposed framework with all features are the basic strategy and the performance comparison of other shown in Table 6. Note that “*” in the table indicates that strategies against the baseline is shown in Table 5. Note that the changes are not significant. When we only use pair fea- “+” and “-” indicate performance improvement and decline, tures, the performance reduces slightly; while there is no respectively. significant performance reduction when we only use top 10 We make the following observations from Table 5: features. Table 6: The Relative Performance Changes Com- ACM international conference on Information and pared to the Proposed Framework with All Features. knowledge management, pages 696–703. ACM, 2005. Note that “*” indicates the changes are not signifi- [8] J. Gao, X. He, S. Xie, and A. Ali. Learning lexicon cant. models from search logs for query expansion. In Proceedings of the 2012 Joint Conference on Empirical Features DCG1 DCG3 DCG5 Methods in Natural Language Processing and Pair Features -0.44% -0.52% -0.61% Computational Natural Language Learning, pages Top 10 Features -0.15%* -0.13%* -0.17%* 666–676. Association for Computational Linguistics, 2012. [9] J. Gao and J.-Y. Nie. Towards concept-based 5 Conclusion translation models using search logs for query This paper proposes a learning to rewrite framework that expansion. In Proceedings of the 21st ACM is flexible to incorporate existing query rewriting algorithms international conference on Information and such as statistical machine translation and recent advanced knowledge management, page 1. ACM, 2012. methods in deep learning such as sequence to sequence LSTM. [10] A. Graves et al. Supervised sequence labelling with The framework formulates the query rewriting problem as recurrent neural networks, volume 385. Springer, 2012. an optimization problem of finding a scoring function to op- [11] M. Grbovic, N. Djuric, V. Radosavljevic, F. Silvestri, timize the relevance performance explicitly. One innovation and N. Bhamidipati. Context-and content-aware of the proposed method is that the supervised information embeddings for query rewriting in sponsored search. could be cheaply constructed from the click graph. Experi- In Proceedings of the 38th International ACM SIGIR ments conducted on a commercial search engine demonstrate Conference on Research and Development in that the framework is very effective to improve search rele- Information Retrieval, pages 383–392. ACM, 2015. vance. [12] C.-K. Huang, L.-F. Chien, and Y.-J. Oyang. Relevant In this future, we plan to explore the framework from term suggestion in interactive web search based on three perspectives. First, we plan to investigate other forms contextual information in query session logs. Journal of scoring functions. Second, we plan to investigate the ef- of the American Society for Information Science and fect of including more candidate generating methods in our Technology, 54(7):638–649, 2003. framework. Third, we will study methods for generating [13]K.J ¨arvelin and J. Kek¨al¨ainen. Cumulated gain-based pair-wise and list-wise learning targets that are complemen- evaluation of ir techniques. ACM TOIS. tary to the point-wise learning target generated from the [14] R. Jones, B. Rey, O. Madani, and W. Greiner. click graph. Generating query substitutions. In Proceedings of the 15th international conference on World Wide Web, 6 References pages 387–396. ACM, 2006. [15] T.-Y. Liu. Learning to rank for information retrieval. [1] I. Antonellis, H. G. Molina, and C. C. Chang. Foundations and Trends in Information Retrieval, Simrank++: query rewriting through link analysis of 3(3):225–331, 2009. the click graph. Proceedings of the VLDB Endowment, [16] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and 1(1):408–421, 2008. J. Dean. Distributed representations of words and [2] R. Baeza-Yates, C. Hurtado, and M. Mendoza. Query phrases and their compositionality. In Advances in recommendation using query logs in search engines. In neural information processing systems, pages Current Trends in Database Technology-EDBT 2004 3111–3119, 2013. Workshops, pages 588–596. Springer, 2005. [17] T. Qin, X.-D. Zhang, M.-F. Tsai, D.-S. Wang, T.-Y. [3] R. Baeza-Yates, B. Ribeiro-Neto, et al. Modern Liu, and H. Li. Query-level loss functions for information retrieval, volume 463. ACM press New information retrieval. Information Processing & York, 1999. Management, 44(2):838–855, 2008. [4] R. Baeza-Yates and A. Tiberi. Extracting semantic [18] S. Riezler and Y. Liu. Query rewriting using relations from query logs. In Proceedings of the 13th monolingual statistical machine translation. ACM SIGKDD international conference on Knowledge Computational Linguistics, 36(3):569–582, 2010. discovery and data mining, pages 76–85. ACM, 2007. [19] S. Riezler, Y. Liu, and A. Vasserman. Translating [5] H. Cao, D. Jiang, J. Pei, Q. He, Z. Liao, E. Chen, and queries into snippets for improved query expansion. In H. Li. Context-aware query suggestion by mining Proceedings of the 22nd International Conference on click-through and session data. In Proceedings of the Computational Linguistics-Volume 1, pages 737–744. 14th ACM SIGKDD international conference on Association for Computational Linguistics, 2008. Knowledge discovery and data mining, pages 875–883. [20] A. Sordoni, Y. Bengio, H. Vahabi, C. Lioma, ACM, 2008. J. Grue Simonsen, and J.-Y. Nie. A hierarchical [6] H. Cui, J.-R. Wen, J.-Y. Nie, and W.-Y. Ma. recurrent encoder-decoder for generative context-aware Probabilistic query expansion using query logs. In query suggestion. In Proceedings of the 24th ACM Proceedings of the 11th international conference on International on Conference on Information and World Wide Web, pages 325–332. ACM, 2002. Knowledge Management, pages 553–562. ACM, 2015. [7] B. M. Fonseca, P. Golgher, B. Pˆossas, [21] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to B. Ribeiro-Neto, and N. Ziviani. Concept-based sequence learning with neural networks. In Advances interactive query expansion. In Proceedings of the 14th in neural information processing systems, pages [24] W. V. Zhang and R. Jones. Comparing click logs and 3104–3112, 2014. editorial labels for training query rewriting. In WWW [22] F. Xia, T.-Y. Liu, J. Wang, W. Zhang, and H. Li. 2007 Workshop on Query Log Analysis: Social And Listwise approach to learning to rank: theory and Technological Challenges, 2007. algorithm. In Proceedings of the 25th international [25] Z. Zheng, K. Chen, G. Sun, and H. Zha. A regression conference on Machine learning, pages 1192–1199. framework for learning ranking functions using ACM, 2008. relative relevance judgments. In Proceedings of the [23] J. Xu and W. B. Croft. Query expansion using local 30th annual international ACM SIGIR conference on and global document analysis. In Proceedings of the Research and development in information retrieval, 19th annual international ACM SIGIR conference on pages 287–294. ACM, 2007. Research and development in information retrieval, pages 4–11. ACM, 1996.