Inferring Networks of Substitutable and Complementary Products

1 2 2,3 Julian McAuley∗ , Rahul Pandey† , and Jure Leskovec‡

1Department of Computer Science, UC San Diego 2Pinterest 3Department of Computer Science, Stanford University

July 1, 2015

Abstract For example, when a user in an online store is examining t- shirts she should receive recommendations for similar t-shirts, In a modern recommender system, it is important to understand or otherwise jeans, sweatshirts, and socks, rather than (say) a how products relate to each other. For example, while a user is movie even though she may very well be interested in it. From looking for mobile phones, it might make sense to recommend these relationships we can construct a product graph, where other phones, but once they buy a phone, we might instead want nodes represent products, and edges represent various types of to recommend batteries, cases, or chargers. These two types product relationships. Such product graphs facilitate many im- of recommendations are referred to as substitutes and comple- portant applications: Navigation between related products, dis- ments: substitutes are products that can be purchased instead covery of new and previously unknown products, identification of each other, while complements are products that can be pur- of interesting product combinations, and generation of better chased in addition to each other. and more context-relevant recommendations. Here we develop a method to infer networks of substitutable Despite the importance of understanding relationships be- and complementary products. We formulate this as a supervised tween products there are several interesting questions that make link prediction task, where we learn the semantics of substi- the problem of building product graphs challenging: What are tutes and complements from data associated with products. The the common types of relationships we might want to discover? primary source of data we use is the text of product reviews, What data will allow us to reliably discover relationships be- though our method also makes use of features such as ratings, tween products? How do we model the semantics of why cer- specifications, prices, and brands. Methodologically, we build tain products are related?—For example, the semantics of why topic models that are trained to automatically discover topics a given t-shirt might be related to a particular pair of jeans are from text that are successful at predicting and explaining such intricate and can only be captured by a highly flexible model. relationships. Experimentally, we evaluate our system on the And finally, how do we scale-up our methods to handle graphs Amazon product catalog, a large dataset consisting of 9 million of millions of products and hundreds of millions of relations? products, 237 million links, and 144 million reviews. Inferring networks of product relationships. Here we are interested in inferring networks of relationships between mil- 1 Introduction lions of products. Even though our method can be used to learn any type of relationship, we focus on identifying two types of arXiv:1506.08839v1 [cs.SI] 29 Jun 2015 Recommender systems are ubiquitous in applications ranging links between products: substitutes and complements [21]. Sub- from e-commerce to social media, video, and online news plat- stitutable products are those that are interchangeable—such as forms. Such systems help users to navigate a huge selection one t-shirt for another, while complementary products are those of items with unprecedented opportunities to meet a variety of that might be purchased together, such as a t-shirt and jeans. special needs and user tastes. Making sense of a large num- We design a system titled Sceptre (Substitute and Comple- ber of products and driving users to new and previously un- mentary Edges between Products from Topics in Reviews), that known items is key to enhancing user experience and satisfac- is capable of modeling and predicting relationships between tion [2, 14, 15]. products from the text of their reviews and descriptions. At its While most recommender systems focus on analyzing pat- core, Sceptre combines topic modeling and supervised link pre- terns of interest in products to provide personalized recom- diction, by identifying topics in text that are useful as features mendations [14, 30, 34, 36], another important problem is to for predicting links between products. Our model also handles understand relationships between products, in order to surface additional features such as brand, price, and rating information, recommendations that are relevant to a given context [17, 35]. product category information, and allows us to predict multi- e.g. ∗[email protected] ple types of relations ( substitutes and complements) simul- †[email protected] taneously. Moreover, Sceptre harnesses the fact that products ‡[email protected] are arranged in a category hierarchy and allows us to extend

1 candidate-generation step in providing better and more context- relevant recommendations. ery product Substitutes 2 Related work Complements The basic task of a recommender system is to suggest relevant items to users, based on their opinions, context, and behavior. One component of this task is that of estimating users’ ratings or rankings of products [14], e.g. by matrix factorization [15] or collaborative filtering [17]. Our goal here is related but complementary to rating estimation as we aim to discover relations between products. In principle the types of relationships in which we are interested can be mined from behavioral data, such as browsing and co-purchasing logs. For example, Amazon allows users to navigate between products through links such as ‘users who bought X also bought Y’ and ‘users who viewed X also viewed Y’ [17]. Such a ‘co-counting’ solution, while simple, has a few shortcomings, for example it may produce noisy recommendations for infrequently-purchased products, and has limited ability to explain the recommendations it provides. More sophisticated solutions have been proposed that make use of browsing and co-purchasing data (e.g. [35]), but in contrast to Figure 1: Sceptre learns the concept of substitute and comple- such ‘behavioral-based’ solutions our goal is to learn the se- ment goods from product information (descriptions, reviews, mantics of ‘what makes products related?’ in order to generate etc.). Given a query item, Sceptre allows us to generate sub- new links, adapt to different notions of relatedness, and to un- stitute and complementary recommendations as shown above. derstand and explain the features that cause humans to consider products to be related. Topic models are a fundamental building block of text modeling [3, 4, 5] and form the cornerstone of our model. A variety this hierarchy to discover ‘micro-categories’—fine-grained cat- of works have used topic models within recommender systems egories of closely related products. (e.g. [6, 10, 11, 22, 23, 28, 31, 32]), though generally with the An example of the output of Sceptre is shown in Figure 1. goal of predicting user ratings (or opinions) rather than learn- Here, given a query item (a hiking boot), our system identifies ing relationships between products as we do here. More specif- a ranked list of potential substitutes (other hiking boots), and ically, our work builds on topic models for networks: Block- complements (heavy-duty socks, shoe polish, etc.). LDA [1], topic-link LDA [18], and relational topic models [7] We train Sceptre on a large corpus of 9 million products from all attempt to identify topics that explain the links in document Amazon, with 237 million connections derived from browsing networks. A promising line of work uses such ideas to model and co-purchasing data. We evaluate Sceptre in terms of its ac- social and citation networks [8, 33, 34]. However, these meth- curacy at link prediction and ranking, where we find it to be ods have trouble scaling to large networks, while Sceptre scales significantly more accurate than alternatives. We also use Scep- to corpora with millions of documents (products) and hundreds tre to build a product graph, where for every product we recom- of millions of links. mend a list of the most related complementary and substitutable Last, a related branch of work aims to enhance e-commerce products. Finally, we show that Sceptre can be applied in ‘cold- using browsing data. For example, [12] aims to forecast com- start’ settings, by using other sources of text when reviews are mercial intent based on query logs; and in [26] the authors use unavailable. Overall, we find that the most useful source of in- query data to identify attributes that are important to users in formation to identify substitutes and complements is the text order to surface recommendations. While different in terms of associated with each product (i.e., reviews, descriptions, and the data and problem setting, such works are similar in that they specifications), from which we are able to uncover the key fea- uncover relationships from large sources of weakly-structured tures and relationships between products, and also to explain data. these relationships through textual signals. We envision several applications of the product graphs produced by our system. Our system can help users to navigate, 3 The Sceptre Model explore and discover new and previously unknown products. Or, it can be used to identify interesting product combinations, In the following we build Sceptre gradually, but in such a way e.g. we can recommend outfits by matching a shirt with comple- that at each step we are specifying a usable model. We highlight mentary trousers and a jacket. And, our system can be used as a the differences between successive iterations of our model in

2 blue. We do this to emphasize the fact that while Sceptre makes Symbol Description use of several interacting components, each of these compo- d i nents brings an additional modeling aspect into the framework. i document associated with an item (product) T Table 1 describes the notation we use throughout the paper. document corpus K number of topics θi K-dimensional topic distribution for item i 3.1 High-level Overview φk word distribution for topic k w jth d We first present a high-level, conceptual view of Sceptre, to d,j word of document z jth d explain the intuition behind the model before we fully specify d,j topic of the word document N d it. d number of words in document F (x) logistic (sigmoid) function, 1/(1 + e−x) Topic Models. We use topic models [4] to discover topics from Eg observed edges in graph g product reviews and other sources of text. Conceptually, this ψ(i, j) pairwise (undirected) features for items i and j means that the text from millions of products can be clustered ϕ(i, j) pairwise (directed) features for items i and j into a small number of dimensions, so that each product i (and β logistic weights associated with ψ(i, j) its text) can be represented by a topic vector θi encoding the η logistic weights associated with ϕ(i, j) extent to which reviews/descriptions of a given product discuss each of the topics. Table 1: Notation. Link Prediction. Topic models allow us to represent each product i by a vector θi. On top of this we can have a statis- tical model to predict properties about products. In our case, captures the notion of substitutes. Thus, for every product, we we use logistic regression to make predictions about pairs of predict a list of complementary and substitutable products and items, using features that combine the topics of two products θi collect them into a giant network of related products. and θj simultaneously. The classification task we are interested in is: does a relationship exist between i and j? Using pairwise e.g. ψ i, j θ −θ features of the products, ( ) = j i, we build logistic 3.2 Detailed Technical Description classifiers such that hβ, ψ(i, j)i takes a positive value if i and j are connected by an edge. We further develop this model so that 3.2.1 Background: Latent Dirichlet Allocation predicting the presence of an edge and the direction of an edge are treated as two separate tasks, to account for asymmetries Latent Dirichlet Allocation (LDA, [4]) uncovers latent structure and to help with interpretability. in document corpora. For the moment, ‘documents’ shall be the Importantly, it should be noted that we do not train topic set of reviews associated with a particular product. LDA asso- then models and perform link prediction, but rather we define a ciates each document in a corpus d ∈ T with a K-dimensional joint objective such that we discover topics that are informative P topic distribution θd (a stochastic vector, i.e., k θd,k = 1), for our link prediction task. In this way our model uncovers which encodes the fraction of words in d that discuss each of topics that are good at ‘explaining’ the relationships between the K topics. That is, words in the document d discuss topic k products. with probability θd,k.

Micro-Categories. An additional goal of Sceptre is to be able Each topic k also has an associated word distribution, φk, to discover micro-categories of closely related products. We which encodes the probability that a particular word is used for achieve this by using sparse representations of very high di- that topic. Finally, the topic distributions themselves (θd) are mensional topic vectors for each product. We make use of ex- assumed to be drawn from a Dirichlet prior. plicit product hierarchies (such as the category tree available The ﬁnal model includes word distributions for each topic from Amazon), where each node of the hierarchy has a small φk, topic distributions for each document θd, and topic assign- number of topics associated with it. The hierarchical nature of ments for each word zd,j. Parameters Φ = {θ, φ} and topic the category tree means that topics associated with top-level assignments z are traditionally updated via sampling [4]. The nodes are general and broad, while topics associated with leaf likelihood of a particular text corpus T (given the word distri- categories focus on differentiating between subtle product fea- bution φ, topics θ, and topic assignments for each word z) is tures, which can be interpreted as micro-categories (e.g. differ- then ent styles of running shoes). N Y Yd Product graph. Finally, we have a supervised learning frame- p(T |θ, φ, z) = θzd,j · φzd,j ,wd,j , (1) work to predict relationships between products. Discovering d∈T j=1 substitutes and complements then depends on the choices of graph we use to train the model, for which we collect several where we are multiplying over all documents in the corpus, graphs of related products from Amazon. For example, a co- and all words in each document. The two terms in the product purchasing graph such as ‘users frequently purchased a and are the likelihood of seeing these particular topics (θzd,j ), and b together’ encodes some notion of complements, whereas a the likelihood of seeing these particular words for this topic graph such as ‘users who viewed a eventually purchased b’ (φzd,j ,wd,j ).

3 3.2.2 Link Prediction with Topic Models This idea of jointly training topic and regression models is closely related to the model of [22], where topics were dis- ‘Supervised Topic Models’ [3] allow topics to be discovered covered that are useful as parameters in a latent-factor recom- that are predictive of an output variable associated with each mender system. Roughly, in the model of [22], a user would document. We propose a variant of a supervised topic model give a high rating to a product if their latent user parameters that identifies topics that are useful as features for link predic- were similar to the topics discovered from reviews of that item; tion. We choose an approach based on logistic regression be- topics were then identified that were good at predicting users’ cause (1) It can be scaled to millions of documents/products ratings of items. The basic model of (eq. 4) is similar in the by hundreds of millions of edges, and (2) It can be adapted to sense that we are coupling parameters θ and β in a joint likeli- incorporate both latent features (topics) and manifest features hood in order to predict the output variable y. (such as brand, price, and rating information), as well as arbi- trary transformations and combinations of these features. Our Directed vs. Undirected Graphs. So far we have shown how goal here is to predict links, that is labels at the level of pairs of to train topic models to predict links between products. How- products. In particular, we want to train logistic classifiers that ever, the feature vector of (eq. 3) is symmetric (ψθ(i, j) = for each pair of products (i, j) predicts whether they are related ψθ(j, i)), meaning that it is only useful for predicting undirected (yi,j = 1) or not (yi,j = 0). For now we will consider the case relationships. However, none of the relationships we where we are predicting just a single type of relationship and want to predict are necessarily symmetric. For example y may we will later generalize the model to predict multiple types of be a good substitute for x if y is a similar product that is cheaper relationships (substitutes and complements) simultaneously. and better rated, but in this case x would be a poor substitute for We want the topics associated with each product to be ‘use- y. Or, while a replacement battery may be a good complement ful’ for logistic regression in the sense that we are able to learn for a laptop, recommending a laptop to a user already purchasing a battery makes little sense. Thus we ought to account for a logistic regressor parametrized by β that predicts yi,j, using such asymmetries in our model. the topics θi and θj as features. That is, we want the logistic function We model such asymmetries by first predicting whether two products are related, and then predicting in which direction the F (θ ) = σ(hβ, ψ (i, j)i) (2) β d θ relation flows. That is, we predict to match y as closely as possible, where ψ (i, j) is a pair- i,j θ p(a has an edge toward b) = wise feature vector describing the two products. We then aim to design features that encode the similarity between the two p(a is related to b)×p(edge flows from a to b | a is related to b), products (documents). The specific choice we adopt is which we denote ‘does the edge flow in this direction?’ ψθ(i, j) = (1, θi,1 · θj,1, θi,2 · θj,2, . . . , θi,K · θj,K ). (3) z }| { p((a, b) ∈ E) = p(a ↔ b) p(a → b|a ↔ b), (5) | {z } Intuitively, by defining our features to be the elementwise prod- ‘are they related?’ uct between θi and θj, we are saying that products with similar a, b ∈ E ordered topic vectors are likely to be linked. The logistic vector β then where relations ( ) are now pairs (that may ex- determines which topic memberships should should be similar ist in both directions). We model relations in this way since we (or dissimilar) in order for the products to be related. expect different types of language or features to be useful for the two tasks—relatedness is a function of what two products Our goal then is to simultaneously optimize both topic dis- have in common, whereas the direction the link flows is a func- tributions θd and logistic parameters β to maximize the joint tion of how the products differ. Indeed, in practice we find that likelihood of topic memberships and relationships in the prod- the second predictor p(a → b|a ↔ b) tends to pick up qual- uct graph: itative language that explains why one product is ‘better than’ another, while the first tends to focus on high-level category corpus likelihood z }| { specific topics. Our objective now becomes N Y Yd L(y, T |β, θ, φ, z) = θzd,j φzd,j ,wd,j L(y, T |β, η, θ, φ, z) = d∈T j=1 positive relations (F ↔) and their direction of flow (F →) Y Y z }| { Fβ(ψθ(i, j)) (1 − Fβ(ψθ(i, j))) . (4) Y ↔ → → Fβ (ψθ(i, j))Fη (ϕθ(i, j))(1 − Fη (ϕθ(j, i))) (i,j)∈E ¯ (i,j)∈E (i,j)∈E | {z } logistic likelihood of the observed graph Nd Y ↔ Y Y (1 − Fβ (ψθ(i, j))) θzd,j φzd,j ,wd,j . (6) This expression says that the review corpus should have high (i,j)∈E¯ d∈T j=1 likelihood according to a topic model, but also that those top- | {z } | {z } ics should be useful as predictors in a logistic regressor that non-relations corpus likelihood uses their similarity as features. In this way, we will intuitively Here F ↔ is the same as in the previous section, though we → discover topics that correspond to some concept of document have added Fη (ϕθ(i, j)) to predict edge directedness; this in- ‘relatedness’. cludes an additional logistic parameter vector η, as well as an

4 ϕ i, j Cables additional feature vector θ( ). The specific feature vector Audio & Video we use is Headphones Batteries & Accessories ϕθ(i, j) = (1, θj,1 − θi,1, . . . , θj,K − θi,K ), (7) Chargers Camera Bags & Cases i.e. the direction in which an edge flows between two items is a Digital Cameras Point & function of the difference between their topic vectors. Shoot Electronics Car & Vehicle Car Audio Incorporating Other Types of Features. We can easily incor- Cables & Accessories porate manifest features into our logistic regressors, which sim- Components ply become additional dimensions in ϕθ(i, j). We include the difference in price, difference in average (star-) rating, and an Data Storage Bages & Cases indicator that takes the value 1 if the items were manufactured Laptops Batteries by different companies, allowing the model to capture the fact Computers Accessories that users may navigate towards (or away from) cheaper prod- Chargers & Laptops Adapters ucts, better rated products, or products from a certain brand. Our entire model ultimately captures the following simple Networking intuition: (1) Users navigate between related products, which Tablet Cases & Sleeves should have similar topics (“what do a and b have in com- Accessories mon?”), and (2) The direction in which users navigate should Figure 2: Part of the product hierarchy for Amazon Electronics be related to the difference between topics (“what does b have products (the complete tree, even for Electronics alone, is too that a doesn’t?”). Ultimately, all of the above machinery has large to include here). been designed to discover topics and predictors that capture this intuition. Learning Multiple Graphs. Next we must generalize our ap- more robust to overfitting compared to training two models sep- proach to simultaneously learn multiple types of relationships. arately. Moreover it means that both logistic regressors operate In our case we wish to discover a graph of products that users in the same feature space; this means that by carefully con- might purchase instead (substitute products), as well as a graph structing our labeled training set (to be described in the follow- of products users might purchase in addition (complementary ing section), we can train the model not only to predict substi- products). Naturally, one could train models independently for tute and complementary relationships, but also to differentiate each type of relationship. But then one would have two sets of between the two, by training it to identify substitutes as non- topics, and two predictors that could be used to predict links in complements and vice versa. each graph. Instead we decide to extend the model from the previous sec- 3.2.3 Sparse Topics via Product Hierarchies tion so that it can predict multiple types of relations simultaneously. We do this by discovering a single set of topics that work Our goal is to learn topic models on corpora with millions well with multiple logistic predictors. This is a small change of products and hundreds of topics. However, training mod- from the previous model of (eq. 6): els with hundreds of topics per product is not practical, nor is it realistic from a modeling perspective. Rather, each product corpus likelihood should draw from a small number of topics, which can be en- z }| { Nd coded using a sparse representation [13]. To achieve this we Y Y enforce sparsity through a specific type of hierarchical topic L(y, T |β, η, θ, φ, z) = θzd,j φzd,j ,wd,j d∈T j=1 representation that makes use of an explicit category tree, such Amazon Y Y as the one available from . F ↔ ψ i, j F → ϕ i, j − F → ϕ i, j βg ( θ( )) ηg ( θ( ))(1 ηg ( θ( ))) An example of the product hierarchy we obtain from Amazon g∈G (i,j)∈Eg is shown in Figure 2. The category of each product is a node in Y ↔ this tree (though not necessarily a leaf node); some products (1 − F (ψθ(i, j))) . βg may also belong to multiple categories simultaneously. ¯ (i,j)∈Eg We build our topic representation using the following | {z } accuracy of the predictors βg and ηg for the graph g scheme: First, each product is represented by a path, or more (8) simply a set of nodes, in the category tree. For products belonging to multiple categories, we take the union of those paths. where each graph g ∈ G contains all relations of a particular Second, each topic is associated with a particular node in the type. category tree. Every time we observe, say, a thousand instances Note that we learn separate predictors βg and ηg for each of a node, we associate an additional topic with that node, up graph g, but we learn a single set of topics (θ) and features (ψ to some maximum. In this way we have many topics associated and ϕ) that work well for all graphs simultaneously. We adopt with popular categories (like ‘Electronics’) and fewer topics as- this approach because it provides a larger training set that is sociated with more obscure categories. Third, we use a sparse

5 Figure 3: A demonstration of our topic hierarchy. A product (left) is shown with its associated topics (right). (a) the category tree (b) the topic vector (c) the product’s ground-truth category. The product’s position in the category tree is highlighted in red, and the set of topics that are ‘activated’ is highlighted in gray. representation for each product’s topic vector. Specifically, if a assign each word to a topic (an integer between 1 and K) ran- product occupies a particular path in the category tree, then it domly, with probability proportional to the likelihood of that can only draw words from topics associated with nodes in that topic occurring with that word. The expression θd,kφk,wd,j is path. In this way, even though our model may have hundreds the probability of the topic k for the product d (θd,k), multi- of topics, only around 10-20 of these will be ‘active’ for any plied by the probability of the word at position j (wd,j) being particular product. This is not only necessary from a scalabil- used in topic k (φk,wd,j ). ity standpoint, but it also helps the model quickly converge to meaningful topic representations. This process is depicted in Figure 3. Here a product like 4 Experiments a laptop charger draws from ‘generic’ topics that apply to all electronics products, as well as topics that are specific to lap- Next we evaluate Sceptre. We first describe the data as well as tops or chargers; but it cannot draw from topics that are spe- the baselines and then proceed with experimental results. cific to mobile phones or laptop cases (for example). Thus all products have some high-level categories in common, but are 4.1 Data also assumed to use their own unique sub-category specific language. Then, at the lowest level, each leaf node in the category We use data crawled from Amazon.com, whose characteristics tree is associated with multiple topics; thus we might learn sev- are shown in Table 2. This data was collected by performing eral ‘microcategories’ of laptop chargers, e.g. for different lap- a breadth-first search on the user-product-review graph until top types, price points, or brands. We present some examples termination, meaning that it is a fairly comprehensive collec- of the types of microcategory we discover in Section 4.6. tion of English-language product data. We split the full dataset into top-level categories, e.g. Books, Movies, Music. We do this mainly for practical reasons, as it allows each model and dataset 3.3 Optimization and Training to fit in memory on a single machine (requiring around 64GB Optimizing an objective such as the one in (eq. 8) is a diffi- RAM and 2-3 days to run our largest experiment). Note that cult task, for instance it is certainly non-convex.1 We solve it splitting the data in this way has little impact on performance, using the following EM-like procedure, in which we alternate as there are few links that cross top-level categories, and the between optimizing the model parameters Θ = (β, η, θ, φ) and hierarchical nature of our model means that few parameters are topic assignments (latent variables) z: shared across categories. To obtain ground-truth for pairs of substitutable and com- update Θ(t) = argmax l(y, T |β, η, θ, φ, z(t−1)) (9) plementary products we also crawl graphs of four types from Θ Amazon: z(t) p(z(t) = k) = θ φ(t) , sample d,j with probability d,j d,k k,wd,j (10) 1. ‘Users who viewed x also viewed y’; 91M edges. 2. ‘Users who viewed x eventually bought y’; 8.18M edges. where l(y, T |β, η, θ, φ, z) is the log-likelihood from (eq. 8). To 3. ‘Users who bought x also bought y’; 133M edges. generate initial values for Θ and z we initialize continuous pa- 4. ‘Users frequently bought x and y together’; 4.6M edges. rameters and topics uniformly at random (continuous parame- We refer to edges of type 1 and 2 as substitutes and edges of ters are sampled from [0, 1)). type 3 or 4 as complements, though we focus on ‘also viewed’ In the first step (eq. 9), topic assignments for each word (z) and ‘also bought’ links in our experiments, since these form the are fixed. We fit the remaining terms, β, η, θ, and φ, by gradi- vast majority of the dataset. Note the minor differences between ent ascent. We use the Hybrid LBFGS solver of [19], a quasi- certain edge types, e.g. edges of type 4 indicate that two items Newton method for non-linear optimization of problems with were purchased as part of a single basket, rather than across many variables [25]. Computing the partial derivatives them- sessions. selves, while computationally expensive, is na¨ıvely paralleliz- able over edges in E and documents (i.e., products) in T . The second step iterates through all products d and all word 4.2 Experimental Setting positions j and updates topic assignments. As with LDA, we We split our training data (E and E¯) into 80% training, 10% 1It is smooth, and multiple local minima can be found by permuting the validation, 10% test, discarding products with fewer than 20 order of topics and logistic parameters. reviews. In all cases we report the error on the test set. The

6 Category Users Items Reviews Edges defined over words rather than topics. That is, topics θi for each product are replaced by TF-IDF scores for words in its reviews Men’s Clothing 1.25M 371K 8.20M 8.22M [20]. Logistic parameters β and η are then trained to determine Women’s Clothing 1.82M 838K 14.5M 17.5M which tf-idf-weighted words are good at predicting the pres- Music 1.13M 557K 6.40M 7.98M ence or absence of edges. This baseline was uniformly weaker Movies 2.11M 208K 6.17M 4.96M than our other baselines, so we shall not discuss its performance Electronics 4.25M 498K 11.4M 7.87M further. Books 8.20M 2.37M 25.9M 50.0M Category-Tree (CT). Since we make use of Amazon’s category All 21.0M 9.35M 144M 237M tree when building our model, it is worth questioning the extent to which the performance of our model owes to our decision to Table 2: Dataset statistics for a selection of categories on Ama- use this source of data. For this baseline, we compute the co- zon. counts between categories c1 → c2 that are observed in our training data. Then we predict that an edge exists if it is among the 50th percentile of most commonly co-occurring categories. iterative fitting process described in (eqs. 9 and 10) continues In other words this baseline ‘lifts’ links to the level of categories until no further improvement is gained on the validation set. rather than individual products.2 Sampling Non-Edges. Since it is impractical to train on all Item-to-Item Collaborative Filtering (CF). In 2003 Ama- pairs of non-links, we start by building a balanced dataset by zon reported that their own recommendation solution was a sampling as many non-links as there are links (i.e., |E| = |E|¯ ). collaborative-filtering approach, that identified items that had However, we must be careful about how non-links (i.e., neg- been browsed or purchased by similar sets of users [17]. This ative examples) are sampled. Sampling random pairs of unre- baseline follows the same procedure, though in lieu of actual lated products makes negative examples very ‘easy’ to classify; browsing or purchasing data we consider sets of users who have rather, since we want the model to be able to differentiate be- reviewed each item. We then proceed by computing for each tween edge types, we treat substitute links as negative exam- pair of products a and b the cosine similarity between the set ples of complementary edges and vice versa. Thus, we explic- of users who reviewed a and the set of users who reviewed b. itly train the model to identify substitutes as non-complements Sorting by similarity generates a ranked list of recommenda- and vice versa (in addition to a random sample of non-edges). tions for each product. Since this method is not probabilistic This does not make prediction ‘easier’, but it helps the model we only report its performance at ranking tasks. to learn a better separation between the two edge types, by explicitly training it to learn distinct notions of the two concepts. In the following, we consider both link prediction and rank- 4.4 Link Prediction and Ranking ing tasks: (1) to estimate for a pair of products whether they are related, and (2) for a given query, rank those items that are most Link Prediction. Our first goal is to predict for a given pair of likely to be related. We first describe the baselines we compare products (a, b), and a graph type g, whether there is a link from against. a to b in Eg. We optimize exactly the objective in (eq. 8). Note that a prediction is correct when 4.3 Baselines • for each test edge (in each graph): a → b, ↔ → Fθ (ψ(a, b), β) > 0 and Fθ (ϕ(a, b), η) > 0 • non a 6→ b Random. Link probabilities Fβ and Fη are replaced with ran- for each -edge , ↔ → dom numbers between 0 and 1. Note that since both predictors Fθ (ψ(a, b), β) ≤ 0 or Fθ (ϕ(a, b), η) ≤ 0, have to ‘fire’ to predict a relation, random classification identi- in other words the model must correctly predict both that the fies 75% of directed edges as non-links; imbalance in the num- link exists and its direction. ber of positive vs. negative relations of each type (due to our Results are shown in Table 3 for each of the datasets in Ta- sampling procedure described above) explains why the perfor- ble 2. We also show results from ‘Baby’ clothes, to demon- mance of random classification is slightly different across ex- strate that performance does not degrade significantly on a (rel- periments. atively) smaller dataset (43k products). ‘Substitute’ links were unavailable for the vast majority of products from Music and LDA + logistic regression (LDA). Rather than training topic Movies in our crawl, so results are not shown. We summarize models and logistic parameters simultaneously, this baseline the main findings from this table as follows: first trains a topic model and then trains logistic classifiers on 1. Sceptre is able to accurately predict both substitute and the pre-trained topics. This baseline assesses our claim that complement links across all product categories, with per- Sceptre learns topics that are ‘good as’ features for edge predic- formance being especially accurate for clothing and elec- tion, by comparing it to a model whose topics were not trained tronics products. Accuracy is between 85.57-96.76% for specifically for this purpose. We used Vowpal Wabbit to pre- the binary prediction tasks we considered. train the topic model, and fit models with K = 100 topics for each Amazon category. 2We experimented with several variations on this theme, and this approach We also experimented with a baseline in which features were yielded the best performance.

7 Women’s (Substitutes) Women’s (Complements) 2. Prediction of ‘substitute’ links is uniformly more accurate 100 100 than ‘complement’ links for all methods, both in absolute Sceptre Sceptre Collab. Filtering Collab. Filtering 1 1 (left two columns) and relative (right two columns) terms. 10− random 10− random

This matches our intuition that substitute links should be 2 2 10− 10− ‘easier’ to predict, as they essentially correspond to some 3 3 notion of similarity, whereas the semantics of comple- 10− 10− precision @ k precision @ k 4 4 ments are more subtle. 10− 10− 3. The performance of the baselines is variable. For substi- 5 5 10− 10− tute links, our LDA baseline obtains reasonable perfor- 0 200 400 600 800 1000 0 200 400 600 800 1000 mance on Books and Electronics, whereas the Category k k Tree (CT) baseline is better for Clothing. In fact, the CT Men’s (Substitutes) Men’s (Complements) 100 100 baseline performs surprisingly well at predicting substi- Sceptre Sceptre Collab. Filtering Collab. Filtering 1 1 tute links, for the simple reason that substitutable products 10− random 10− random often belong to the same category as each other. 2 2 4. None of the baselines yield satisfactory performance when 10− 10− 3 3 predicting complement links. Thus we conclude that nei- 10− 10−

ther the topics uncovered by a regular topic model, nor precision @ k precision @ k 4 4 the category tree alone are capable of capturing the subtle 10− 10− 5 5 notions of what makes items complementary. 10− 10− 0 200 400 600 800 1000 0 200 400 600 800 1000 Ultimately we conclude that each of the parts of Sceptre con- k k tribute to its accurate performance. Category information is Figure 4: Precision@k for Women’s and Men’s clothing. helpful, but alone is not useful to predict complements; and si- multaneous training of topic models and link prediction is necessary to learn useful topic representations. TF-IDF baselines were less accurate than collaborative filtering Ranking. In many applications distinguishing links from non- and are not shown).3 links is not enough as for each product we would like to recom- Examples of recommendations generated by Sceptre are mend a limited number of substitutes and complements. Thus, shown in Figure 6. it is important that relevant items (i.e., relevant relations) are ranked higher than irrelevant ones, regardless of the likelihood that the model assigns to each recommendation. A standard measure to evaluate ranking methods is the preci- 4.5 Cold-Start Prediction without Reviews sion@k. Given a set of recommended relations of a given type rec, and a set of known-relevant products rel (i.e., ground-truth Although it is promising that we are able to recommend sub- links) the precision is defined as stitutes and complements from the text of product reviews, an obvious question remains as to what can be done for new precision = |rel ∩ rec|/|rec|, (11) products, that do not yet have any reviews associated with them, known as the ‘cold-start’ problem in recommender sys- i.e., the fraction of recommended relations that were relevant. tems [16, 24, 27, 30, 36]. To address this problem, we note that The precision@k is then the precision obtained given a fixed Sceptre merely requires that we have some source of text asso- budget, i.e., when |rec| = k. This is relevant when only a small ciated with each linked item in order to learn a model of which number of recommendations can be surfaced to the user, where products are likely to be related. it is important that relevant products appear amongst the first few suggestions. To evaluate the possibility of using sources of text other Figure 4 reports the precision@k on Men’s and Women’s than reviews, we collected descriptive text about each item in clothing. Note that we naturally discard candidate links that ap- our Amazon Books catalog, including blurbs, ‘about the au- peared during training. This leaves only a small number of rel- thor’ snippets, and editorial reviews. We also collected man- evant products for each query item in the corpus—the random ufacturer’s descriptions for a subset of our Electronics data. baseline (which up to noise should be flat) has precision around Training on these sources of data, Sceptre obtained accuracies 5×10−5, indicating that only around 5 in 100,000 products are between 91.28% and 93.67% at predicting substitutes and com- labeled as ‘relevant’ in this experiment. This, in addition to the plements (see Table 4). This result implies that training is possi- fact that many relevant items may not be labeled as such (there ble on diverse sources of text beyond just product reviews, and are presumably thousands of pairs of substitutable pants in our that Sceptre can be applied in cold-start settings, even when no 4 corpus, but only 30 or so are recommended for each product) reviews are available. highlights the incredible difficulty of obtaining high precision scores for this task. 3Note that collaborative filtering is done here at the level of reviewed prod- Overall, collaborative filtering is one to two orders-of- ucts, which is naturally much sparser than the purchase and browsing data used to produce the ground-truth. magnitude more accurate than random rankings, while Scep- 4Note that this is not directly comparable to review-based results since dif- tre is an order of magnitude more accurate again (our LDA and ferent subsets of our corpus have reviews vs. descriptions.

8 Electronics e111 e92 e75 e79 e78 e50 e69 e85 e96 e89 e99 high-end budget Samsung heavy-duty cameras portable speakers cases styli batteries portable radios car radios head- head- cases cases phones phones camera little speaker leather Galaxy Otterbox pen batteries radio radio bass bass zoom bose case elastic Defender tip battery weather Pioneer Sennheiser Skullcandy pictures portable speaker soft magnets protection Bamboo charged crank factory Bose sound Kodak small speaker Roocase Samsung bulky Wacom rechargeable solar Metra Shure bud Canon sound velcro leather kids styli oem Eton Ford Beats outside noise flash iHome closed closed shell gloves Sanyo Baofeng dash Koss another pair digital bass material auto Survivor Friendly Swede Lenmar radio reception Honda Akg comfortable optical wireless speaker snug closing protected pencil alkaline miles Jeep music gym taken great speaker protection elastic strap safe capacitive Energizer fm wiring classical Beats picture mini speaker standing cover protective precise full charge alert deck Klipsch head Men’s clothing c44 c107 c75 c49 c52 c110 c156 c134 c133 c24 c9 three-wolf minimalist athletic sports generic generic dress shirts dress shoes dress pants polo shirts boots shirt running running shoes clothing clothing sleeves leather expandable wolf Polo Bates running Balance court dry same arms sole expandable waist moon Lauren Red Wing trail New play cold durable neck dress Dockers three Ralph leather barefoot wide Nike working store shoulders brown iron power Beene good boot Altra running running shoe short different dress shirt dress shoe khaki trailer nice shirt casual boot running shoe series running hot two dress polish stretch waist hair Geoffrey dress boot minimalist feet games weather brand jacket brown pair hidden man great shirt right boot zero drop usa light shoe tight comfort long sleeve toe ironed short-sleeve quality shirt motorcycle road cross training great shoe cool fine iron looking shoe dress pant magic white shirt Wings glove athletic shoe support down tight tucked formal elastic waist powerful fitted shirt Rangers run cross miles regular another pair

Table 5: A selection of topics from Electronics and Men’s Clothing along with our labels for each topic. Top 10 words/bigrams from each topic are shown after subtracting the background distribution. Capital letters denote brand names (Bamboo, Wacom, Red Wing, etc.).

4.6 Topic Analysis ics appears only in the background distribution. The topics we obtain are closely aligned with categories from Next we analyze the types of topics discovered by Sceptre. As Amazon (e.g. electronics topic e111, or clothing topic c110), we recall from Section 3.2.3, each topic is associated with a though this is to be expected since our topic model is built on node in Amazon’s category tree. But, just as a high-level cat- top of an explicit hierarchy as in Figure 3. However, we note egory such as clothing can naturally be separated into finer- that finer-grained ‘microcategories’ are discovered that are not grained categories like socks, shoes, hats, pants, dresses (etc.), explicitly present in the hierarchy, e.g. high-end headphones are we hope that Sceptre will discover even subtler groupings distinguished from cheaper models (e89 and e99), and running of products that are not immediately obvious from the hand- shoes are distinguished based on distinct running styles (c133, labeled category hierarchy. c134, and c156). Table 5 shows some of the topics discovered by Sceptre, on We also note that brand words predominate in several top- two Amazon categories: Electronics and Men’s Clothing. We ics, e.g. high-end headphones can be identified by words like pruned our dictionary by using adjectives, nouns, and adjective- ‘Sennheiser’, ‘AKG’ etc. (e89), and high-end t-shirts can be noun pairs (as determined by WordNet [9]), as well as any identified by words like ‘Ralph Lauren’ and ‘Geoffrey Beene’ words appearing in the ‘brand’ field of our crawled data. For (c52). At the extreme end, a single product may have its own visualization we compute the 10 highest-weight words from all topic, e.g. the popular ‘three-wolf moon’ shirt (c49), whose re- topics, after first subtracting a ‘background’ topic containing views have already inspired academic discussion [29]. Here the the average weight across all topics. That is for each topic k we product’s high popularity and unique word distribution means report the 10 highest values of that dedicating it an entire topic substantially increases the cor- 1 X pus likelihood in (eq. 8). Note that we only show a fraction of φ − φ 0 . (12) k K k k0 the full set of topics discovered by our model; other common | {z } camera brands (etc.) are covered among the large range of top- background word distribution ics not shown here. By doing so, stopwords and other language common to all top- Finally, while some topics are highly specific, like those

9 substitutes complements those which have the highest likelihood under the model. That 1 is, for each graph type g we would like to recommend amazon X Sceptre rec i F ↔ ψ i, j F → ϕ i, j , g( ) = argmax βg ( θ( )) ηg ( θ( )) (13) R S∈(T \{i}) j∈S 0 top-1 top-5 top-1 top-5 frac. of times selected latex sucks R Figure 5: Results of our user study. Users were asked to se- where S ∈ (T\{i}) is a set of R products other than i itself. lect which recommendations (ours or Amazon’s) were prefer- While computing the score for a single candidate edge is able substitutes/complements (users could also select neither or very fast (O(K) operations), on a dataset with millions of prod- both). ucts this already results in an unacceptable delay when ranking all possible recommendations. Similar to [17] we implemented two modifications that make this enumeration procedure feasi- referring to individual products or brands, others are more ble (on the order of a few milliseconds). The first is to ignore generic, such as clothing topics c9 and c24. Such topics tend obscure products by limiting the search space by some popu- to appear toward the top of the category hierarchy (see Fig. 3), larity threshold; we consider the hundred-thousand most pop- for example the topic c9 is associated with the ‘Clothing’ ular products per-category when generating new recommenda- node, whereas c24 is associated with its child, ‘Clothing: Men’, tions. The second is to cull the search space using the category of which all other topics in Table 5 are descendants. Intu- tree explicitly; e.g. when browsing for running shoes we can itively, these are much like ‘background’ distributions, contain- ignore, say, camera equipment and limit our search to clothing ing words that are relevant to the majority of clothing products, and shoes. Specifically, we only consider items belonging to like durability, fit, warmth, color, packaging, etc. the same category, its parent category, its child categories, and its sibling categories (in other words, its ‘immediate family’). It is very rare that the top-ranked recommendations belong to 4.7 User Study distant categories, so this has minimal impact on performance. Finally we perform a user study to evaluate the quality of the Another issue is that of adding new products to the system. recommendations produced by Sceptre. Naturally we would not Naturally, it is not feasible to re-train the system every time a expect that a fully-supervised algorithm would produce predic- new product is added. However, this is thankfully not neces- tions that were somehow ‘better’ than the ground-truth used to sary, as the introduction of a small number of products will not train it. However, we hope Sceptre may correct for some noise fundamentally change the word distribution φ. Thus it is sim- in the ground-truth, since while users may often buy multiple ply a matter of estimating the product’s topic distribution under pairs of jeans together (for example) we are explicitly training the existing model, as can be done using LDA [4]. the system to identify complementary items that would not be When assembling our user interface (see Figs. 1 and 6) we substitutable. use the discovered topics from Section 4.6 to ‘explain’ recom- We used Mechanical Turk to compare Sceptre’s recommen- mendations to users, by selecting sentences whose language dations to Amazon’s ‘also viewed’ and ‘also bought’ sugges- best explains why the recommended product was predicted. tions, for a random sample of 200 Clothing items. Human Specifically, we highlight sentences whose words yield the largest response to F →. judges identified which recommendations they felt were ac- ηg ceptable substitutes and complements (surfaced in a random Reproducing Sceptre. All data and code used in this pa- order without labels; a screenshot is shown in Fig. 6d). Judges per, as well as the interface from Figure 6 is available on evaluated the top recommendation, and top-5 recommendations the first author’s webpage: http://cseweb.ucsd.edu/ separately, yielding results shown in Figure 5. ˜jmcauley/. We see here that Amazon’s ‘also viewed’ links generate preferable substitutes, indicating that large volumes of browsing data yield acceptable substitute relationships with relatively 6 Conclusion little noise. On the other hand, Sceptre’s complement links are overwhelmingly preferred, suggesting that our decision to A useful recommender system must produce recommendations model complements as non-substitutes qualitatively improves that not only match our preferences, but which are also rel- performance. evant to our current topic of interest. For a user browsing a particular product, two useful notions of relevant recommendations include substitutes and complements: products that can 5 Building the product graph be purchased instead of each other, and products that can be purchased in addition to each other. In this paper, our goal has Having produced ranked lists of recommended relationships, been to learn these concepts from product features, especially our final task is to surface these recommendations to a potential from the text of their reviews. user of our system in a meaningful way. We have presented Sceptre, a model for predicting and un- While conceptually simple, comparing all products against derstanding relationships between linked products. We have ap- all others quickly becomes impractical in a corpus with mil- plied this to the problem of identifying substitutable and com- lions of products. Our goal here is to rank all links, and surface plementary products on a large collection of Amazon data, in-

10 (a) Men’s clothing (b) Women’s clothing (c) Electronics (d) mturk interface Figure 6: (a,b,c) Examples of recommendations produced by Sceptre; the top of each subfigure shows the query product, the left column shows substitutes recommended by Sceptre, and the right column shows complements. (d) Interface used to evaluate Sceptre on Mechanical Turk; Turkers are shown lists of items suggested by Amazon (i.e., the ground-truth) and Sceptre and must identify which lists they prefer. cluding 144 million reviews and 237 million ground-truth rela- L. Garcia-Pueyo. Supercharging recommender systems using tionships based on browsing and co-purchasing logs. taxonomies for learning user purchase behavior. VLDB, 2012. [14] Y. Koren and R. Bell. Advances in collaborative filtering. In Recommender Systems Handbook. Springer, 2011. References [15] Y. Koren, R. Bell, and C. Volinsky. Matrix factorization tech- niques for recommender systems. Computer, 2009. [1] R. Balasubramanyan and W. Cohen. Block-LDA: Jointly model- [16] A. Levi, O. Mokryn, C. Diot, and N. Taft. Finding a needle in a ing entity-annotated text and entity-entity links. In SDM, 2011. haystack of reviews: cold start context-based hotel recommender [2] J. Bennett and S. Lanning. The Netflix prize. In KDD Cup and system. In RecSys, 2012. Workshop, 2007. [17] G. Linden, B. Smith, and J. York. Amazon.com recommenda- [3] D. Blei and J. McAuliffe. Supervised topic models. In NIPS, tions: Item-to-item collaborative filtering. IEEE Internet Com- 2007. puting, 2003. [4] D. Blei, A. Ng, and M. Jordan. Latent Dirichlet allocation. [18] Y. Liu, A. Niculescu-Mizil, and W. Gryc. Topic-link LDA: joint JMLR, 2003. models of topic and author community. In ICML, 2009. [5] D. M. Blei, T. Griffiths, M. Jordan, and J. Tenenbaum. Hierar- [19] Y.Liu, W. Wang, B. Levy,´ F. Sun, D.-M. Yan, L. Lu, and C. Yang. chical topic models and the nested Chinese restaurant process. On centroidal Voronoi tessellation – energy smoothness and fast In NIPS, 2003. computation. ACM Trans. on Graphics, 2009. [6] S. Brody and N. Elhadad. An unsupervised aspect-sentiment [20] C. D. Manning, P. Raghavan, and H. Schutze.¨ Introduction to model for online reviews. In ACL, 2010. Information Retrieval. Cambridge University Press, 2008. [7] J. Chang and D. Blei. Relational topic models for document [21] A. Mas-Colell, M. Whinston, and J. Green. Microeconomic The- networks. In AIStats, 2009. ory. Oxford University Press, 1995. [8] J. Chang, J. Boyd-Graber, and D. Blei. Connections between the [22] J. McAuley and J. Leskovec. Hidden factors and hidden topics: lines: augmenting social networks with text. In KDD, 2009. understanding rating dimensions with review text. In RecSys, [9] C. Fellbaum. WordNet: An Electronic Lexical Database. Brad- 2013. ford Books, 1998. [23] S. Moghaddam and M. Ester. On the design of LDA models for [10] M. Gamon, A. Aue, S. Corston-Oliver, and E. Ringger. Pulse: aspect-based opinion mining. In CIKM, 2012. Mining customer opinions from free text. In IDA, 2005. [24] S. Moghaddam and M. Ester. The FLDA model for aspect-based [11] G. Ganu, N. Elhadad, and A. Marian. Beyond the stars: Im- opinion mining: Addressing the cold start problem. In WWW, proving rating predictions using review text content. In WebDB, 2013. 2009. [25] J. Nocedal. Updating quasi-newton matrices with limited stor- [12] S. Jagabathula, N. Mishra, and S. Gollapudi. Shopping for prod- age. Mathematics of Computation, 1980. ucts you don’t know you need. In WSDM, 2011. [26] D. Panigrahi and S. Gollapudi. Result enrichment in commerce [13] B. Kanagal, A. Ahmed, S. Pandey, V. Josifovski, J. Yuan, and search using browse trails. In WSDM, 2011.

11 [27] S.-T. Park and W. Chu. Pairwise preference regression for cold- start recommendation. In RecSys, 2009. Error reduction [28] A. Popescu and O. Etzioni. Extracting product features and opin- Accuracy ions from reviews. In HLT, 2005. vs. random Category Method Subst. Compl. Subst. Compl. [29] A. Reyes and P. Rosso. Mining subjective knowledge from customer reviews: A speciﬁc case of irony detection. In HLT, 2011. Random 60.27% 57.70% 0.0% 0.0% [30] A. Schein, A. Popescul, L. Ungar, and D. Pennock. Methods and Men’s LDA 70.62% 65.95% 26.05% 19.50% metrics for cold-start recommendations. In SIGIR, 2002. Clothing CT 78.69% 61.06% 46.38% 7.946% [31] I. Titov and R. McDonald. A joint model of text and aspect Sceptre 96.69% 94.06% 91.67% 85.97% ratings for sentiment summarization. In ACL, 2008. Random 60.35% 56.67% 0.0% 0.0% [32] I. Titov and R. McDonald. Modeling online reviews with multi- Women’s LDA 70.70% 64.80% 26.11% 18.75% grain topic models. In WWW, 2008. Clothing CT 81.05% 69.08% 52.21% 28.63% [33] D. Vu, A. Asuncion, D. Hunter, and P. Smyth. Dynamic egocen- Sceptre 95.87% 94.14% 89.59% 86.47% tric models for citation networks. In ICML, 2011. [34] C. Wang and D. Blei. Collaborative topic modeling for recom- Random - 50.18% - 0.0% LDA - 52.39% - 4.428% mending scientiﬁc articles. In KDD, 2011. Music CT - 57.02% - 13.71% [35] J. Zheng, X. Wu, J. Niu, and A. Bolivar. Substitutes or comple- Sceptre ments: another step forward in recommendations. In EC, 2009. - 90.43% - 80.78% [36] K. Zhou, S.-H. Yang, and H. Zha. Functional matrix factoriza- Random - 51.22% - 0.0% tions for cold-start recommendation. In SIGIR, 2011. LDA - 54.26% - 6.235% Movies CT - 66.34% - 30.99% Sceptre - 85.57% - 70.42% Random 69.98% 55.67% 0.0% 0.0% LDA 89.90% 61.90% 66.35% 14.06% Electronics CT 87.26% 60.18% 57.57% 10.17% Sceptre 95.70% 88.80% 85.69% 74.74% Random 69.93% 55.35% 0.0% 0.0% LDA 89.91% 60.59% 66.47% 11.75% Books CT 87.80% 66.28% 59.42% 24.49% Sceptre 93.76% 89.86% 79.25% 77.29% random 62.93% 52.47% 0.0% 0.0% Baby LDA 75.86% 54.73% 34.89% 4.75% Clothes CT 79.31% 64.56% 44.18% 25.43% Sceptre 92.18% 93.65% 78.91% 86.65% Average Sceptre 94.83% 90.23% 85.02% 80.33%

Table 3: Link prediction accuracy for substitute and complement links (the former are not available for the majority of Mu- sic/Movies products in our dataset). Absolute performance is shown at left, reduction in error vs. random classiﬁcation at right.

Error reduction Accuracy vs. random Category Subst. Compl. Subst. Compl. Electronics, cold-start 91.28% 93.22% 70.95% 84.71% Books, cold-start 96.76% 93.67% 89.22% 85.81%

Table 4: Link prediction accuracy using cold-start data (manu- facturer’s and editorial descriptions).