Factorization Machines Steffen Rendle Department of Reasoning for Intelligence The Institute of Scientific and Industrial Research Osaka University, Japan [email protected] Abstract—In this paper, we introduce Factorization Machines variable interactions (comparable to a polynomial kernel in (FM) which are a new model class that combines the advantages SVM), but uses a factorized parametrization instead of a of Support Vector Machines (SVM) with factorization models. dense parametrization like in SVMs. We show that the model Like SVMs, FMs are a general predictor working with any real valued feature vector. In contrast to SVMs, FMs model all equation of FMs can be computed in linear time and that it interactions between variables using factorized parameters. Thus depends only on a linear number of parameters. This allows they are able to estimate interactions even in problems with huge direct optimization and storage of model parameters without sparsity (like recommender systems) where SVMs fail. We show the need of storing any training data (e.g. support vectors) for that the model equation of FMs can be calculated in linear time prediction. In contrast to this, non-linear SVMs are usually and thus FMs can be optimized directly. So unlike nonlinear SVMs, a transformation in the dual form is not necessary and optimized in the dual form and computing a prediction (the the model parameters can be estimated directly without the need model equation) depends on parts of the training data (the of any support vector in the solution. We show the relationship support vectors). We also show that FMs subsume many of to SVMs and the advantages of FMs for parameter estimation the most successful approaches for the task of collaborative in sparse settings. filtering including biased MF, SVD++ [2], PITF [3] and FPMC On the other hand there are many different factorization mod- els like matrix factorization, parallel factor analysis or specialized [4]. models like SVD++, PITF or FPMC. The drawback of these In total, the advantages of our proposed FM are: models is that they are not applicable for general prediction tasks 1) FMs allow parameter estimation under very sparse data but work only with special input data. Furthermore their model where SVMs fail. equations and optimization algorithms are derived individually for each task. We show that FMs can mimic these models just 2) FMs have linear complexity, can be optimized in the by specifying the input data (i.e. the feature vectors). This makes primal and do not rely on support vectors like SVMs. FMs easily applicable even for users without expert knowledge We show that FMs scale to large datasets like Netflix in factorization models. with 100 millions of training instances. Index Terms —factorization machine; sparse data; tensor fac- 3) FMs are a general predictor that can work with any real torization; support vector machine valued feature vector. In contrast to this, other state-of- the-art factorization models work only on very restricted I. INTRODUCTION input data. We will show that just by defining the feature Support Vector Machines are one of the most popular vectors of the input data, FMs can mimic state-of-the-art predictors in machine learning and data mining. Nevertheless models like biased MF, SVD++, PITF or FPMC. in settings like collaborative filtering, SVMs play no important role and the best models are either direct applications of II. PREDICTION UNDER SPARSITY standard matrix/ tensor factorization models like PARAFAC The most common prediction task is to estimate a function [1] or specialized models using factorized parameters [2], [3], y : Rn T from a real valued feature vector x Rn to a [4]. In this paper, we show that the only reason why standard target domain→ T (e.g. T = R for regression or T ∈= +, SVM predictors are not successful in these tasks is that they for classification). In supervised settings, it is assumed{ t−}hat cannot learn reliable parameters (‘hyperplanes’) in complex there is a training dataset D = (x(1),y(1)), (x(2),y(2)),... (non-linear) kernel spaces under very sparse data. On the other of examples for the target function{ y given. We also investigate} hand, the drawback of tensor factorization models and even the ranking task where the function y with target T = R can more for specialized factorization models is that (1) they are be used to score feature vectors x and sort them according to not applicable to standard prediction data (e.g. a real valued their score. Scoring functions can be learned with pairwise feature vector in Rn.) and (2) that specialized models are training data [5], where a feature tuple (x(A), x(B)) D usually derived individually for a specific task requiring effort means that x(A) should be ranked higher than x(B). As∈ the in modelling and design of a learning algorithm. pairwise ranking relation is antisymmetric, it is sufficient to In this paper, we introduce a new predictor, the Factor- use only positive training instances. ization Machine (FM), that is a general predictor like SVMs In this paper, we deal with problems where x is highly but is also able to estimate reliable parameters under very sparse, i.e. almost all of the elements xi of a vector x are high sparsity. The factorization machine models all nested zero. Let m(x) be the number of non-zero elements in the has ever rated. For each user, the variables are normalized such that they sum up to 1. E.g. Alice has rated Titanic, Notting Hill and Star Wars. Additionally the example contains a variable (green) holding the time in months starting from January, 2009. And finally the vector contains information of the last movie (brown) the user has rated before (s)he rated the active one – e.g. for x(2), Alice rated Titanic before she rated Notting Hill. In section V, we show how factorization machines using such feature vectors as input data are related to specialized state-of-the-art factorization models. Fig. 1. Example for sparse real valued feature vectors x that are created from the transactions of example 1. Every row represents a feature vector x(i) with We will use this example data throughout the paper for illus- (i) its corresponding target y . The first 4 columns (blue) represent indicator tration. However please note that FMs are general predictors variables for the active user; the next 5 (red) indicator variables for the active item. The next 5 columns (yellow) hold additional implicit indicators (i.e. like SVMs and thus are applicable to any real valued feature other movies the user has rated). One feature (green) represents the time in vectors and are not restricted to recommender systems. months. The last 5 columns (brown) have indicators for the last movie the user has rated before the active one. The rightmost column is the target – III. FACTORIZATION MACHINES (FM) here the rating. In this section, we introduce factorization machines. We discuss the model equation in detail and show shortly how to apply FMs to several prediction tasks. feature vector x and mD be the average number of non-zero elements m(x) of all vectors x D. Huge sparsity (mD A. Factorization Machine Model ∈ ≪ n) appears in many real-world data like feature vectors of 1) Model Equation: The model equation for a factorization event transactions (e.g. purchases in recommender systems) machine of degree d =2 is defined as: or text analysis (e.g. bag of word approach). One reason for n n n huge sparsity is that the underlying problem deals with large yˆ(x) := w + wi xi + vi, vj xi xj (1) 0 categorical variable domains. i=1 i=1 j=i+1 Example 1 Assume we have the transaction data of a movie where the model parameters that have to be estimated are: n n×k review system. The system records which user u U rates w0 R, w R , V R (2) a movie (item) i I at a certain time t R with∈ a rating ∈ ∈ ∈ ∈ ∈ And , is the dot product of two vectors of size k: r 1, 2, 3, 4, 5 . Let the users U and items I be: · · ∈{ } k U = Alice (A), Bob (B), Charlie (C),... vi, vj := vi,f vj,f (3) { } · f I = Titanic (TI), Notting Hill (NH), Star Wars (SW), =1 { v V Star Trek (ST),... A row i within describes the i-th variable with k factors. N+ } k 0 is a hyperparameter that defines the dimensionality Let the observed data S be: of∈ the factorization. A 2-way FM (degree d =2) captures all single and pairwise S = (A, TI, 2010-1, 5), (A, NH, 2010-2, 3), (A, SW, 2010-4, 1), { interactions between variables: (B, SW, 2009-5, 4), (B, ST, 2009-8, 5), • w0 is the global bias. (C, TI, 2009-9, 1), (C, SW, 2009-12, 5) • wi models the strength of the i-th variable. } • wˆi,j := vi, vj models the interaction between the i- An example for a prediction task using this data, is to estimate th and j-th variable. Instead of using an own model a function yˆ that predicts the rating behaviour of a user for an parameter wi,j R for each interaction, the FM models item at a certain point in time. the interaction by∈ factorizing it. We will see later on, that Figure 1 shows one example of how feature vectors can this is the key point which allows high quality parameter be created from S for this task.1 Here, first there are U | | estimates of higher-order interactions (d 2) under binary indicator variables (blue) that represent the active user sparsity.