Automatic Device Segmentation for Conversion Optimization A Forecasting Approach to Device Clustering Based on Multivariate Time Series Data from the Food and Beverage Industry David Johansson Computer Science and Engineering, master's level 2020 Luleå University of Technology Department of Computer Science, Electrical and Space Engineering Abstract This thesis investigates a forecasting approach to clustering device be- havior based on multivariate time series data. Identifying an equitable selec- tion to use in conversion optimization testing is a difficult task. As devices are able to collect larger amounts of data about their behavior it becomes increasingly difficult to utilize manual selection of segments in traditional conversion optimization systems. Forecasting the segments can be done au- tomatically to reduce the time spent on testing while increasing the test ac- curacy and relevance. The thesis evaluates the results of utilizing multiple forecasting models, clustering models and data pre-processing techniques. With optimal conditions, the proposed model achieves an average accuracy of 97,7%. i Preface There are plenty of people who helped bring this thesis to fruition, and I am grateful to each and every one of them. The completion of this thesis would not have been possible without the support and insights of Robert Westerlund and Andreas Stormvinge at Future Ordering. I also had the great pleasure of working with Johan Kristiansson, whose knowledge and ideas surrounding data analysis and machine learning had a great impact on my work. I would like to extend my sincere gratitude to Ahmed Elragal for supervising this thesis along with all the valuable guidance he has provided. Especially helpful to me during this time were Adam Sawert and Johan Delissen, who were always there to discuss the difficult problems that I encountered while working on the thesis. ii Contents 1 Introduction1 1.1 Background...............................1 1.2 Motivation................................1 1.3 Problem definition...........................2 1.4 Delimitations..............................2 1.5 Thesis structure.............................2 2 Related Work4 3 Theory5 3.1 Clustering................................5 3.1.1 Prototype Based Clustering..................5 3.1.2 Hierarchical Clustering.....................6 3.1.3 Density Based Clustering....................6 3.2 Forecasting...............................6 3.2.1 SARIMA............................7 3.2.2 Theta Model..........................7 3.2.3 Vector Auto Regression.....................7 4 Method8 4.1 Data Pre-processing..........................8 4.1.1 Data Overview.........................8 4.1.2 Data Preparation........................9 4.1.3 Data Transformation...................... 10 4.1.4 Outlier Analysis......................... 10 4.1.5 Dimensionality Reduction................... 11 4.2 Model Selection............................. 11 4.2.1 Selection of Clustering Models................. 11 4.2.2 Selection of Forecasting Models................ 13 4.3 Model Validation............................ 14 4.4 Experimental Methodology....................... 15 4.5 Software................................. 16 5 Results 17 5.1 Cluster Results............................. 17 5.2 Forecast Results............................. 23 6 Discussion 26 6.1 Generalizability............................. 27 iii 6.2 Computational Complexity....................... 28 7 Conclusions and Future Work 30 A Training Set Results 35 iv 1 Introduction 1.1 Background Future Ordering is a company that provides digital ordering solutions for the Food and Beverage industry. One of the solutions utilizes self-service kiosks, where customers place their orders through a digital interface. The software interface of such kiosks is optimized to maximize sales for the restaurant. A common strategy for increasing customer purchases in traditional e-commerce systems is conversion optimization. It allows businesses to compare the sales performance of different software versions. Comparisons are made by splitting devices into two equitable groups. One of the group acts as a control and is left unchanged while the other is updated to a newer version. Performance statistics are then collected over time and compared at the end of the test. It is also possible to take a multivariate approach that utilizes multiple test groups. The results of such a test can then be used to estimate the sales impact of new system updates. To acquire a more comprehensive view of the test results user segmentation is often used[1]. By segmenting the results based on user type (e.g. new user, old user, high spender, low spender) it is possible to observe behaviors that are not applicable to the average user. This is because the effects a change has on a certain type of user might be negated by another type of user. The segmentation process is usually done by manually specifying rules that determine how the users should be divided. Since the kiosks only gather anonymous information about the ordering process for each order it's not feasible to apply user segmentation when comparing kiosks. The behavior of the kiosks also fluctuates depending on external factors such as time of day. This means that comparisons must be performed over long periods of time to negate the impact of seasonal effects. The alternating behavior might also cause segments based on historical differences to correlate poorly with the actual differences during the test periods. In an effort to make effective comparisons between kiosk versions Future Ordering is investigating alternative solutions to automate the segmentation process of self-service kiosks. 1.2 Motivation The main purpose of this thesis is to increase the accuracy of self-service device comparisons. In addition to improvements in test accuracy and relevance, au- tomating the segmentation process reduces the need for manual intervention in the segmentation step. Deploying tests within groups of kiosks that are predicted 1 to exhibit similar behaviors in the future, should decrease the impact of seasonal bias and, by extension, lower the required test duration. The results from this study could be used to segment digital devices in general and might therefore be applied in settings outside the Food and Beverage industry. 1.3 Problem definition • Which clustering method is most suitable for clustering kiosk behavior? • How does the length of the time interval used to create the input features affect the cluster quality? • How does the frequency at which the order data is aggregated affect the cluster quality? • Which forecasting method is most suitable for forecasting kiosk behavior? • How does the length of the previously observed values affect the forecast accuracy? • How does the length of the forecast horizon affect the forecast accuracy? 1.4 Delimitations • The clustering models investigated in the thesis was limited to a selected subset of available clustering models (see section 4.2). • The forecasting models investigated in the thesis was limited to a selected subset of available forecasting models (see section 4.2). • Clustering and forecasting time periods exceeding four weeks was not not in- vestigated, since test periods that require more than four weeks were deemed to be too long. • In the absence of labeled data, internal clustering measures was used to validate the performance of the different clustering models (see section 4.3). 1.5 Thesis structure In section2 previous works related to the problems and solutions introduced in the thesis are presented. In section3 underlying theory about clustering and forecasting techniques are introduced. Section4 starts by describing the data pre- processing and then moves on to outline the experimental methodology used in the thesis. It also introduces some of the theory that was not substantial enough to 2 include in section3. In section5 the experimental results are presented, with the exception of the training set results, which can be found in appendixA. Section6 discusses the results from section5 in relation to the problem definition in section 1.3. Section7 gives a brief conclusion and discusses potential improvements and ways to continue the work started in this thesis. 3 2 Related Work There are several instances where data mining has successfully been used in au- tomating the segmentation process for conversion optimization. Although most of the solutions utilize the K-means clustering algorithm to generate the seg- ments[2][3], there is no established consensus on which clustering method performs the best for this particular task. Beyond K-means, there are examples utilizing hi- erarchical clustering[4][5] in e-commerce segmentation, as well as, instances where soft[6] clustering was used. Most of the existing research on automatic behavior segmentation in e-commerce systems is based on the behavior of individual users. This thesis investigates solutions for segmenting the behavior of individual devices in environments where data on individual users is unobtainable. Many of the existing solutions base the segments on historical data gathered over long periods of time resulting in generalized partitions. In comparison, this thesis takes a forecasting approach to the segmentation problem, basing the segments on forecasted values instead of historical values. The most frequently utilized forecasting techniques are based on statistical models, with ARIMA[7] being one of the more widely used. Recent years have introduced more sophisticated techniques based