arXiv:1901.00431v2 [cs.IR] 6 Jan 2019 ti motn htA gnscnudrtn n dp to upsur adapt AI and The understand traits. can personality agents or AI differences that human individual important with collaborate is and understand it truly generatio can human-machin next that of better agents development the and facilitate To adaptability symbiosis. (AI) char- greater interaction Intelligence by human-computer of acterized Artificial forms [3] new in enabled collaborative have advances and machine recent conversational underlying Meanwhile, the more making models by learning information opportuni recom- several user are elicit in there to strategies practice learning in systems, machine mendation Howeve existing [2]. to suggestions capt make compared and to preferences associations user complex sys elicit their recommendation intelligently now strategies, can learning With tems life. machine daily in of th of make improve space people advent and that search decisions support the the to of down also quality but narrow overload, rec- to information make potential the the and not have individuals systems only Recommendation of [1]. preferences accordingly ommendations and interests the htos ovrainlsystems conversational chatbots, eaatdt uasi al life. daily in prov machines humans actually that to so to adapted services machines, be go recommendation to better The humans with conditions. adapting users different of under instead behave is, users taking how while recommendations account conver accurate intelligent producing m personalized for a the agent implement virtual answer differen techn to missing to factors between the of cal are try which gap terms we question, research particular, the in fundamental In on systems aspects. light recommendation technological sheds and article assistants This recommend Su from qui systems. missing conversations. by currently user, through one-sho is the mechanism preferences for the interactive support his/her insuffic decision to to offer be limited to adapting can is longer goal which the no lists, when is recommendation suggest making of actions by presentation of life daily repertoire domains/applicati citizens’ The the various improve in prefe to aiming multi-aspect recommendations and generate diverse evolving, to users’ tec recommendation model existing gies Meanwhile, App queries tasks. users’ and specific to Assistant ish responses human-like Virtual with Google’s interact as Siri, such systems sational h ehooia a ewe ita Assistants Virtual Between Gap Technological The eomnainssesaeitlietaet htelicit that agents intelligent are systems Recommendation ne Terms Index Abstract Vrulassat,as nw sitlietconver- intelligent as known also assistants, —Virtual Vrulassat,rcmedto systems, recommendation assistants, —Virtual [email protected] .I I. asrct h Netherlands The Maastricht, NTRODUCTION asrctUniversity Maastricht iiro Rafailidis Dimitrios n eomnainSystems Recommendation and sational n fin- and hnolo- rences han ch ologi- ation AI n ions. ckly ons, into ient will ties ure le’s the ide ost ge al s, r, e e - t t . au o epeb arwn ontesto hie and choices of set the down narrowing by people for provi systems pr value applica- Recommendation many over-choice. ameliorate web to to related many impact lems potential in its with adoption along widespread tions, its given stated, n iesgetosjs ieahmnfin ol have would friend human a like just done? suggestions give and question whe fundamental preferences is: most ignori multi-aspect The [6]–[8], recommendations. and producing systems diverse evolving, dialog users’ multi-round engin search traditional ad-hoc round single or on based recommenda either conversational are our systems in to solutions machines Existing adapting dations. recom really personalized generating is while reach that preferences to virtual personal pursue, solved of we be field dream to the the systems in recommendation lying and problems assistants immense them learnin are between machine gap There technological various [5]. large systems on a is based recommendation there are strategies, both assistants Although virtual and pa lives. integral people’s becoming human- started with of have assistant endowed they often and virtual behaviour”, systems like These “dialogue as machines. termed and are we daily phones way our the with Amazon revolutionized accomplish Siri, interact have Cortana Apple’s also Microsoft Assistant, but and Google Alexa, machines, u example, to For Intelligent allowed talk tasks. commands. have just via assistants) to computers not (virtual to Agents talk Conversational to us allowed lot osdrues neatoswt h recommendati bu the conversations. with preferences, via interactions multi-aspect system capture users’ to consider and only to not diverse also framework, evolving, unified a users’ develop to l ways shedding betwe on systems, gap recommendation technological and the assistants with virtual recommendatio deals conversational article unified single This no system. a but forms strategies, recommendation that learning and approach machine assistants on have virtual based There of systems flawed. studies is many systems been recommendation and sistants o a epoiepol iha Ifin h iltalk will who friend AI an with people provide we can How h tlt frcmedto ytm antb over- be cannot systems recommendation of utility The u nweg fbign h a ewe ita as- virtual between gap the bridging of knowledge Our rsol nvriyo Thessaloniki of University Aristotle I R II. ansManolopoulos Yannis [email protected] hsaoii Greece Thessaloniki, ECOMMENDATION S YSTEMS men- ight tion [4], ob- rts ng on en de es n g n s s t helping them explore the space of available options, or serve III. VIRTUAL ASSISTANTS IN PEOPLE’S DAILY LIVES as a filtering component in situations of information overload. From the provider perspective, recommendation systems are The growth of the global virtual assistants market is being personalized services that increase users’ trust and loyalty, primarily driven by the penetration of smartphones along with as well as obtain more knowledge about what people are a rapid growth in the social media traffic which has led to really looking for. Given the explosive growth of information a substantial rise in the consumer awareness about benefits available on the web and Internet of Thing devices, users offered by virtual assistants. With virtual assistants the user are often greeted with more than countless products, movies, and system can interact for multiple semantically coherent restaurants, information about healthcare services and so on. rounds on a task through natural language dialog, and it As such, personalization is an essential strategy for facilitating becomes possible for the system to understand the user needs user experience. Recommendation systems have been playing or to help users clarify their needs by asking appropriate a vital and indispensable role in various information access questions to the users directly. The system has to be capable systems to boost business and facilitate decision-making pro- of asking aspect-based questions in the right order so as to cesses, and are pervasive across numerous web domains such understand the user needs, while search is conducted during as e-commerce, news and media websites. For example, 80% the conversation, and results are provided when the system of movies watched on Netflix came from recommendations feels confident [5]. Virtual assistants interact with the user [9], 60% of video clicks came from home page recommenda- in a simplified dialogue to perform a task, support interfaces tion on YouTube [10], and Amazon announced that 35% of that adapt to the user’s queries, and personal agents that sales comes from recommendation systems. The core mecha- can proactively support the user, modelling his or her needs. nisms of recommendation systems are mainly categorized into Nowadays, with the advent of virtual assistants, there is a collaborative filtering, content-based recommendation systems proliferating demand for technology in various applications and hybrid recommendation systems based on the types of including Banking, Financial Services and Insurance, automo- input data. Collaborative filtering makes recommendations by tive, IT & telecommunications, retail, healthcare, education learning from user-item interactions [1], [11], content-based and others. Recently, it has been reported that about one in recommendation is based on comparisons across items’ and six physicians in EU are already using virtual assistants [22]. users’ auxiliary information, such as text, images and videos It is clear that virtual assistants have now entered people’s [12], and hybrid models refer to recommendation systems that daily life to accomplish tasks. integrate collaborative and content-based strategies, to solve Based on the product, the virtual assistants market has been the data scarcity of user preferences and the cold-start problem segmented into Chatbots and smart speakers. A Chatbot is with users having poor history records [13]–[17]. In a similar a computer program that carries out a conversation through, spirit, over the past decade recommendation algorithms for whereas smart speakers are a type of wireless speakers and rating prediction and item ranking have steadily matured with voice command devices. Virtual assistants try to interact with matrix factorization and other latent factor models emerging human-like responses that are reasonable or interesting [3], as state-of-the-art algorithms to apply in both existing and new [23], [24]. Informational virtual assistants try to help users find applications/domains [18]–[21]. information or directly answer user questions. Task oriented virtual assistants try to help users finish a specific task, such as booking a flight or cancelling a trip. Virtual assistants are However, the recommendation systems algorithms are typi- usually built for a specific domain, such as music, books, cally applied in relatively straightforward and static scenarios: movies, and so on. A recent report shows how virtual assistants given information about a user’s past item preferences, can we are currently used by their owners in the US, UK, France predict whether they will like a new item or rank all unseen and Germany, with 82% of the virtual assistants owners in items based on the predicted interest? In reality, recommenda- these countries using virtual assistants to seek information tion is often a more complex problem, as the evaluation of a such as news, weather, recipes, appointments, advice, offers list of recommended items never takes place in a vacuum. With and so on [25]. The Google Assistant is primarily available richer user interaction models, more elaborate recommenda- on mobile and smart home devices. The Google Assistant can tion systems become possible, which can stimulate, accept engage in two-way conversations. Users primarily interact with and process various types of user input. At the same time the Google Assistant through natural voice, though keyboard the repertoire of actions is no longer limited to the one-shot input is also supported. The Google Assistant is able to search presentation of recommendation lists, which can be insufficient the Internet, schedule events and alarms, adjust hardware when the goal of the system is to offer decision support for the settings on the user’s device, and show information from the user. State-of-the-art methods of recommendation systems are user’s Google account. Google has recently announced that not applicable to a majority of practical scenarios due to the the Google Assistant will be able to identify objects and dynamic change of content e.g., latest news, new products, gather visual information through the device’s camera, and and so on. Thus, it is highly desirable to quickly adapt to support purchasing products and sending money, as well as users’ preferences on new content through effective interactive identifying songs. In a similar spirit, Apple’s Siri is a virtual mechanisms, such as conversations. assistant which uses voice queries and a natural-language user interface to answer questions, and performs actions by to clarify and refine his/her preferences. To achieve this, more delegating requests to a set of Internet services. The software interactive and possibly complex systems are required, so that adapts to users’ individual language usages, searches, and users can fine-tune their profiles to provide the system with a preferences, with continuing use. Finally, the returned results richer repertoire of “conversational moves”. Virtual assistants are individualized. Alexa is a virtual assistant developed by could solve this problem as users discuss with the system Amazon. It is capable of voice interaction, music playback, and enable more interactive recommendation systems without making to-do lists, setting alarms, streaming podcasts, playing complex interfaces while at the same time providing more audiobooks, and providing weather, traffic, sports, and other accurate recommendations. For example, you are considering real-time information, such as news. Microsoft Cortana is to watch a movie but you are not sure you would enjoy it, a virtual assistant that can set reminders, recognize natural and then you would ask your friends for advice. Alternatively, voice without the requirement of keyboard input, and answer imagine that an acquaintance recommends a movie that you do questions using information from the Bing search engine. not think you would enjoy. In the latter case, you would be the Microsoft recently reported that Cortana now has 133 million one willing to provide information to help your friend make monthly users [26], estimating that 325.8 million people per better recommendations in the future. Current recommendation month will use any type of virtual assistants worldwide [27], systems do not allow this type of interactive process to occur [28]. However, all the above virtual assistants are designed between the system and its users, while virtual assistants are to complete certain tasks and do not capture users’ personal, typically oriented towards executing standalone commands evolving and multi-aspect preferences. rather than complex conversations. On the one hand, a plethora In addition, health virtual assistants have also been designed, of personal virtual assistants have started to arise in a variety of such as PocketSkills which supports dialectical behavioural products across domains, ranging from entertainment or retail therapy, aiming at decreasing depression and anxiety trough bots to health virtual assistants. However, virtual assistants are conversations [29]. Chatbots are usually programs that are powered by recent advances in natural language understanding meant to have conversations with users via text or speech and focus on conversations, not on recommendations. On the methods. They are meant for specific tasks in various compa- other hand, conversations in recommendation systems have to nies and sometimes for general chit-chat purposes. They are focus on balancing the explore-exploit trade-off of users. subset or parts of AI bots/assistants rather than being complete A. Information Need virtual assistants [30]. Compared to Chatbots, virtual assistants are built based on complex algorithms of Natural Language The central difference of virtual assistants and recommen- Processing (NLP), Machine Learning, and Artificial Neural dation systems is the representation of the information need: Networks (ANNs), learning throughout their usage and have while virtual assistants, as Information Retrieval (IR) systems, better performance, while Chatbots are based on fixed rules typically use an explicit query prompted by the user, recom- which cannot be further modified. mendation systems exploit user’s data in an implicit manner With the emerging of various conversational devices, and the [40]. In contrast to existing virtual assistants, recommendation progress of deep learning and neural NLP research, especially systems have not only to generate accurate recommendations, on natural language dialog systems, virtual assistants based but novel ones to surprise users and trigger their interest, on direct user-system dialoguing has gained attention by the covering users’ diverse tastes and making it easier for them to academia as well [31]–[38]. Spina and Trippas [34], [35] understand which alternatives exist [41], [42]. studied the ways of presenting search results over speech- B. Scarcity of Users’ Preferences only channels and transcribing the spoken search recordings When virtual assistants seek information, they rely on a to support conversational search via deep learning, and Kang large amount of labelled data, which may not be available et al. [39] explored the initial and follow-up queries users in real-world applications, such as users’ preferences in rec- tend to issue to virtual assistants. However, most of those ommendation systems [43]. The scarcity of users’ preferences deep learning strategies for virtual assistants focus on NLP has a negative impact on the quality of recommendations of challenges instead of recommendation systems. They neither collaborative filtering models, a mainstay strategy in recom- focus on recommendation problems nor do they model and mendation systems. On the contrary, virtual assistants do not utilize users’ preferences to generate recommendations via account the user data scarcity. More recently, several deep user-system conversations. learning strategies have been introduced to solve the data IV. THE TECHNOLOGICAL GAP BETWEEN VIRTUAL scarcity of users’ preferences [44]–[46]. ASSISTANTS AND RECOMMENDATION SYSTEMS C. Adaptation to Evolving Preferences Academic research in recommendation systems is largely In recommendation systems users shift their preferences focused on algorithmic approaches for item selection and over time, depending on different factors [47]–[51]. For exam- ranking, trying to predict the ratings or generate ranked lists. ple, curiosity leads users to explore new items contrary to their However, presenting an ordered list of recommendations might ordinary choices and/or users interact with a bias based on not be the most suitable mechanism to support users in a popularity irrespective to their history record. Users’ dynamic decision-making problem, for example, when the user needs preferences are not yet considered by virtual assistants. D. Adjustment to Cross-domain Recommendation Tasks device, while users in the typing condition by typing into an While virtual assistants are designed to complete specific input box. For the speaking interface, we have to support tasks in users’ daily life, the goal of recommendation systems a voice-to-speech service such as [60], to convert the audio is also to transfer the knowledge of users across different to text. In addition, we have also to allow users to view domains/tasks, also known as cross-domain recommenda- the results and to retry or edit the results manually if the tion systems. The challenge in cross-domain recommendation transcription results in errors, or if their microphone is not systems is to capture users’ multi-aspect behaviours when working. In general, instead of asking users to provide all transferring knowledge and generating recommendations for requirements in one step, the Conversation Manager usually various domains [52]–[56]. Adjustment to different domains guides users through an interactive dialog, following different based on users’ preferences is a key factor that is currently strategies of follow-up queries based on users’ satisfaction missing from virtual assistants which focus only on a specific of the results. Therefore, it is required to investigate various domain. Conversation Manager policies to select what questions to ask and how ask, to minimize the human effort, that is the length of E. Transparency & Explainability questioning-answering response, and emphasize on capturing Another important difference between recommendation sys- user preferences based on the personality and interests. tems and virtual assistants is that in order to build trust G. Quality Metrics between recommendation systems and users, it has become important to complement recommendations with explanations Quality Information Retrieval metrics that proved successful so that users can understand why a particular item has been in recommendation systems [18]–[21] are weak indicators suggested [13]. Transparent and explainable explanations help to evaluate the recommendation performance of a system convincing users that the system knows them very well and with conversations in real-time. The quality of conversational makes custom-made recommendations for them. In fact, when recommendations should be evaluated in terms of number of users understand the recommendation logic, they can even be recommendations that a user will choose expressing the level empowered to correct the system’s proposals. This means that of a participant’s satisfaction; the ranking performance of the recommendations without context lack motivation for a user recommendation mechanism such as Normalized Discounting to pay attention to them. Adding an associated explanation Cumulative Gain, Precision and Recall; the dwelling time, that for a recommendation increases user satisfaction and the is the amount of time that a participant spends before choosing persuasiveness of recommendations [57]. Nonetheless, until a recommendation [61]. Also the goal must be to minimize now, virtual assistants do not provide explanations to users. the number of questions asked to obtain users’ selections and consequently minimize the user time and effort. In particular, F. Preference Elicitation we have to determine which aspect to ask at each time with 1) Capture users’ various feedback: In recommendation a carefully trained strategy, so that the system can always ask systems, there are several ways to state their preferences, with- the most important question to improve its confidence about out involving conversations. For example, users are requested user needs and search results, thus keeping the conversation either to rate the items on a predefined scale, as well as to add as short as possible, and satisfy the user needs as soon as comments. Other recommendation systems limit the feedback possible. Furthermore, measuring the semantic coherence of scale to “thumbs up/down” or positive only “like” statements. conversations of users is also a key performance indicator. Users might be also requested to name a few favorite artists, To be able to provide intelligent responses, the system must movies, books, social events, and points-of-interest, to specify correctly model the structure and semantics of a conversation, their interests in different categories such as “Entertainment”, as it is also pointed out at [62]. Thus, it is required to “Politics”, “Sports”, and so on. While these types of user design numeric scores that indicate more coherent parts of a feedback is be expressed in an absolute manner, relevant conversation and provide a signal for topic drift. For example, studies point out that pairwise preferences are important in this will be achieved by applying the Word2Vec strategy to recommendation systems, as pairwise preferences naturally create the textual embedding and measure their coherence arise and are expressed by users in many decision making based on the respective embeddings’ similarities [63], as scenarios [58], [59]. In everyday life, there are situations where well as following collaborative topic modeling strategies [64]. rating alternative options is not the most natural mechanism A/B online testing in conversational systems has also to be for expressing preferences and making decisions, for instance, performed, allowing the comparison of different conversation we do not rate sweaters when we want to buy one [58]. policies [65]. Pairwise preferences are a pivotal issue in designing effective recommendation systems, as they can lead to larger system V. CONCLUSION usability compared to absolute preferences. Summarizing, there are still many technological gaps be- 2) Users’ conversation strategies: In virtual assistants to tween recommendation systems and virtual assistants that initiate users’ conversations with the system, we have to design researchers have to account when designing a conversational a Conversation Manager. Users in the speaking condition start system, that is learning users’ evolving, diverse and multi- a dialogue with the system by speaking at their computer or aspect preferences via human-computer conversations. The dream of having a really artificial friend to make suggestions [13] Julian J. McAuley and Jure Leskovec. Hidden factors and hidden topics: is not far anymore. understanding rating dimensions with review text. In Seventh ACM Conference on Recommender Systems, RecSys ’13, Hong Kong, China, However, to fulfill this dream, researchers have to answer October 12-16, 2013, pages 165–172, 2013. the following fundamental questions: [14] Dimitrios Rafailidis and Fabio Crestani. A regularization method with • How can user preferences via conversations be modelled inference of trust and distrust in recommender systems. In Machine Learning and Knowledge Discovery in Databases - European Confer- into machine learning models in recommendation sys- ence, ECML PKDD 2017, Skopje, Macedonia, September 18-22, 2017, tems? Proceedings, Part II, pages 253–268, 2017. • Which are the right junctions to perform cross-domain [15] Dimitrios Rafailidis and Petros Daras. The TFC model: Tensor factorization and tag clustering for item recommendation in social recommendation with machine learning models? tagging systems. IEEE Trans. Systems, Man, and Cybernetics: Systems, • To what extent can we provide explainable recommenda- 43(3):673–688, 2013. tions via conversations? [16] Dimitrios Rafailidis, Apostolos Axenopoulos, Jonas Etzold, Stavroula • Manolopoulou, and Petros Daras. Content-based tag propagation and Which are reliable indicators to evaluate the quality of tensor factorization for personalized item recommendation based on recommendation systems via conversations in real-time? social tagging. TiiS, 3(4):26:1–26:27, 2014. Filling the technological gap between recommendation sys- [17] Alexandros Nanopoulos, Dimitrios Rafailidis, Panagiotis Symeonidis, and Yannis Manolopoulos. Musicbox: Personalized music recommen- tems and virtual assistants could help in building a system dation based on cubic analysis of social tags. IEEE Trans. Audio, Speech that is more conversational to allow users to “work together” & Language Processing, 18(2):407–412, 2010. (collaborate) to improve the quality of recommendations and [18] Mohammad Aliannejadi, Dimitrios Rafailidis, and Fabio Crestani. A user experience. collaborative ranking model with multiple location-based similarities for venue suggestion. In Proceedings of the 2018 ACM SIGIR International REFERENCES Conference on Theory of Information Retrieval, ICTIR 2018, Tianjin, China, September 14-17, 2018, pages 19–26, 2018. [1] Yehuda Koren, Robert M. Bell, and Chris Volinsky. Matrix factorization [19] Dimitrios Rafailidis and Fabio Crestani. Learning to rank with trust techniques for recommender systems. IEEE Computer, 42(8):30–37, and distrust in recommender systems. In Proceedings of the Eleventh 2009. ACM Conference on Recommender Systems, RecSys 2017, Como, Italy, [2] Ali Mamdouh Elkahky, Yang Song, and Xiaodong He. A multi-view August 27-31, 2017, pages 5–13, 2017. deep learning approach for cross domain user modeling in recommen- [20] Dimitrios Rafailidis and Fabio Crestani. Joint collaborative ranking dation systems. In Proceedings of the 24th International Conference on with social relationships in top-n recommendation. In Proceedings of World Wide Web, WWW 2015, Florence, Italy, May 18-22, 2015, pages the 25th ACM International Conference on Information and Knowledge 278–288, 2015. Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016, [3] Yueming Sun and Yi Zhang. Conversational recommender system. pages 1393–1402, 2016. In The 41st International ACM SIGIR Conference on Research & [21] Dimitrios Rafailidis and Fabio Crestani. Collaborative ranking with so- Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, cial relationships for top-n recommendations. In Proceedings of the 39th July 08-12, 2018, pages 235–244, 2018. International ACM SIGIR conference on Research and Development in [4] Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. To- Information Retrieval, SIGIR 2016, Pisa, Italy, July 17-21, 2016, pages wards conversational recommender systems. In Proceedings of the 22nd 785–788, 2016. ACM SIGKDD International Conference on Knowledge Discovery and [22] [Online]. Available: https://www.drgdigital.com/ Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 815– drg-digital-innovation-blog/hallo-siri-voice-assistants- 824, 2016. are-set-to-take-off-in-europe, accessed Oct. 21, 2018. [5] Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W. Bruce Croft. [23] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A Towards conversational search and recommendation: System ask, user diversity-promoting objective function for neural conversation models. respond. In Proceedings of the 27th ACM International Conference on In NAACL HLT 2016, The 2016 Conference of the North American Information and Knowledge Management, CIKM 2018, Torino, Italy, Chapter of the Association for Computational Linguistics: Human October 22-26, 2018, pages 177–186, 2018. Language Technologies, San Diego California, USA, June 12-17, 2016, [6] Lorraine McGinty and James Reilly. On the evolution of critiquing pages 110–119, 2016. Recommender Systems Handbook recommenders. In , pages 419–453. [24] Oriol Vinyals and Quoc V. Le. A neural conversational model. CoRR, 2011. abs/1506.05869, 2015. [7] Mehmet H. G¨oker and Cynthia A. Thompson. Personalized conversa- [25] [Online]. Available: https://www.statista.com/statistics, accessed Oct. 21, tional case-based recommendation. In Advances in Case-Based Reason- 2018. ing, 5th European Workshop, EWCBR 2000, Trento, Italy, September 6-9, 2000, Proceedings, pages 99–111, 2000. [26] [Online]. Available: https://www.techradar.com/, accessed Oct. 21, 2018. [8] Mehmet H. G¨oker, Pat Langley, and Cynthia A. Thompson. A personal- [27] [Online]. Available: https://aytm.com/blog/ai-personal-assistants-survey/, ized system for conversational recommendations. CoRR, abs/1107.0029, accessed Oct. 21, 2018. 2011. [28] [Online]. Available: http://www.internetlivestats.com/internet-users/, ac- [9] Carlos A. Gomez-Uribe and Neil Hunt. The netflix recommender cessed Oct. 21, 2018. system: Algorithms, business value, and innovation. ACM Trans. [29] Jessica Schroeder, Chelsey Wilkes, Kael Rowan, Arturo Toledo, Ann Management Inf. Syst., 6(4):13:1–13:19, 2016. Paradiso, Mary Czerwinski, Gloria Mark, and Marsha M. Linehan. [10] James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Tay- Pocket skills: A conversational mobile web app to support dialectical lor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake behavioral therapy. In Proceedings of the 2018 CHI Conference Livingston, and Dasarathi Sampath. The youtube video recommendation on Human Factors in Computing Systems, CHI 2018, Montreal, QC, system. In Proceedings of the 2010 ACM Conference on Recommender Canada, April 21-26, 2018, page 398, 2018. Systems, RecSys 2010, Barcelona, Spain, September 26-30, 2010, pages [30] Mahipal Jadeja and Neelanshi Varia. Perspectives for evaluating con- 293–296, 2010. versational AI. CoRR, abs/1709.04734, 2017. [11] Jiho Yoo and Seungjin Choi. Probabilistic matrix tri-factorization. In [31] Filip Radlinski and Nick Craswell. A theoretical framework for Proceedings of the IEEE International Conference on Acoustics, Speech, conversational search. In Proceedings of the 2017 Conference on and Signal Processing, ICASSP 2009, 19-24 April 2009, Taipei, Taiwan, Conference Human Information Interaction and Retrieval, CHIIR 2017, pages 1553–1556, 2009. Oslo, Norway, March 7-11, 2017, pages 117–126, 2017. [12] Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. Content- [32] Tom Kenter and Maarten de Rijke. Attentive memory networks: Efficient based recommender systems: State of the art and trends. In Recom- machine reading for conversational search. CoRR, abs/1712.07229, mender Systems Handbook, pages 73–105. 2011. 2017. [33] Liu Yang, Minghui Qiu, Chen Qu, Jiafeng Guo, Yongfeng Zhang, Conference on Recommender Systems, RecSys ’14, Foster City, Silicon W. Bruce Croft, Jun Huang, and Haiqing Chen. Response ranking with Valley, CA, USA - October 06 - 10, 2014, pages 321–324, 2014. deep matching networks and external knowledge in information-seeking [49] Dimitrios Rafailidis. A multi-latent transition model for evolving conversation systems. In The 41st International ACM SIGIR Conference preferences in recommender systems. Expert Syst. Appl., 104:97–106, on Research & Development in Information Retrieval, SIGIR 2018, Ann 2018. Arbor, MI, USA, July 08-12, 2018, pages 245–254, 2018. [50] Dimitrios Rafailidis, Pavlos Kefalas, and Yannis Manolopoulos. Prefer- [34] Damiano Spina, Johanne R. Trippas, Lawrence Cavedon, and Mark ence dynamics with multimodal user-item interactions in social media Sanderson. Extracting audio summaries to support effective spoken recommendation. Expert Syst. Appl., 74:11–18, 2017. document search. JASIST, 68(9):2101–2115, 2017. [51] Dimitrios Rafailidis and Alexandros Nanopoulos. Modeling users [35] Johanne R. Trippas, Damiano Spina, Lawrence Cavedon, and Mark preference dynamics and side information in recommender systems. Sanderson. How do people interact in conversational speech-only search IEEE Trans. Systems, Man, and Cybernetics: Systems, 46(6):782–792, tasks: A preliminary analysis. In Proceedings of the 2017 Conference on 2016. Conference Human Information Interaction and Retrieval, CHIIR 2017, [52] Jarana Manotumruksa, Dimitrios Rafailidis, Craig Macdonald, and Iadh Oslo, Norway, March 7-11, 2017, pages 325–328, 2017. Ounis. On cross-domain transfer in venue recommendation. In Advances [36] Tiancheng Zhao and Maxine Esk´enazi. Towards end-to-end learning for in Information Retrieval - 41st European Conference on IR Research, dialog state tracking and management using deep reinforcement learning. ECIR 2019, Proceedings, 2019. CoRR, abs/1606.02560, 2016. [53] Dimitrios Rafailidis and Fabio Crestani. A collaborative ranking model [37] Lina Maria Rojas-Barahona, Milica Gasic, Nikola Mrksic, Pei-Hao Su, for cross-domain recommendations. In Proceedings of the 2017 ACM on Stefan Ultes, Tsung-Hsien Wen, Steve J. Young, and David Vandyke. Conference on Information and Knowledge Management, CIKM 2017, A network-based end-to-end trainable task-oriented dialogue system. In Singapore, November 06 - 10, 2017, pages 2263–2266, 2017. Proceedings of the 15th Conference of the European Chapter of the [54] Dimitrios Rafailidis and Fabio Crestani. Top-n recommendation via Association for Computational Linguistics, EACL 2017, Valencia, Spain, joint cross-domain user clustering and similarity learning. In Machine April 3-7, 2017, Volume 1: Long Papers, pages 438–449, 2017. Learning and Knowledge Discovery in Databases - European Confer- [38] Bhuwan Dhingra, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung Chen, ence, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Faisal Ahmed, and Li Deng. Towards end-to-end reinforcement learning Proceedings, Part II, pages 426–441, 2016. of dialogue agents for information access. In Proceedings of the 55th [55] Mohammad Aliannejadi, Dimitrios Rafailidis, and Fabio Crestani. Per- Annual Meeting of the Association for Computational Linguistics, ACL sonalized keyword boosting for venue suggestion based on multiple 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, lbsns. In Advances in Information Retrieval - 39th European Conference pages 484–495, 2017. on IR Research, ECIR 2017, Aberdeen, UK, April 8-13, 2017, Proceed- [39] Jie Kang, Kyle Condiff, Shuo Chang, Joseph A. Konstan, Loren G. ings, pages 291–303, 2017. Terveen, and F. Maxwell Harper. Understanding how people use natural [56] Nima Mirbakhsh and Charles X. Ling. Improving top-n recommendation language to ask for recommendations. In Proceedings of the Eleventh for cold-start users via cross-domain information. TKDD, 9(4):33:1– ACM Conference on Recommender Systems, RecSys 2017, Como, Italy, 33:19, 2015. August 27-31, 2017, pages 229–237, 2017. [57] James McInerney, Benjamin Lacker, Samantha Hansen, Karl Higley, [40] Daniel Valcarce, Alejandro Bellog´ın, Javier Parapar, and Pablo Castells. Hugues Bouchard, Alois Gruson, and Rishabh Mehrotra. Explore, On the robustness and discriminative power of information retrieval exploit, and explain: personalizing explainable recommendations with metrics for top-n recommendation. In Proceedings of the 12th ACM bandits. In Proceedings of the 12th ACM Conference on Recommender Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Systems, RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018, Canada, October 2-7, 2018, pages 260–268, 2018. pages 31–39, 2018. [41] Sa´ul Vargas. Novelty and diversity enhancement and evaluation in rec- [58] Saikishore Kalloori, Francesco Ricci, and Marko Tkalcic. Pairwise ommender systems and information retrieval. In The 37th International preferences based matrix factorization and nearest neighbor recommen- ACM SIGIR Conference on Research and Development in Information dation techniques. In Proceedings of the 10th ACM Conference on Retrieval, SIGIR ’14, Gold Coast , QLD, Australia - July 06 - 11, 2014, Recommender Systems, Boston, MA, USA, September 15-19, 2016, pages page 1281, 2014. 143–146, 2016. [42] Asela Gunawardana and Guy Shani. Evaluating recommender systems. [59] Saikishore Kalloori, Francesco Ricci, and Rosella Gennari. Eliciting In Recommender Systems Handbook, pages 265–308. 2015. pairwise preferences in recommender systems. In Proceedings of [43] Minghui Qiu, Liu Yang, Feng Ji, Wei Zhou, Jun Huang, Haiqing Chen, the 12th ACM Conference on Recommender Systems, RecSys 2018, W. Bruce Croft, and Wei Lin. Transfer learning for context-aware Vancouver, BC, Canada, October 2-7, 2018, pages 329–337, 2018. question matching in information-seeking conversations in e-commerce. [60] [Online]. Available: https://wit.ai/, accessed Oct. 21, 2018. In Proceedings of the 56th Annual Meeting of the Association for [61] Marisa Vasconcelos, Heloisa Candello, Claudio S. Pinhanez, and Thiago Computational Linguistics, ACL 2018, Melbourne, Australia, July 15- dos Santos. Bottester: Testing conversational systems with simulated 20, 2018, Volume 2: Short Papers, pages 208–213, 2018. users. In Proceedings of the XVI Brazilian Symposium on Human [44] Dimitrios Rafailidis and Fabio Crestani. Recommendation with social Factors in Computing Systems, IHC 2017, Joinville, Brazil, October relationships via deep learning. In Proceedings of the ACM SIGIR 23-27, 2017, pages 73:1–73:4, 2017. International Conference on Theory of Information Retrieval, ICTIR [62] Svitlana Vakulenko, Maarten de Rijke, Michael Cochez, Vadim 2017, Amsterdam, The Netherlands, October 1-4, 2017, pages 151–158, Savenkov, and Axel Polleres. Measuring semantic coherence of a 2017. conversation. In The Semantic Web - ISWC 2018 - 17th International [45] Dimitrios Rafailidis and Fabio Crestani. GeoDCF: Deep collaborative Semantic Web Conference, Monterey, CA, USA, October 8-12, 2018, filtering with multifaceted contextual information in location-based Proceedings, Part I, pages 634–651, 2018. social networks. In Machine Learning and Knowledge Discovery in [63] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Databases - European Conference, ECML PKDD 2018, Dublin, Ireland, Jeffrey Dean. Distributed representations of words and phrases and Proceedings, Part II, 2018. their compositionality. In Advances in Neural Information Processing [46] Dimitrios Rafailidis and Fabio Crestani. Friend recommendation in Systems 26: 27th Annual Conference on Neural Information Processing location-based social networks via deep pairwise learning. In IEEE/ACM Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake 2018 International Conference on Advances in Social Networks Analysis Tahoe, Nevada, United States., pages 3111–3119, 2013. and Mining, ASONAM 2018, Barcelona, Spain, August 28-31, 2018, [64] Chong Wang and David M. Blei. Collaborative topic modeling for pages 421–428, 2018. recommending scientific articles. In Proceedings of the 17th ACM [47] Dimitrios Rafailidis and Alexandros Nanopoulos. Repeat consumption SIGKDD International Conference on Knowledge Discovery and Data recommendation based on users preference dynamics and side informa- Mining, San Diego, CA, USA, August 21-24, 2011, pages 448–456, 2011. tion. In Proceedings of the 24th International Conference on World [65] Andr´es Mu˜noz Medina, Sergei Vassilvitskii, and Dong Yin. Online Wide Web Companion, WWW 2015, Florence, Italy, May 18-22, 2015 - learning for non-stationary A/B tests. In Proceedings of the 27th ACM Companion Volume, pages 99–100, 2015. International Conference on Information and Knowledge Management, [48] Dimitrios Rafailidis and Alexandros Nanopoulos. Modeling the dynam- CIKM 2018, Torino, Italy, October 22-26, 2018, pages 317–326, 2018. ics of user preferences in coupled tensor factorization. In Eighth ACM