0 Time-Aware Click Model

Total Page:16

File Type:pdf, Size:1020Kb

0 Time-Aware Click Model 0 Time-Aware Click Model YIQUN LIU, Tsinghua University XIAOHUI XIE, Tsinghua University CHAO WANG, Tsinghua University JIAN-YUN NIE, Universite´ de Montreal´ MIN ZHANG, Tsinghua University SHAOPING MA, Tsinghua University Click-through information is considered as a valuable source of users’ implicit relevance feedback for commercial search engines. As existing studies have shown that search result position in search engine result page(SERP) has a very strong influence on users’ examination behavior, most existing click models are position-based, assuming that users examine results from top to bottom in a linear fashion. While these click models have been successful, most do not take temporal information into account. As many existing studies have shown, click dwell time and click sequence information are strongly correlated with users’ perceived relevance and search satisfaction. Incorporating temporal information may be important to improve performance of user click models for Web searches. In this paper, we investigate the problem of properly incorporating temporal information into click models. We firstly carry out a laboratory eye-tracking study to analyze users’ examination behavior in different click sequences and find that user common examination path among adjacent clicks is linear. Afterwards, we analyze user dwell time distribution in different search logs and find that we cannot simply use a click dwell time threshold (e.g. 30s) to distinguish relevant/irrelevant results. Finally, we propose a novel click model named Time-Aware Click Model (TACM) that captures the temporal information of user behavior. We compare TACM with a number of existing click models using two real-world search engine logs. Experimental results show that TACM outperforms other click models in terms of both predicting click behavior (perplexity) and estimating result relevance (NDCG). CCS Concepts: •Information systems ! Web searching and information discovery; Retrieval models and ranking; Additional Key Words and Phrases: Click model, click sequence, click dwell time 1. INTRODUCTION Modern search engines record user interactions and use them to improve search quality. In particular, users’ click-through has been successfully used to improve click-through rates (CTR), Web search ranking, query recommendation and suggestions, and so on. Although click-through logs can provide implicit feedback of users’ click preferences [Agichtein et al. 2006b], it is difficult to derive accurate absolute relevance judgments This paper is an extension of [Wang et al. 2015]. Compared with the previous conference version, it introduces a new time-aware click model (TACM) that incorporates click dwell time information. It also includes an extensive experimental assessment of the new model and compares the performance with a number of existing models including PSCM. This work was supported by Tsinghua University Initiative Scientific Research Program (2014Z21032), National Key Basic Research Program (2015CB358700) and Natural Science Foundation (61532011, 61472206) of China. Author’s addresses: State Key Laboratory of Intelligent Technology and Systems, Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2016 ACM. 1539-9087/2016/-ART0 $15.00 DOI: 0000001.0000001 ACM Transactions on Information Systems, Vol. 9, No. 4, Article 0, Publication date: 2016. 0:2 Y. LIU et al. owing to the existence of click noises and behavior biases. Joachims et al. [2005] worked on extracting reliable implicit feedback from user behaviors and concluded that click logs are informative yet biased. Previous studies showed that users’ clicking behaviors are biased towards many aspects such as “position” [Craswell et al. 2008; Joachims et al. 2005], “trust” [Yue et al. 2010], “presentation” [Wang et al. 2013], and so on. To address these problems, researchers have proposed a number of click models to describe users’ practical browsing behavior and to obtain an unbiased estimation of result relevance [Chapelle and Zhang 2009; Dupret and Piwowarski 2008; Guo et al. 2009a]. Most click models follow the findings from Craswell et al. [2008]; Joachims et al. [2005] that users’ attention decreases from top to bottom and assume that users’ potential examination/click paths are unique: the examination/click sequence is consistent with result position. Therefore, these models do not actually take practical temporal information into account. As modern search logs contain a time stamp for each user interaction (e.g., querying, clicking, etc.), we can obtain two important messages from this time stamp: the sequence of user clicks and the dwell time after each user click. For the click sequence information, eye-tracking experiments [Lorigo et al. 2006] showed that only 34% of search users’ scan paths are linear, while over 50% of sessions contain revisiting behaviors (i.e., given a search engine result page (SERP), the user first clicks the result at position i and then clicks the one at position j; j ≤ i) or skipping behaviors. We counted the non-sequential click proportion of multi-click query sessions (when a user clicked two or more results on one SERP) from two commercial search engine logs (Sogou and Yandex; see details in in Table I). We find that nearly one-third (27.9% for Sogou and 30.4% for Yandex) of multi-click sessions contain non-sequential click actions. While most existing click models are based on ranking positions rather than action sequences, the click sequence information is usually ignored, and non- sequential clicking behaviors are not considered, either. Dupret and Liao [2010]; Guo et al. [2012] already showed that the last click in a search session may be more reliable than other clicks. However, the last click performed by a user is not necessarily the one at the lowest position, but the last one in the sequence of clicks. It is thus necessary to take the click sequence information into account. As for click dwell time information, existing studies [Fox et al. 2005; Kim et al. 2014] showed that dwell time on the landing page led by user clicks (click dwell time) is a very strong indicator for user-perceived result relevance and user-perceived search satisfaction. Fox et al. [2005] showed that users are more willing to spend longer durations of time on those pages which are interesting and relevant. Kim et al. [2014] also showed that the longer the dwell time, the more satisfied the user will be, and the more relevant the search result tends to be. Therefore, click dwell time information will be very helpful for us to better understand users’ click behavior and make an accurate relevance estimation. Some existing click models [Wang et al. 2010; Xu et al. 2012, 2010] have tried to cope with click sequence information. These models relax the restrictions on users’ examination sequences (e.g., Wang et al. [2010] assumes that examination sequences can be arbitrary) to increase models’ descriptive power. However, most of these methods abandon the prior knowledge of user examination preference generated from other user behavior studies, which has been found useful. In practice, these models cannot achieve performance that is comparable to other popular click models according to our experimental results. To better understand users’ search interaction processes, we design a laboratory study to analyze users’ practical examination patterns. Our observations confirm clearly that many click behaviors are non-sequential. On the other hand, the ACM Transactions on Information Systems, Vol. 9, No. 4, Article 0, Publication date: 2016. Time-Aware Click Model 0:3 examinations of documents between two clicks usually follow one direction, but with possible skips. This observation shows that some of the assumptions used in the previous position-based models (e.g., the sequential examination assumption) are reasonable in local contexts (i.e., between two clicks). It is thus possible to build a new model upon the existing position-based models by adding new hypotheses. By this means, we not only inherit a framework which has already proved to be effective, but also combine sequential information to better capture users’ preferences for different search results. To better use click dwell time information, we analyze the dwell time distribution for different search logs (Sogou and Yandex). We verified the previous findings in Agichtein et al. [2006a] that clicks with dwell time longer than a certain threshold (e.g., 30 seconds) are good indicators of users’ perceived relevance. We also find that the dwell time distribution in different search engines may be rather different, which means that we must take the distribution factor into consideration to better model user behavior. Combining our findings with the previous conclusions from Kim et al. [2014], we design different mapping functions to model user satisfaction based on click dwell
Recommended publications
  • Russia Technology Internet Local Dominance Strengthens
    12 December 2018 | 1:51AM MSK Russia Technology: Internet Local dominance strengthens; competition among ecosystems intensifies It’s been a year since we published Russia’s internet champions positioned to Vyacheslav Degtyarev +7(495)645-4010 | keep US giants at bay. We revisit our thesis, highlighting that the domestic internet [email protected] OOO Goldman Sachs Bank incumbents are successfully defending their home turf from international competition. We have seen only modest incremental efforts from global players, with some recognizing the importance of local expertise (Alibaba’s agreement to transfer control in AliExpress Russia to local partners) or conceding to domestic market leaders (Uber merged its Russian operations with Yandex.Taxi, citing Yandex’s strong technology and brand advantage). The two domestic market leaders, Yandex and Mail.ru, have solidified their dominant positions in search and social networks, respectively, and are leveraging these core businesses to exploit new sources of growth across their ecosystems (e.g. advertising, taxi, food tech, music). While their ever-expanding competitive overlap is worrying, we note this is not unique for global tech and is still relatively limited in scale. We expect the local dominance trend to continue and see significant untapped opportunities in e-commerce, messengers, local services, cloud and fintech. We re-iterate our Buy ratings on Yandex (on CEEMEA FL) and Mail.ru, and view them as the key beneficiaries of internet sector growth in Russia. We believe the market
    [Show full text]
  • Comparative Analysis of Yandex and Google Search Engines
    Anna Paananen Comparative Analysis of Yandex and Google Search Engines Helsinki Metropolia University of Applied Sciences Master’s Degree Information Technology Master’s Thesis 26 May 2012 PREFACE Working in NetBooster Finland as an International Project Manager specialized in Russian market I’ve been asked many times about differences between the search engines Yandex and Google. This Master’s Thesis is the outcome of my professional experience in the Search Engine Optimisation field in Russia and Finland. I would like to thank all the people from NetBooster Finland and Helsinki Metropolia University of Applied Sciences who has helped me in the development of the study. Special thanks to my instructors Timo-Pekka Jäntti and Ville Jääskeläinen for all the support, both in technical and non-technical matters. I would like to thank also my collegues from NetBooster Finland for their help and support while writing the thesis. Last but not least I would like to thank my mother Tamara Kapitonova, who always has been my prior motivator for the education, and of course to my lovely husband Jukka Paananen for his inconditional support and patience. Helsinki, May 26, 2012 Anna Paananen Author(s) Anna Paananen Title Comparative Analysis of Google and Yandex Search Engines Number of Pages 51 pages + 1 appendix Date 26 May 2012 Degree Master’s Degree Degree Programme Degree Programme in Information Technology Specialisation option Instructor Timo-Pekka Jäntti, Supervisor This thesis presents a comparative analysis of algorithms and information retrieval performance of two search engines: Yandex and Google in the Russian language. Comparing two search engines is usually done with user satisfaction studies and market share measures in addition to the basic comparison measures.
    [Show full text]
  • Final Study Report on CEF Automated Translation Value Proposition in the Context of the European LT Market/Ecosystem
    Final study report on CEF Automated Translation value proposition in the context of the European LT market/ecosystem FINAL REPORT A study prepared for the European Commission DG Communications Networks, Content & Technology by: Digital Single Market CEF AT value proposition in the context of the European LT market/ecosystem Final Study Report This study was carried out for the European Commission by Luc MEERTENS 2 Khalid CHOUKRI Stefania AGUZZI Andrejs VASILJEVS Internal identification Contract number: 2017/S 108-216374 SMART number: 2016/0103 DISCLAIMER By the European Commission, Directorate-General of Communications Networks, Content & Technology. The information and views set out in this publication are those of the author(s) and do not necessarily reflect the official opinion of the Commission. The Commission does not guarantee the accuracy of the data included in this study. Neither the Commission nor any person acting on the Commission’s behalf may be held responsible for the use which may be made of the information contained therein. ISBN 978-92-76-00783-8 doi: 10.2759/142151 © European Union, 2019. All rights reserved. Certain parts are licensed under conditions to the EU. Reproduction is authorised provided the source is acknowledged. 2 CEF AT value proposition in the context of the European LT market/ecosystem Final Study Report CONTENTS Table of figures ................................................................................................................................................ 7 List of tables ..................................................................................................................................................
    [Show full text]
  • A Survey of Collaborative Web Search Through Collaboration Among Search Engine Users to More Relevant Results
    A Survey of Collaborative Web Search Through Collaboration among Search Engine Users to More Relevant Results Pavel Surynek Faculty of Mathematics and Physics, Charles University in Prague, Malostranské náměstí 25, Prague, Czech Republic Keywords: Collaborative Web Search, Social Search, Search Engine, Search Results, Collaborative Filtering, Recommender Systems, System Integration. Abstract: A survey on collaborative aspects of web search is presented in this paper. Current state in full-text web search engines with regards on users collaboration is given. The position of the paper is that it is becoming increasingly important to learn from other users searches in a collaborative way in order to provide more relevant results and increase benefit from web search sessions. Recommender systems represent a rich source of concepts that could be employed to enable collaboration in web search. A discussion of techniques used in recommender systems is followed by a suggestion of integration web search with recommender sys- tems. An initial experience with web search powering small academic site is reported finally. 1 INTRODUCTION AND assumption that a series of queries characterize the effort of what the user want to find better than the MOTIVATION single query. The typical search engine however does not help in this effort – users are put into isola- Web search is an area of the information technology tion typically which precludes any cooperation and industry where artificial intelligence and particularly recommendation from other users based on past knowledge engineering techniques can be applied queries. To be honest, for instance the Bing search with potentially significant impacts. Currently users engine (more correctly the decision engine) uses face a still increasing amount of data of many kinds certain technology that provide search results based that can be accessed through web (textual data, mul- on user’s search history and geographical location.
    [Show full text]
  • Efficient Marketing Communications Towards Russian Customers. Case: Grande Orchidée Fashion Center
    Saimaa University of Applied Sciences Faculty of Business Administration, Lappeenranta Degree Programme in International Business Specialisation in International Business Bachelor's Thesis 2014 Ekaterina Evtikhevich Efficient Marketing Communications towards Russian Customers. Case: Grande Orchidée Fashion Center Bachelor's Thesis 2014 ABSTRACT Ekaterina Evtikhevich Efficient Marketing Communications towards Russian Customers. Case: Grande Orchidée Fashion Center, 47 pages, 2 appendices Saimaa University of Applied Sciences Faculty of Business Administration, Lappeenranta Degree Programme in International Business Specialisation in International Business Bachelor’s Thesis 2014 Instructor: Principal Lecturer Minna Ikävalko, Saimaa University of Applied Sciences The objective of this thesis was to research what are the most efficient market- ing communications of Grande Orchidée Fashion Center towards Russian customers. The focus was kept on individual customers who come regularly to do shopping in Lappeenranta. The theory part of this research work includes an examination of conventional theories of marketing communication tools and analysis of modern marketing in Russia. The empirical part was implemented by studying the current marketing com- munications of the company. The data collection methods included a semi structured interview with the CEO's assistant at the company and a customer survey. The outcomes showed the most efficient communication channels that can be utilized and that can positively contribute to the company's marketing
    [Show full text]
  • Accelerating Development Using the Web: Empowering Poor and Marginalized Populations George Sadowsky, Ed
    Accelerating Development Using the Web: Empowering Poor and Marginalized Populations George Sadowsky, ed. George Sadowsky Najeeb Al-Shorbaji Richard Duncombe Torbjörn Fredriksson Alan Greenberg Nancy Hafkin Michael Jensen Shalini Kala Barbara J. Mack Nnenna Nwakanma Daniel Pimienta Tim Unwin Cynthia Waddell Raul Zambrano Cover image: Paul Butler http://paulbutler.org/archives/visualizing-facebook-friends/ Creative Commons License Attribution-ShareAlike 3.0 This work, with the exception of Chapter 7 (Health), is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Chapter 7 (Health) © World Health Organization [2012]. All rights reserved. The World Health Organization has granted the Publisher permission for the reproduction of this chapter. Accelerating Development Using the Web | Foreword from the Rockefeller Foundation i Foreword from the Rockefeller Foundation For almost 100 years, the Rockefeller Foundation has been at the forefront of new ideas and innovations related to emerging areas of technology. In its early years, the Foundation advanced new technologies to eradicate hookworm and develop a vaccine for yellow fever, creating a lasting legacy of strengthening the application of new technologies to improve the lives of the world’s poor and vulnerable. By the middle of the 20th century, this approach led the Foundation to the pre-cursor to the modern day comput- er. At the dawn of the digital era in 1956, the Foundation helped launch the field of artificial intelligence through its support for the work of John McCarthy, the computing visionary who coined the term.
    [Show full text]
  • The Competitiveness Analysis of the European Language Technology Market
    Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3381–3389 Marseille, 11–16 May 2020 c European Language Resources Association (ELRA), licensed under CC-BY-NC The Competitiveness Analysis of the European Language Technology Market Andrejs Vasiļjevs, Inguna Skadiņa, Indra Sāmīte, Kaspars Kauliņš, Ēriks Ajausks, Jūlija Meļņika and Aivars Bērziņš Tilde Vienibas gatve 75a, Riga, Latvia {fistname.lastname}@tilde.lv Abstract This paper presents the key results of a study on the global competitiveness of the European Language Technology market for three areas – machine translation, speech technology, and cross-lingual search. EU competitiveness is analyzed in comparison to North America and Asia. The study focuses on seven dimensions (research, innovations, investments, market dominance, industry, infrastructure, and Open Data) that have been selected to characterize the language technology market. The study concludes that while Europe still has strong positions in Research and Innovation, it lags behind North America and Asia in scaling innovations and conquering market share. Keywords: competitiveness analysis, language technology market, machine translation, speech technology, cross-lingual search search technologies are covered in more detail, while only 1. Introduction the essence is included for machine translation because it has already been discussed by Vasiljevs et al. (2019b). This paper provides the key results of the competitiveness analysis of the European Language Technology (LT) market
    [Show full text]
  • Ngos and Illicit Drug Policy Change in the Russian Federation: 2010-2013 [Doctoral Thesis]
    COPYRIGHT AND USE OF THIS THESIS This thesis must be used in accordance with the provisions of the Copyright Act 1968. Reproduction of material protected by copyright may be an infringement of copyright and copyright owners may be entitled to take legal action against persons who infringe their copyright. Section 51 (2) of the Copyright Act permits an authorized officer of a university library or archives to provide a copy (by communication or otherwise) of an unpublished thesis kept in the library or archives, to a person who satisfies the authorized officer that he or she requires the reproduction for the purposes of research or study. The Copyright Act grants the creator of a work a number of moral rights, specifically the right of attribution, the right against false attribution and the right of integrity. You may infringe the author’s moral rights if you: - fail to acknowledge the author of this thesis if you quote sections from the work - attribute this thesis to another author - subject this thesis to derogatory treatment which may prejudice the author’s reputation For further information contact the University’s Copyright Service. sydney.edu.au/copyright NGOs and illicit drug policy change in the Russian Federation: 2010-2013 Andrey Zheluk University of Sydney 2014 This thesis is submitted in fulfilment of the requirements of the degree of Doctor of Philosophy 1 “Хотели как лучше, а получилось как всегда.” Виктор Черномырдин. Председатель правительства РФ. 6 августа 1993 г. “We'd hoped for the best, but things turned out just as they always do.” Victor Chernomyrdin Prime Minister of the Russian Federation 6th August 1993.
    [Show full text]
  • Market Opportunities and Key Foreign Trade Restrictions
    United States International Trade Commission Global Digital Trade 1: Market Opportunities and Key Foreign Trade Restrictions August 2017 Publication Number: 4716 Investigation Number: 332-561 United States International Trade Commission Commissioners Rhonda K. Schmidtlein, Chairman David S. Johanson, Vice Chairman Irving A. Williamson Meredith M. Broadbent Catherine DeFilippo Director, Office of Operations Jonathan Coleman Director, Office of Industries Address all communications to Secretary to the Commission United States International Trade Commission Washington, DC 20436 United States International Trade Commission Global Digital Trade 1: Market Opportunities and Key Foreign Trade Restrictions August 2017 Publication Number: 4716 Investigation Number: 332-561 United States International Trade Commission This report was prepared principally by: Project Leader David Coffin [email protected] Deputy Project Leader Jeremy Streatfeild [email protected] Office of Industries Jared Angle, Renato Barreda, Laura Bloodgood, Sharifa Crawford, Sharon Ford, Eric Forden, John Giamalva, Fernando Gracia, Jeffrey Horowitz, Mahnaz Khan, Dan Kim, Erick Oh, Sarah Oliver, Chris Robinson, Mitchell Semanik, George Serletis, Isaac Wohl Office of Economics Nabil Abbyad, Tamara Gurevich, Peter Herman, Grace Kenneally, Ricky Ubee, Heather Wickramarachi Content Reviewers Jennifer Powell and David Riker Editorial Reviewers Judy Edelhoff and Peg Hausman Office of Analysis and Research Services Maureen Letostak Document Preparation and Support Jaime
    [Show full text]
  • Written Statement for the Record by Megan Gray, General Counsel And
    Written Statement for the Record by Megan Gray, General Counsel and Policy Advocate for DuckDuckGo for a hearing entitled "Online Platforms and Market Power, Part 2: Innovation and Entrepreneurship" before The House Judiciary Subcommittee on Antitrust, Commercial and Administrative Law Rep. David Cicilline, Chair Rep. James Sensenbrenner, Ranking Member Tuesday, July 16, 2019 DuckDuckGo is a privacy technology company that helps consumers stay more private online. DuckDuckGo has been competing in the U.S. search market for over a decade, and it is currently the 4th largest search engine in this market (see market share section below). From the vantage point of a company vigorously trying to compete, DuckDuckGo can hopefully provide useful background on the U.S. search market. Features of Competitive General Search Engines A competitive U.S. general search engine in 2019 must have a set of high-quality search features, and ensure none are substandard or shown at the wrong times. This set of mandatory high-quality search features includes: An up-to-date index of most all of the English web pages on the Internet (referred to as “organic links”) Maps Local business answers (e.g., restaurant addresses and phone numbers) News Images Videos Products/shopping Definitions Wikipedia reference Quick answers (calculator, conversions, etc.) Additional niche features may also be necessary to be competitive with particular consumer segments, such as: Up-to-date indexes of web pages in other languages Sports scores Airplane flight information Question/Answer reference (e.g., for computer programming) Lyrics When DuckDuckGo launched in 2008, this list was much smaller, and arguably just one item was a required feature: organic links (sometimes referred to as “the ten blue links”).
    [Show full text]
  • Optimizacija Web Stranica Kao Determinanta Uspješnosti U Turizmu
    SVEUČILIŠTE U SPLITU EKONOMSKI FAKULTET DIPLOMSKI RAD OPTIMIZACIJA WEB STRANICA KAO DETERMINANTA USPJEŠNOSTI U TURIZMU Mentor: Studentica: doc. dr. sc. Daniela Garbin Praničević Marija Puljić 2160455 Split, ožujak, 2019. SADRŽAJ 1. UVOD ............................................................................................................... 1 1.1. Problem istraživanja .................................................................................................... 1 1.2. Predmet istraživanja .................................................................................................... 2 1.3. Istraživačka hipoteza .................................................................................................... 3 1.4. Ciljevi istraživanja ........................................................................................................ 3 1.5. Metode istraživanja ...................................................................................................... 4 1.6. Doprinos istraživanja ................................................................................................... 5 1.7. Obrazloženje strukture diplomskog rada .................................................................. 5 2. OPTIMIZACIJA WEB STRANICE ............................................................. 7 2.1. Pojam optimizacije web stranica (SEO) ..................................................................... 7 2.1.1. Search engine optimisation .................................................................................
    [Show full text]
  • Yandex: General Information
    YANDEX YOUR GATEWAY TO RUSSIA Manfred Schlosser Head of Sales and Partnerships – Europe at Yandex www.yandex.com | Your Gateway to Russia | November 2020 Internet and audience in Russia Internet audience in Russia 97 million 71% 190 minutes users or 79% of the population of of users access the internet on average time users Russia mobile devices spend online each day Mediascope WEB-Index. Monthly Reach, population 12+, October 2019 – March 2020, all devices, Russia 0+ www.yandex.com | Your Gateway to Russia | August 2020 Top 10 countries by number of internet users: Russia is 8th largest Internet audience Population in millions 1420 1369 58 41 329 89 270 212 201 168 53 70 59 56 144 79 132 67 127 94 Internet Words Stats, all countries, June 2019. 4 www.yandex.com | Your Gateway to Russia | August 2020 Time spent on the internet 3:28 3:10 3:16 3:08 2:49 Desktop &Mobile 1:18 1:23 1:38 1:52 1:59 Desktop 3:33 3:01 2:53 2:35 2:13 Mobile Mediascope WEB-Index. Average time per day, hours:minutes, March 2020. Russia 0+ 5 www.yandex.com | Your Gateway to Russia | August 2020 Online advertising market 6 Volume of the Russian advertising market in 2019 49.4% 35.4% 8.9% 3.2% 3.1% Total: 494 billion rubles +5% YoY Internet TV Out of Radio News Home media 244 billion 175 billion 43.8 billion 16 billion rubles 15.1 billion rubles rubles rubles -5% YoY rubles +20% YoY -6% YoY 0% YoY -16% YoY ACAR, 2019.
    [Show full text]