Page 1 – CONGRESS 2015 Copyright © ESOMAR 2015 WHEN SHOULD WE ASK, WHEN SHOULD WE MEASURE? COMPARING INFORMATION FROM PASSIVE AND ACTIVE DATA COLLECTION Melanie Revilla • Carlos Ochoa • Roos Voorend • Germán Loewe INTRODUCTION Different sources of data Questionnaires have been a fundamental tool for market research for decades. With the arrival of internet, the questionnaire, a tool invented more than 100 years ago, was simply adapted to online data collection. The arrival of online panels in the 2000s meant an important boost for online questionnaires, and as a consequence, the tipping point for their online migration. We have come this far making all kind of market research projects using almost always the same tool. But this does not mean there is no room or need for improvement. If researchers massively used questionnaires during all these years, it is not because this is the most suited data collection tool for all the problems they faced; but because there were no better alternatives available. In addition, nowadays, things are changing really fast. This affects survey-based market research, particularly when the data collection process relies on respondent’s memory. A tool that was working reasonably in the past may not be sufficient anymore. Indeed, in the last decades, with the development of new technologies, an increased part of the consumer activity takes place on the internet. Also, we have witnessed an explosion of relevant events for marketing: increased consumer ad exposure through multiple channels, increased availability of products and brands, complex and quick decision making processes, etc. Larger memory issues may be expected in this new situation. Moreover, key actions that need to be deeply understood for marketing purposes, like buying a product, used to take time and be rich in context, helping people to remember these behaviours and, therefore, facilitating the research process. Now, with the arrival of e- commerce, there is a loss of context, a reduction of the duration of the action and an increased distraction. All this makes it much more difficult to properly remember the decision making and purchase process, and challenges the traditional survey data collection. In the last years, we have witnessed the emergence of two new ways of gathering information from consumers, which could represent alternatives. On the one hand, social media offers easy access to fresh and spontaneous content. To a certain extent, it is a paradoxical way of collecting data. While online panels are making an effort to get people to answer surveys, some persons share their thoughts and opinions in public on social media without expecting any economic reward. On the other hand, passive online data collection is a research opportunity never seen before, that benefits from the development driven by large online platforms (e.g. Google or Facebook) which led to the emergence of new technologies that can manage large amounts of information efficiently, often called Big Data. These technologies allow massive data collection of the navigation activity of hundreds of thousands of users. These same platforms also promote the development of algorithms and processing technologies to analyze efficiently data. The passive data collection technology studied in this paper uses a small tracking application (meter) installed on the participant’s device to register their online behaviour. Demographic variables can be combined with passive data as well. For example, Ochoa, Bretcha, Fusco, and Tomàs (2015) track the devices that pregnant women and recent mothers use to access the Internet, and define different patterns of surfing the web and apps usage on mobile devices. Page 2 – CONGRESS 2015 Copyright © ESOMAR 2015 These different methods of data collection both offer possibilities and limitations. For example, where the questionnaire is a good tool to collect opinions from consumers, it has been long used to collect recalled information from respondents, such as last seen advertisements, visited websites, etc. But people may not recall this information correctly. Yet, technology is currently offering us a way to actually measure and access such information by observing individuals, instead of asking them. There is a real need for looking into new directions, renewing surveys, improving passive data collection analyses and combining different approaches to handle the same problems, such that instead of focusing only on one facet of this problem, we are able to see its different aspects and we get a better image. Research questions Now that these new innovative data collection technologies are becoming more and more mainstream, it is crucial for market research professionals to find out when to use which method. It is common in market research to distinguish between two types of data: 1) subjective data, such as opinions, emotions, intentions, moods or preferences, that is, all kinds of information that is inside our brains; and 2) objective data, such as behaviour, that is, all kinds of physical actions for which there might be a trace or record outside the brain of the individual. For subjective data, asking is still the most realistic way to collect information. For objective data, there are multiple possibilities to collect the data, including survey and passive data collection. On the one hand, analysing passive data can be challenging because data sets are large and it can be complicated to extract the information of interest, whereas in a survey, the information of interest is collected directly and therefore, analysing the information is simple. On the other hand, passive data provides an exact measure of the behaviour of interest, whereas a questionnaire relies on the ability to recall a specific event or evaluate a certain opinion. Thus, it is important to know when the information from both sources is similar, and when the information of one source is more accurate than the other. In particular, this study focuses on online behaviours, since the online environment offers new and potentially very interesting tools to gather data. The main goal of this paper is to compare the information obtained using passive and active data collection in order to shed some light on a key question: when should we (i.e. when is it more effective to) use each method? Main hypotheses We start from the idea that the meter provides a more objective data collection method for online behaviours, since it avoids the biases inevitably associated with traditional questionnaires, in which the individual himself must self-report his habits (Ochoa and Savín, 2015). The general hypothesis is that people perform quite poorly when remembering and evaluating their online activities. In particular, survey questions that involve recalling past activities are quite difficult to answer because of the limitations of human memory. Indeed, as underlined by Kahneman and Riis (2005, p.285), “an individual's life could be described - at impractical length - as a string of moments. A common estimate is that each of these moments of psychological present may last up to three seconds, suggesting that people experience some 20,000 moments in a waking day, and upwards of 500 million moments in a 70-year life. [...] What happens to these moments? The answer is straightforward: with very few exceptions, they simply disappear.” When survey questions focus on past behaviours, “it is not an experiencing self that answers, but a remembering and evaluating self, the self that keeps score and maintains records”. The recall itself can be quite biased. Thus, if large differences are found between passive and active data collection, we will conclude that the questionnaire is not an accurate tool to gather such behaviours. On the contrary, if small or no differences are found, we will conclude that both sources can be used. Then, the researcher can use practical and economical aspects to decide about which source to use. In this study, we will focus on four aspects/conditions that we think affect the differences between both sources of data: time, frequency of behaviours, social desirability, and device. The time Because of memory limitations, we expect people to report more accurately what they just did than what they did in a longer period (H1a). Besides, how well people report their online behaviour in a longer period is expected to depend on the length of this period: the shorter the period, the better the report (H1b). Page 3 – CONGRESS 2015 Copyright © ESOMAR 2015 Therefore we expect less difference between the information from passive and active data collection when studying the last five websites to which people connected than when studying the five websites where they spent more time in the last x days or that they visited more in the last x days. Moreover, we expect fewer differences between both sources when studying the online activities of the last seven days than when studying those of the last two months. The frequency of the behaviours People report daily behaviour or behaviour that never ocurred more accurately than behaviour they did sometimes. Except for behaviour that “never” happened, the less often people do some activities, the worse they are in reporting the frequency (H2). Therefore when asking about frequency of an online activity, we expect fewer differences between survey answers and observed data for panellists in both end-sides of the distributions (never or daily) than for others. Moreover, we expect people to report behaviour that occurred very rarely less accurately than behaviour that occurred more often. The social desirability People under-report behaviours that are considered socially undesirable, such as visiting adult websites or downloading movies. Therefore, we expect a significant under-reporting of visits to adult websites in the declared data compared with the observed one. We expect this under-reporting to be higher than the one of other activities not considered so socially sensitive, like looking for an itinerary (H3). The device on which the online behaviours occur We measure the activity on both PC and smartphone devices.