People First: A User-Centric

Hybrid Online Audience by John Brauer, Manager, Product Leadership, Measurement Model Nielsen

Overview As with any media measurement methodology, hybrid online audience measurement calculations are only as good as the input data quality. Hybrid online audience measurement goes beyond a single metric to forecast online audiences more accurately by drawing on multiple data sources. The open question for online publishers and advertisers is, ‘Which hybrid audience measurement model provides the most accurate measure?’ • User-centric online hybrid models rely exclusively on high quality panel data for audience measurement calculations, then project activity to all sites informed by consumer behavior patterns on tagged sites • Site-centric online hybrid models depend on cookies for audience calculations, subject to the inherent weaknesses of cookie data that cannot tie directly to unique individuals and their valuable demographic information • Both hybrid models assume that panel members are representative of the universe being measured and comprise a base large enough to provide statistical reliability, measured in a way that accurately captures behavior • The site-centric hybrid model suffers from the issue of inaccurately capturing unique users as a result of cookie deletion and related activities that can result in audience measurement estimates different than the actual number • Rapid uptake of new online access devices further exacerbates site-centric cookie data tracking issues (in some cases overstating unique browsers by a factor of five) by increasing the likelihood of one individual representing multiple cookies

Only the user-centric hybrid model enables a fair comparison of audiences across all web sites, avoids the pitfall of counting multiple cookies versus unique users, and employs a framework that anticipates future changes such as additional data sets and access devices. Source Effect: Cookies, Focus on Panel Focus on Census Panels or Both? Panel-based measurement utilizes a Census-based models collect data via sample of online users to project to a measurement tag, a small piece of Consumer behavior serves as the bedrock the broader population. Panel quality is computer code appended to each page of audience research. Traditionally, online critical to the integrity of the data used of a site, enabling browser behavior browsing behavior was measured using in audience projections. tracking within the site. When loading one of two approaches. In the panel- this code, the browser passes along basic In its “Guide to Understanding Online based method, behavior of an audience information such as URL, operating system Measurement Alternatives,” i the Media sample was extrapolated to the larger and browser type to the measurement Rating Council articulates the benefits of population. In the site-based census firm’s servers. model, measurement tags attached panel-based methods, namely, the robust to web pages enabled census-level demographic information of panels and This provides near census-level accounting behavior measurement. the comprehensive online behavior of of site activity, a direct measure of sampled individuals (as opposed to consumer behavior on a site, helping The most recent developments in online single site or campaign measurement). understand the amount of content audience measurement leverage hybrid consumed. Once filtered for non-relevant The Guide also cites three critical methodologies that combine aspects traffic such as search bots, spiders, auto- requirements for effective panel-based of both the panel-based and census refresh, international and similar traffic, measurement:ii approaches, offsetting the shortcomings the volumetric data provides valuable of each model. The differences between 1. Participants must represent the input to the models. hybrid online audience measurement universe being measured in terms of models reside in how the data inputs are the relevant behaviors used—specifically, the relationship between the monthly audience and the monthly 2. The number of participants needs The threshold of cookie counts. to be large enough to provide the reliability and stability required for “measurement difficulty In genetics, the theory of hybrid vigor measurement applications, and proposes that hybrids will deliver improved for achieving this measure performance results over that of purebreds. 3. People must be measured in a way The same theory applies to online audience that accurately represents their (individual site users) in a measurement. Hybrid models that go behavior. beyond a single metric forecast online Panel quality directly impacts the quality census-based environment audiences more accurately by drawing of data generated using that panel. Hybrid on multiple data sources. The goal of the measurement solutions address some of is quite high. hybrid audience measurement method is to the challenges associated with panel-only project audiences based on all individuals methods by incorporating census-level ”–IAB, 2009 with access to the Internet. Access points behavior. Regardless of which hybrid include home and work PCs, public locations online audience measurement approach Measurement tags accurately measure site (libraries, cafés), dormitories, Internet- is selected, audience calculations are only activity through an exchange of a browser enabled mobile devices and other devices as good as the input data quality. with Internet browsing capabilities (e.g. cookie. Cookies identify the unique Xbox 360). The process involves three basic Reliable audience measurement demands browser accessing content on a site, and steps: establishing the demographic profile an ongoing investment in maintaining a are typically used to customize content of the population with Internet access, high quality panel and the ability to draw and advertising messages, maintain user estimating the likely audience for each on use-based observations to project preferences and generate web analytics. site, and selecting panelists to represent activity across all sites, informed by Among hybrid solutions, only site-centric audience from other locations. measurement-tagged input. hybrid methods are dependent on cookies for audience calculations.

2 One Horse or One Race?

Horse racing provides a helpful analogy for understanding The problem comes down to this: one person equals many the difference between panel-based and cookie-based data. cookies. The cookie approach can over- or under-count due Whether judging a race or evaluating a media plan, it all centers to the way people use the Web. Users access from multiple around accurate measurement. devices, share web-enabled devices and switch to browsers that may clear or reject cookies, which thereby overwhelm systems. In media planning, performance evaluation across competitors is key. In horse racing, when the playing field is level and Cookies are demographically “blind,” and unable to link the conditions are consistent, each race provides real-time surfing behavior back to audience demographics, which fails performance evaluation of all the horses in the race. Panel data to meet even the minimal audience reach criteria established compares to a single race where all horses are running at the same by the IAB: “…the measurement organization must utilize in time, on the same length track under identical turf and weather its identification and attribution processes underlying data conditions on a consistent, comparable basis. Cookie [census] that is, at least in a reasonable proportion, attributed directly data compares to multiple one-horse races, where all the horses to a person.” are running at different times, on different length tracks, and under different turf and weather conditions, so rendering data comparisons is difficult at best.

User-Centric Hybrid Audience Measurement Cookie-Dependent Measurement

Consistent conditions create a level playing field and a fair race for all participants, creating an ability to effectively compare horses since they are running the same race on the same track

Independent conditions lead to individual races with individual results: no ability to compare horses when they are running their own races

3 Cookie Cutters Despite the benefits of web analytics Even more potentially damaging to access devices. Tablets, smartphones, and user experience customization, cookie-dependent models than the video game consoles and televisions now online researchers have discovered a over- and under-counting associated with come equipped with Internet browsing fundamental problem with cookies. cookie deletion and suppression are: capabilities, often usurping standard issue Transient elements by design, cookies are laptops and desktops as the access portal • Private browsing not permanent elements associated with of choice. a single user. As a result, users who delete • Inherent browser bias A more connected population increases cookies may have many cookies assigned • Opt-in privacy legislation the likelihood of one individual to them while online, resulting in inflated representing many cookies. To audience numbers. • Internet device proliferation demonstrate the phenomenon of cookie It is this aspect of cookies that defines an • Cookie bloat bloat, look at the case of Australia, where underlying difference between user-centric the number of unique browsers recorded Private browsing is becoming an and site-centric hybrid models. Numerous by cookies in April 2011 was five times increasingly available option on popular studies underscore the tenuous accuracy the number of Australians with Internet browsers such as Microsoft Internet of cookies, revealing that cookie deletion access—a clear impossibility. Explorer®, Google Chrome, Apple Safari rates range from 30 to 43 percent. and Mozilla Firefox. Private browsing eliminates any trace of the browsing or Chart 1: Cookie Deletion Rates search history by wiping out all cookies The People Factor and cached files retrieved during the One or More Times per Month The potential for inaccurate reporting of online session. unique visitors through cookie counting Study Source % of Users Research from Stanford tested the impact alone prompted the Interactive Advertising of private browsing on ad tracking for Bureau (IAB) to write into its 2009 three ad campaigns. Private browsing online audience research guidelines a Belden Associates (2004)iii 40% activity was detected in approximately recommendation that audience/visitor/ seven percent of the cases, meaning site user data for any site “must include a component that is directly attributable to Jupiter Media (2005)iv 39% visitation numbers based on cookies were inaccurately under-recording unique users people,” such as panel data. for these campaigns. Adjusting cookie counts using panel Atlas Digital Marketing (2005)v 43% Further compounding the issue, private behavioral data may help site-centric browsing utilization rates differed approaches to meet this minimum Nielsen (2005) 30% significantly from browser to browser, threshold standard. However, the number with higher adoption rates registered of challenges (e.g., cookie suppression, for Safari than for Internet Explorer. This deletion, private browsing, opt-in vi comScore (2007) 31% introduces the issue of browser bias into requirements, devices per user, users per the private browsing discussion. device, browsers per device and other variables) affecting the accuracy of Another complicating factor for individual site user counts generated by researchers in the European Union will be cookies heightens the risk of error. This To further illustrate the concerns related the different ways member states choose fact was underscored by the IAB, which to cookie deletion in site-centric audience to implement Directive, 2009/136/EC, noted in its guidelines that “the threshold measurement models, see “Estimating the which requires an explicit “opt-in” consent of measurement difficulty for achieving Effects of Cookie Deletion” on page 5. to place or read cookies. this (audience) measure in a census-based ix Perhaps the most significant issue of all environment is quite high.” affecting the accuracy of cookie counts is the explosion in the number of Internet

4 Estimating the Effects of Cookie Deletion For illustration purposes, imagine a website with 10,000 visitors per month. Approximately 10 percent of these 10,000 people delete their cookies every day; another 20 percent delete their cookies once a week, and the rest don’t delete their cookies at all. Daily deleters Weekly deleters Non-deleters Cookie In a hypothetical 30 day month, the following cookie counts were observed per type of visitor: 10% 20% 70% count Visit every day 2000 X 10% X 30 = 2000 X 20% X 4 = 2000 X 70% X 1 = (2000) 6000 1600 1400 9000 Daily deleters Weekly deleters Non-deleters Visit every dayCOOKIES Visit once/week 3000 X 10% X 4 = 3000 X 20% X 4 = 3000 X 70% X 1 = (3000) 1200 2400 2100 5700 Visit once/week Visit once/month 5000 X 10% X 1 = 5000 X 20% X 1 = 5000 X 70% X 1 = (5000) 500 1000 3500 5000 Visit once/month TOTAL 19,700 Now, calculate the audience estimated based on the cookies counted. Assuming that 20 percent of the 10,000 site visitors go to the website daily, 30 percent visit weekly, and the rest only visit once monthly, these calculations generate the following numbers:

Daily deleters Weekly deleters Non-deleters Cookie 10% 20% 70% count Visit every day 2000 X 10% X 30 = 2000 X 20% X 4 = 2000 X 70% X 1 = (2000) 6000 1600 1400 9000 Daily deleters Weekly deleters Non-deleters Visit every dayCOOKIES Visit once/week 3000 X 10% X 4 = 3000 X 20% X 4 = 3000 X 70% X 1 = (3000) 1200 2400 2100 5700 Visit once/week Visit once/month 5000 X 10% X 1 = 5000 X 20% X 1 = 5000 X 70% X 1 = (5000) 500 1000 3500 5000 Visit once/month TOTAL 19,700

This highly simplified example underscores how the 10,000 individual visitors were counted incorrectly as 19,700 visitors, an audience overstatement by a factor almost twice the correct number as a direct result of cookie deletion.

5 cookies, it is virtually impossible to meet This user-centric hybrid model focuses Model Evolution all three assumptions. Further confounding on quality behavioral data available from Despite pioneering the site-centric, site-centric measurements are the both data sources—panel and census-level. cookie-driven hybrid methodology in 2005, mixed methodologies used to determine This more robust, people-focused model Nielsen later reversed its opinion based on audiences on tagged and untagged sites. demonstrates a greater degree of flexibility four subsequent years of intensive study. and an open design to meet future After vetting the approach in a number challenges such as reporting on a device- or of markets, Nielsen withdrew its support Cookies: No location-specific basis and accepting input formally in a 2009 paper presented to from new metering devices and new data the Advertising Research Foundation. The Common Standard sources that may be developed. paper cites three primary areas of concern The problem with cookie-dependent about site-centric methodologies: audience measurement is the same • Data is only generated for the problem all Web analytics share: each site Testing Assumptions portion of the market willing to tag captures and cleanses data differently, As a proof of concept for the user-centric making it virtually impossible to establish • A lack of respondent-level data hybrid audience measurement model, any consistent benchmarks to compare Nielsen tested the underlying premise that sources limits the ability to drill one site to another. This proves especially into reports the online behavior of a demographic group problematic for advertisers, whose on one device can predict the behavior of • Demographics are not directly linked expectations are grounded in experience others in the same demographic group on to the cookie-derived audience with reliable, robust syndicated data that another device (assumption #1 above). fosters industry-wide comparability. Only These factors led Nielsen to conclude that a user-centric online hybrid audience By extension, this principle suggests that data from the site-centric hybrid method measurement methodology delivers browsing behavior within the measured has “limited utility for media planning the uniform, industry-wide perspective sample of home and work computers, because it represents a report on values required for critical media buying decisions. when combined with census-level and not actual people.”x behavior from tagged sites, can be used to As a result of cookie-based model understand browsing behavior from other limitations, Nielsen developed an devices and other locations. Assumption alternative, user-centric hybrid approach. Proliferation Additional research reinforced that, Chart 2: Evolution of Hybrid Methodology to achieve a truly accurate audience measurement, any site-centric hybrid online audience model must fulfill three important and distinct assumptions: 1. Site visitation and behavior on one metered device can predict visitation and behavior on another device 2. Site visitation and behavior on one metered device can predict cookie duplication on the same device 3. Site visitation and behavior on one metered device can predict cookie duplication on another device Mathematically, the more assumptions underlying a model, the higher the probability that such a model will fail or generate inaccurate results. Given the site-centric approach dependency on

6 The soundness of this assumption was one individual on one metered device sample were observed to determine tested and authenticated by performing predict the additional cookies that may whether they could predict usage patterns a study with Nielsen measured samples be acquired on additional devices used for the same demographic strata in the using negative binominal distribution to access a site? To test this assumption, Nielsen work sample. Results confirmed [NBD]. The NBD technique offers a usage patterns for the age/sex home the correlation. statistical framework to describe the frequency distribution of behavior, such as Chart 3: One Device/One Location Predictive Power the percentage of homes that purchase a product and the average number of items Visits per Person (Home vs Work) purchased in a period, or the number of 20.0 people exposed to a medium and the distribution of number of exposures to the 17.5 medium in a period. 15.0 The NBD model is very well established as 12.5 a means of estimating media reach and has been validated by several studies over the 10.0 years. NBD enables the user-centric hybrid model to understand the totality of online 7.5 access, accounting even for locations and 5.0 devices other than those directly measured. 2.5 Home and work panelists that visited a Work at Visits per Person selection of tagged sites were grouped 0.0 1 2 3 4 5 6 7 8 9 10 11 12 into location-specific demographic and behavior groups such as: M25-45 + Heavy Visits per Person at Home usage or F25-45 + Light usage. As both Chart 3 and Table 1 show, usage Table 1: Predicting Visits in One Location Based on Another Location patterns of a demographic group with Home/ Dependent Intercept Visits/ F-statistic p-value Error Adjusted one device and location did indeed serve Work Variable person DF R-squared as a significant predictor of the same model at home demographic group’s usage on another device coefficients visits per person (0.07) 1.15 315.89 <.0001 182 0.635 and location. The exceedingly low p-value at work and robust adjusted R-squared coefficient T-statistic visits per person (0.36) 17.77 reinforce the strength of the finding. at work Working from this established principle, p-hyphen- visits per person 0.72 <.0001 Nielsen developed a sophisticated method value at work for analyzing behavior in metered and tag- based data and generating a database that relates each action on a site with a specific individual and their demographic profile. Table 2: Predicting Cookie Duplication in Location Based on Another Location Home/ Dependent Intercept Visits/ F-statistic p-value Error Adjusted Unlike site-centric hybrid approaches that Work Variable person DF R-squared do not link demographics to behavior, model at home the Nielsen user-centric hybrid audience coefficients cookies per (0.12) 1.11 164.2 <.0001 113 0.589 measurement database establishes that person at work direct link and includes all the reporting capabilities of traditional panel-based data. T-statistic cookies per (1.21) 12.81 person at work Having established that user behavior on p-value cookies per 0.23 <.0001 a site can serve as a predictor of cookie person at work duplication on the same machine, one question remains: does the behavior of

7 Chart 4: Visitation and Behavior on One Device (home) is Not Site Uncertainty a Good Predictor of Cookie Duplication on Another Device (work) After confirming the relationship between website usage and behavior across Cookies per Person (Home vs. Work) machines, the next study examined the correlation between cookies per person 3.00 on a work PC and cookies per person for 2.75 demographically similar people on a home PC. The results proved less than convincing. 2.50 In tests, site-centric model assumption 2.00 #3 failed when home device visitation 1.75 and behavior did not predict work device observations as shown in Chart 4. 1.50

This finding calls into question the 1.25 underlying principle of the site-centric hybrid methodology which holds that Work at per Person Cookies 1.00 adjustments based on observations of 0.75 individuals on one online access point can 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4 be applied to the population as a whole. Cookies per Person at Home Acting on this assumption carries the risk that each site may be over- or under- represented in resulting data.

Chart 5: Overcoming a Complex Relationship

VISITS Each VISIT represents the opportunity to place another CT cookie on a user’s machine. PA IM

ONE PERSON, CAUSE FOR MANY COOKIES Do people access my content from multiple devices MULTIPLE COOKIES? over a month?

MANY DEVICES YES

Do people visit my site from different browsers and/or use browser privacy settings?

BROWSER www www www www www CHOICES YES

of people that visit my site, presenting the opportunity to create more cookies? CONTENT DIVERSITY YES

TIME As the period of TIME under measurement increases, the

CT opportunity for more cookies to be placed increases. PA IM

8 Exacerbating the situation further, this study only incorporated one additional Conclusion Future Outlook device. Because users may likely use User-centric hybrid online audience Nielsen’s hybrid audience measurement multiple additional devices to access a site, measurement avoids many serious pitfalls provides a better and more accurate way to each with unique cookie duplication rates, because it is not dependent on cookies. view Internet usage on a monthly basis. As the complexity of adjusting for cookie By relying equally on both panel- and more data sources, such as mobile phone duplication increases significantly, and with census-based behavioral data, the user- metering, become available, Nielsen will it, the risk of inaccuracy within the site- centric approach offers comprehensive, continue to enhance reporting by providing centric hybrid data sources. representative and consistent audience even more detail about how individuals information across websites. By ascribing access the Internet from those devices. behavior directly to measured panelists, this The hybrid process has been intentionally Confounding Effects model bypasses cookie-related issues such designed to allow for improvements, as double-counting or legal restrictions including the potential for additional Individually, each assumption supporting on setting or reading cookies and readily insights from metering on non-PC devices, site-centric hybrid audience calculations accommodates future expansion as improved panel representation and more. may be suitable for audience estimates, but additional data sets come online. As always, Nielsen strives to refine its the complexity of the relationship between strategy for maintaining the highest quality Measurement tags will continue to play those variables in real-world situations Internet measurement data in each of the a supporting role by providing useful undoubtedly introduces interaction effects markets served. that can skew findings. transaction data. This technology affords tremendous latitude for advertisers to Moreover, site-centric methodologies dynamically customize online content About Nielsen fall short in satisfying the demands of and serve-up relevant advertising. But the marketers and media owners interested liabilities associated with cookie counting Nielsen Holdings N.V. (NYSE: NLSN) is in quality online audience data on a offset its usefulness as a primary audience a global information and measurement number of critical criteria. First, the data measurement tool. company with leading market positions does not present a consistent view of the in marketing and consumer information, marketplace because tagged sites are By contrast, in addition to more robust, television and other media measurement, treated differently from untagged ones. reliable data sets, the user-centric hybrid online intelligence, mobile measure- VISITS methodology was purposely built to Each VISIT represents the opportunity to place another Second, the data is not derived at the ment, trade shows and related properties. CT cookie on a user’s machine. individual respondent level, restricting accept more data sources, provide more Nielsen has a presence in approximately PA

IM both demographic analysis and drill- detail on access points and devices, allow 100 countries, with headquarters in New ONE PERSON, down reporting capabilities. Third, relying improvements such as additional insights York, USA and Diemen, the Netherlands. CAUSE FOR from non-PC device meters and expand MANY COOKIES Do people access my content from multiple devices MULTIPLE COOKIES? on cookies as the independent variable For more information, visit over a month? represents a challenge to data accuracy panel representation. www.nielsen.com. MANY and consistency on a site-to-site and DEVICES YES Nielsen remains committed to exploring, month-to-month basis. expanding and examining online audience measurement methodologies, in turn Do people visit my site from different browsers and/or use browser privacy settings? enabling clients to craft more efficient and

BROWSER www www www www www effective online strategies. CHOICES YES

of people that visit my site, presenting the opportunity to create more cookies? CONTENT i A Guide to Understanding Online Measurement Alternatives (A Media Rating Council Staff Point of View); MRC Staff, August 3, 2007, p9 DIVERSITY YES ii Ibid. iii Cookie Deletion Survey Results, Harmon (Belden Associates) , 2004 iv Measuring Unique Visitors: Addressing the Dramatic Decline in the Accuracy of Cookie-Based Measurement, Petersen (Jupiter Media), 2005 TIME v Is the Sky Falling on Cookies? Understanding the Impact of Cookie Deletion on Web Metrics, Song (Atlas Digital Marketing), 2005 As the period of TIME under measurement increases, the

CT opportunity for more cookies to be placed increases. vi The Impact of Cookie Deletion on Accuracy of Site-Server and Ad-Server Metrics: An Empirical Study, Abraham et. al. (comScore) 2007

PA vii An Analysis of Private Browsing Modes in Modern Browsers, Aggarwal, et al. (Stanford), 2010

IM viii Interactive Advertising Bureau Audience Reach Measurement Guidelines, Version 1.0, IAB, February 23, 2009 ix Ibid. x Hybrid: Innovation in Panel and Census Integration, Duterque, Stackhouse and Mazumdar (Nielsen), 2009

For more information visit www.nielsen.com

Copyright © 2011 The Nielsen Company. All rights reserved. Printed in the USA. Nielsen and the Nielsen logo are trademarks or registered trademarks of CZT/ACN Trademarks, LLC. Other product and service names are trademarks or registered trademarks of their respective companies. 11/3600