ESOMAR 28QUESTIONS ESOMAR 28 QUESTIONS

SAMPLE SOURCES & RECRUITMENT

What experience does your company Please describe and explain the type(s) 01 have in providing online samples for 02 of online sources from which you ? get respondents. Are these databases? Actively managed research panels? Direct Context: This answer might help you marketing lists? Social networks? Web to form an opinion about the relevant intercept (also known as river) samples? experience of the sample provider. How long has the sample provider Context: The description of the types of been providing this service and do they sources a provider uses for delivering an have for example a market research, online sample will provide insight into the direct marketing or more technological quality of the sample. background? Are the samples solely provided for third party research, or does PaidViewpoint is a double-opt managed the company also conduct proprietary panel, built by referral only. We don’t work using their panels? recruit through any affiliates, websites, social networks, etc. Aytm is a quantitative market research automation platform that incorporates Our experience has shown that the programming, fielding, and real-time sample source plays a vital role in execution of complex, sophisticated the health and the quality of the panel. research tests including max-diff, We strategically focus on respondents’ choice-based conjoint, automated experience. By keeping our panel exclu- TURF, pricing optimization, and more. sive to aytm, we ensure that PaidView- PaidViewpoint.com is a proprietary point panelists are treated fairly. We panel, designed, built and managed believe that people who are happy with by aytm since 2011. Our panel doesn’t their panel and experiences would participate in any external marketplac- refer their friends and families to join. es; it’s exclusively available to aytm clients only. The only way to access our proprietary panel is by launching surveys through our survey platform tools.

Aytm has panels in 34 countries in North America, Europe, Latin America, Asia-Pacific, Africa, and the Middle East. All countries’ panels adhere to the same online research industry principles and panel management guidelines, though some differences in regional approaches exist. Rules and privacy policies may vary to reflect the needs in the respective countries. ESOMAR 28 QUESTIONS

If you provide samples from more How do you source groups that may 03 than one source: How are the different 05 be hard to reach on the internet? sample sources blended together to ensure validity? How can this be repli- Context: Ensuring the inclusion cated over time to provide reliability? of hard-to-reach groups on the internet How do you deal with the possibility (like ethnic minority groups, young people, of duplication of respondents across seniors etc.) may increase population sources? coverage and improve the quality of the sample provided. Context: The variation in data coming from different sources has been well We ensure extensive sample coverage documented. Overlap between different through our comprehensive panel panel providers can be significant in some profiling and our API integrations with cases and deduplication removes this preferred panel partners. source of error, and frustration for respondents. In order to guarantee detailed knowledge of our respondents’ unique traits, Paid- All PaidViewpoint panelists are sourced Viewpoint panelists are extensively via one method, referral. Each person profiled, to enable relevant and speedy may only be registered one time. De- sample selection. Our meticulous and up duplication is assured by a unique to date profiling cover phone number, PayPal account, and 3,000+ psychographic & personality traits digital fingerprinting. that can be utilized beyond superior survey feasibility. PaidViewpoint attri- butes categories include purchasing Are your sample source(s) used solely habits, food & beverage, media, automo- 04 for market research? If not, what other tive, gadgets & devices, money manage- purposes are they used for? ment, green life, home & family, health & wellness, personal care, hobbies & inter- Context: Combining respondents from ests, sports, and travel. sources set up primarily for different purposes (like direct marketing for example) may cause undesirable survey effects.

PaidViewpoint panel is used solely for market research purposes. ESOMAR 28 QUESTIONS

If, on a particular project, you need to Occasionally, we need to supplement 06 supplement your sample(s) with sam- our PaidViewpoint sample with a sample ple(s) from other providers, how do you provided by preferred panel partners via select those partners? Is it your policy API integration or on an ad-hoc basis. to notify a client in advance when using All new panel partners undergo a strict a third party provider? vetting process through which we identify companies that adhere to quality and Context: Many providers work with third respondent management processes parties. This means that the quality of similar to our own. We run a test project the sample is also dependent on the and validate their sample data by compar- quality of sample providers that the buyer ing them to our own panel data and other did not select. Transparency is essential established panel partners. Panels select- in this situation. Overlap between different ed as preferred partners adhere to strict providers can be significant in some guidelines on each project commissioned. cases and de-duplication removes this We capture our panel partners’ qualitative source of error, and frustration for respon- & quantitative performances and review dents. Providers who observe process them monthly. standards like the ISO standards are required to give you this information. We ensure transparency for our clients by always informing them, should an external panel partner be required to complete the survey. This is done after a feasibility check for the corresponding project is completed, or at a later stage, if due to lower than anticipated incidence rate, an additional sample source may need to be added. ESOMAR 28 QUESTIONS

SAMPLING & PROJECT MANAGEMENT

What steps do you take to achieve Do you employ a survey router? 07 a representative sample of the target 08 population? Context: A survey router is a software system that allocates willing respondents Context: The processes to surveys for which they are likely to (ex. how individuals are selected or qualify. Respondents will have been allocated from the sample sources?) directed to the router for different reasons, used are the main factor in sample perhaps after not qualifying for another provision. A systematic approach based survey in which they had been directly on market research fundamentals may invited to participate, or maybe as a result increase sample quality. of a general invitation from the router itself. There is no consensus at present Every survey is unique – we tailor our about whether and how the use of a router sampling to our clients’ specifications affects the responses that individuals and requirements. We have the ability give to survey questions. to automatically target demographic specifications based on country- No, we don’t utilize survey routers. specific data (nested and non-nested) or based on custom speci- If you use a router: Please describe fications. For specific sub-populations 09 the allocation process within your router. (e.g., primary grocery shoppers, electric How do you decide which surveys might car owners, etc.), we can generate be considered for a respondent? On what representative quotas based on a priority basis are respondents allocated sub-sample of our panel, and then to surveys? use these quotas to obtain a balanced sample of the target population. Our Context: Biases of varying severity may proprietary sampling system allows arise from the prioritization in choices of us to construct complex samples based surveys to present to respondents and the on screening targets and demographic method of allocation requirements. The sampling algorithm randomly selects potential respondents N/A according to the targets and balances the sample based on demographic If you use a router: What measures do criteria. 10 you take to guard against, or mitigate, any bias arising from employing a router? How do you measure and report any bias?

Context: If Person A is allocated to Survey X on the basis of some characteristics then they may not be allowed to also do Survey Y. The sample for Survey Y is potentially biased by the absence of people like Person A.

N/A ESOMAR 28 QUESTIONS

If you use a router: Who in your company What profiling data is held on respon- 11 sets the parameters of the router? Is it 12 dents? How is it done? How does this a dedicated team or individual project differ across sample sources? How is it managers? kept up-to-date? If no relevant profiling data is held, how are low incidence Context: It may be necessary to try to projects dealt with? replicate your project in the future with as many of the parameters as possible set to Context: The usefulness to your project the same values. How difficult or easy will of pre-profiled information will depend on this be? the precise question asked and may also depend on when it was asked. If real time N/A profiling is used, what control do you have over what question is actually asked?

To join the PaidViewpoint panel, all panelists must provide the following information at registration:

• Name • Email address • Address • Gender • Date of birth • Ethnicity (US only) • Relationship status • Education • Household income • Employment status • Career • Age and gender of children under 18 in the household

Respondents are periodically asked (every 3 months) to review and update their registration information as needed. PaidViewpoint panelists are also addi- tionally profiled on more than 3,000 consumer data points and a variety of psychographic & personality attributes. Attribute categories include Purchasing Habits, Food & Beverage, Gadgets and Devices, Media, Automotive, Money Management, Green Life, Home & Family, Health & Wellness, Personal Care, Hob- bies & Interests, Sports, and Travel. This information is gathered through desig- nated short “trait surveys” and updated regularly as well. Are your sample source(s) used solely for market research? If not, what other purposes are they used for?

Context: Combining respondents from sources set up primarily for different purposes (like direct marketing for example) may cause undesirable survey effects.

PaidViewpoint panel is used solely for market research purposes.

ESOMAR 28 QUESTIONS

Please describe your survey invitation Please describe the incentives that 13 process. What is the proposition that 14 respondents are offered for taking part people are offered to take part in individ- in your surveys. How does this differ ual surveys? What information about by sample source, by length, the project itself is given in the process? by respondent characteristics? Apart from direct invitations to specific surveys (or to a router), what other Context: The reward or incentive system means of invitation to surveys are may impact on the reasons why people respondents exposed to? You should participate in a specific project and these note that not all invitations to participate effects can cause bias to the sample. take the form of emails. All panelists receive a cash incentive Context: The type of proposition (and for responding to a survey, even if they associated rewards) could influence the don’t qualify. The incentive for survey type of people who agree to take part in completion varies depending on the specific projects and can therefore influ- country, survey length, and complexity. ence sample quality. The level of detail Panelists also have the option to given about the project may also influence redeem their incentives for eGift cards. response. All online (includ- ing surveys) is conducted via websites. Through the sampling process, panelists What information about a project do are pre-selected to answer a certain survey; 15 you need in order to give an accurate the surveys are not “open access” (ex. estimate of feasibility using your own respondents are not self-selected), and resources? respondents do not know the survey content. Context: The “size” of any panel or source may not necessarily be an accu- Based on their preferences, PaidView- rate indicator that your specific project point panelists receive an invitation can be completed or completed within email, SMS, automated phone call, or your desired time frame. push notification to inform them they are invited to a new survey. We provide We automatically estimate feasibility a unique URL to access the survey they based on the following information: the are invited to, information about the overall number of completes required, incentive amount they will receive for demographic specifications and quotas, responding and for completing the other survey qualification criteria, and survey, member support email link, link additional quotas based on survey to PaidViewpoint website, opt-out infor- responses, incidence rate, and estimat- mation and physical address for aytm. ed survey length. We also offer a guar- Information about the survey topic is not anteed day and time of data collection included in any manner. Also, panelists completion. who happen to be on their dashboards when a new survey launches and match the survey demographic criteria may see the survey available for them to take. ESOMAR 28 QUESTIONS

Do you measure respondent satisfac- 17. What information do you provide to 16 tion? Is this information made available 17 debrief your client after the project has to clients? finished?

Context: Respondent satisfaction may Context: One should expect a full sample be an indicator of willingness to take future provider debrief report, including gross surveys. Respondent reactions to your sample, start rate, participation rate, survey from self-reported feedback or from drop-out rate, the invitation/contact text, an analysis of suspend points might be a description of the field work process, and very valuable to help understand survey so on. Sample providers should be able to results. list the standard reports and metrics that they make available. We utilize several approaches to monitor- ing respondent satisfaction. Aytm provides fielding time, termination report, incidence rate, and average Periodically, we launch a panel satisfac- survey length. We also provide a sample tion survey that focuses on panelists’ of a PV survey invite, if requested by the overall experience and measures their client. All clients have access to the data satisfaction with the completed surveys in real time via their aytm account dash- and the rewarded incentives. Data is board and/or a direct statistics page link evaluated and translated into panel and that we provide when confirming the soft survey instrument improvements. launch data check is complete, and we’re fully launching. We also communicate with our panelists through our member support team, social media, and the independent panel review site Survey Police, where Paid- Viewpoint is ranked #1 year over year by survey respondents. Panelists’ feedback triggers improvements and helps main- tain a continuous focus on a quality experience.

We also monitor completion rates, which are an indirect measure of respondents’ satisfaction. Completion rates are moni- tored at the individual survey level and at the overall monthly level, to identify potential issues or fluctuations. ESOMAR 28 QUESTIONS

18. Who is responsible for data quality How often can the same individual be 18 checks? If it is you, do you have proce- 19 contacted to take part in a survey within dures to reduce or eliminate undesired a specified period whether they respond within survey behaviours, such as (a) to the contact or not? How does this vary random responding, (b) Illogical or incon- across your sample sources? sistent responding, (c) overuse of item non-response (ex. “Don’t Know”) or (d) Context: Over solicitation may have an speeding (too rapid survey completion)? impact on respondent engagement or on Please describe these procedures. self selection and non-response bias.

Context: The use of such procedures may We have strict limits for invitations sent increase the reliability and validity of survey to each panelist every 30 days. On data. average, our panelists receive two to three invitations per week. However, that Aytm panel operations, project manage- number can vary based on profile char- ment, and research teams work closely acteristics and surveys in field. together to implement effective data quality controls and ensure panelists’ survey engagement.

In order to verify our panelists are real and attentive, we employ several quality control mechanisms at the panel registra- tion level (described in question #22 below) - technical safeguards, lie detec- tion and a system of random checks and automatic red flag notifications for when a specific panelist’s behavior seems out of the norm.

We also take a multi-faceted approach to data cleaning. Our research and project managers’ teams evaluate respondents’ length of interview (i.e., identify “speeders” and in some cases “laggers”), responses to grid questions (i.e., “straight-lining” or repeatedly selecting the same answer options), responses to open-ended ques- tions and quality control questions (also called “red herring” or “trap” questions). In addition, our research team also reviews and analyzes the data, allowing them to detect any inconsistencies or illogical data patterns.

ESOMAR 28 QUESTIONS

How often can the same individual Do you maintain individual level data such 20 take part in a survey within a specified 21 as recent participation history, date of period? How does this vary across your entry, source, etc. on your survey respon- sample sources? How do you manage dents? Are you able to supply your client this within categories and/or time with a project analysis of such individual periods? level data?

Context: Frequency of survey participation Context: This type of data per respondent may increase the risk of undesirable including how the total population is defined conditioning effects or other potential and how the sample was selected and biases. drawn, may increase the possibilities for analysis of data quality. We also have strict rules for the number of survey invites each panelist receives We maintain a database that records per day, per five days, and per month. all details of respondents’ panel activity: We allow respondents to “participate” by their recruitment date and tenure with responding to all survey invitations they the panel, their complete demographic have been sent; however, it is highly profile, as well as survey participation unlikely for someone to qualify and history. We can provide non-personally complete every survey they are invited identifiable information upon client to. Additionally, surveys typically expire request. within a few days. ESOMAR 28 QUESTIONS

Do you have a confirmation of respon- We also remove duplicates and verify 22 dent identity procedure? Do you have all IDs. During registration, we go way procedures to detect fraudulent respon- beyond double opt-in. We ask people dents? Please describe these proce- (interested in becoming members of our dures as they are implemented at US panel) to enter and verify a unique sample source registration and/or at the phone number that is not associated point of entry to a survey or router. If you with any existing account and belongs offer B2B samples what are their proce- to the same country/region specified by dures, if any? the respondent. They also are required to provide a valid PayPal account to be Context: Confirmation of identity can able to claim their incentive. The same increase quality by decreasing multiple phone and PayPal will be required for entries, fraudulent panelists etc. a second verification before they can cash out. (This happens several months We have several layers of panelist valida- after signing up so they can’t be easily tion: borrowed). We automatically detect and block IP numbers and use PayPal’s API Layer 1, Technical: A number of purely to link different email aliases to validate technical safeguards are in place to that the account is linked to a unique, identify and prevent bots from taking valid bank checking account. client surveys. Such methods include a wide variety of approaches, ranging Aytm has also developed a unique from IP/cookie verifications, log in in-house de-duplication solution that patterns of different connected goes beyond our proprietary panel. It’s accounts, reCAPTCHA, honeypots to powered by Graphs and capable of uniqueness and validity verification of linking millions of objects together in a phone number, and a PayPal account. real time to identify and prevent situa- tions when a suspicious number of Layer 2, Lie detection: Our panel is such objects as IP addresses, session managed by TrustScoreSM – an auto- cookies, surveys, and UUIDs are linked matic system of consistency verification. together. For example, it becomes very It’s easy for a human to stay consistent apparent for the algorithm and super- in their profiling questions when they are vising QA managers when many telling the truth, but it’s virtually impossi- attempts to take the same survey came ble for a bot to understand questions, from the same IP address or the same formulated in a slightly different manner device (regardless of which panel (e.g., Do you own a dog? Do you own respondent it’s coming from). a pet?) and answer consistently. Additionally, we use a third-party Layer 3, Economic & Manual: If service called IMPERIUM / Verity someone successfully bypasses all our to check respondents’ name and US safeguards (highly unlikely), it still takes mailing address that we ask for at the many weeks to achieve our cash out registration against their database. threshold and then requires a cash out Based on the response, we can adjust specialist to review and approve the the TrustScoreSM (our level of trust request. We have a system of random toward the user) and, ultimately, their checks and automatic red flag notifica- chances of receiving a client survey. tions when something seems out of the norm. ESOMAR 28 QUESTIONS

POLICIES AND COMPLIANCE

Please describe the ‘opt-in for market Please provide a link to your Privacy 23 research’ processes for all your online 24 Policy. How is your Privacy Policy provid- sample sources. ed to your respondents?

Context: The opt-in process indicates the Context: Not complying with local and respondents’ relationship with the sample international privacy laws might mean the source provider. The market generally makes sample provider is operating illegally. An a distinction between single and double example privacy policy is given in the opt-in. Double opt-in refers to the process ESOMAR Guideline for Online Research. by which a check is made to confirm that the person joining a panel or database wishes Our extensive Privacy Policy can be to be a member and understands what to reviewed at our panel site: https://paid- expect (in advance of participating in an viewpoint.com/pages/privacy. All panel- actual survey for a paying client). ists are required to review and agree to the privacy policy at the time of panel registra- PaidViewpoint uses a “double opt-in” tion. In addition, the introduction to our process for all panelists. Individuals who standard survey invitation includes a link wish to join our panel complete the online to the policy and reminds panelists that registration survey and accept the terms we hold all information provided to us in and conditions of membership. We inform the strictest of confidence. our panelists that their information will be kept in the strictest confidence according We comply with the standards of MRIA, to all market research standards. Likewise, MRA, and CASRO in North America, and panelists are requested to adhere to strict with ESOMAR, AIMRI and EFAMRO stan- confidentiality of the content of our dards in Europe. surveys. Agreement to these terms and conditions completes the first step in the “opt-in” to panel membership.

Upon completion of the registration ques- tionnaire, panelists receive an email and are asked to click on a link to confirm they would like to participate in panel member- ship. Once they click on the link, they are redirected back to our website, and the second step of the “opt-in” process is concluded. ESOMAR 28 QUESTIONS

Please describe the measures you take What practices do you follow to decide 25 to ensure data protection and data 26 whether online research should be used security. to present commercially sensitive client data or materials to survey respondents? Context: The sample provider usually stores sensitive and confidential informa- Context: There are no foolproof methods tion on panelists and clients in databases. for protecting audio, video, still images or These data need to be properly secured concept descriptions in online surveys. In and backed-up, as does any confidential today’s social media world, clients should information provided by the client. The be aware that the combination of technology sample provider should be able to provide solutions and respondent confidentiality you with the latest date at which their agreements are “speed bumps” that mitigate security has been evaluated by a credible but cannot guarantee that a client’s stimuli third-party. will not be shared or described in social media. We have implemented strict security measures to protect the loss, misuse, Our terms of service enforce strict guide- and alteration of any panelists’ and client lines regarding the exposure of “Restrict- information under our control. For exam- ed Content” (confidential information of ple, the access to clients’ accounts and our clients). Such information may include the transfer of all personal information but is not limited to brand new product collected by us is done over https, we and marketing ideas or materials, as well limit access to the information only to as various other proprietary information. individuals who need the information to Panelists are required to agree to protect perform their duties, and we use reliable the confidentiality and secrecy of the third-party providers to back up personal Restricted Content. They are also asked information. We are also ISO-27001 to agree not to use, modify, copy, repro- certified. duce, re-publish, display, transmit, or share Restricted Content. Panelists are also required to acknowledge and agree that Restricted Content may be subject to and protected by intellectual property laws and regulations.

In addition to our terms of use, we have long-term and short-term plans in place for protecting our stimuli. The short-term includes serving a placeholder image whenever the participant is trying to save our images from the page. A long-term solution includes incorporating water- marks into all images so that we can associate a leak with a particular survey. ESOMAR 28 QUESTIONS

Are you certified to any specific quality Do you conduct online surveys with 27 system? If so, which one(s)? 28 children and young people? If so, do you adhere to the standards that ESOMAR Context: Being certified may require the provides? What other rules or standards, supplier to perform tasks in a predeter- for example COPPA in the United States, mined manner and document procedures do you comply with? that should be followed. Context: The ICC/ESOMAR International We are one of the very few market Code requires special permissions for research platforms certified and annually interviewing children. These are described audited according to ISO 27001 certifica- in the ESOMAR Online Research Guideline. tion. The certification is for the informa- In the USA researchers must adhere to the tion security management system requirements of the Children’s Online Privacy (ISMS) supporting aytm’s survey Act (COPPA). Further information on legisla- platform, as well as our proprietary panel tion and codes of practice can be found in PaidViewpoint, to ensure the utmost Section 6 of ESOMAR’s Guideline for Online quality of data security for market Research. researchers and survey respondents. ISO 27001 is one of the most widely recog- Compliance with the Children’s Online nized, internationally accepted indepen- Privacy Protection Act (COPPA) is guaran- dent security standards. teed because we do not store personal information from children, and surveying We established a formal, annual, of panelists’ children is only done through program to continue being audited and advocacy. certified to this ISO standard. This program will consistently be performed Explicit parental consent for inviting by an independent, third-party auditor. children (age of 12 and under) in the Aytm’s compliance with the ISO standard surveys is mandatory. Parental status is was certified by A-LIGN, an ISO certifica- defined by legislation as a parent, or other tion body accredited by the ANSI-ASQ legal guardian. Our process includes National Accreditation Board (ANAB). documentation and evidence of consent for gathering data and interviewing children. aytm.com [email protected] +1 (415) 364-8601 3000 Atrium Way Ste 260 Mt. Laurel NJ 08054