International Statistical Institute, 55th Session 2005 Inference from Non-Probability Samples in Marketing Research Bill Blyth TNS plc Westgate, London, W5 1UA, UK [email protected] 1. Introduction This paper discusses the manner in which marketing research sample designers have evolved robust methods for producing reliable results from which inference can confidently be made. It draws on the structure proposed by Groves (1) to argue that market researchers can provide academic survey statisticians with a different and valuable perspective to their own about survey design and stakeholder value. 2. Marketing research today Annual industry data for 2003 (2) estimates the global expenditure on marketing research at $ 18.9 billion. 38% of this was in North America, 44% in Europe and 14% in Asia Pacific. The latter area provided the fastest annual growth with China growing at 28% and India at 17%. Contrary to popular belief very little marketing research consists of either street interviewing or opinion polling. The greatest proportion of activity is in the area of continuous measurement. Market size measurement, be it via retailers or consumers, or audience measurement is the single largest area of activity. This and the majority of other work undertaken on consumer behaviour are invariably continuous, in the sense of being repeated through time. The movement towards evidence based policy and the need for performance indicator measurement has particularly spurred expenditure by the public sector. Our best estimate is that 7% or $1.3 billion dollars came from the public sector excluding utilities. The largest agencies span the world with revenues of several billions of dollars and workforces many thousand strong. In recent years consolidation has been rapid and less than ten, mainly publicly quoted, companies account for more than half of global revenues. Consolidation has produced increased standardisation between and within countries. Whilst market research agencies still present a wide range of expertise and resource, competition and standardisation has increasingly undermined poor quality suppliers. The comments in this paper derive from the practices of these larger successful agencies, rather than fringe or ‘rogue’ elements. Standards of staff qualifications, training and fieldwork are high. The draft of an ISO quality standard specifically for market research has recently been published for consultation. It has been drawn up by a global working party with representation from the private sector, official statistics organisations and academia. 3. Sampling and survey design considerations Marketing research survey designers carry out a complex trade-off between speed, accuracy and cost to provide ’research value.’ Tender deadlines and budgets are tight and are an important part of the evaluative mix and the design consideration. I prefer to use the description of survey designer, rather than statistician. This is because the question of sample design is inextricably entwined with other aspects of survey design, not least the method of data collection. Each country has evolved a preferred method or restricted set of methods International Statistical Institute, 55th Session 2005 for the conduct of quantitative studies. These are influenced by local geography, population characteristics, relative costs, labour force availability and characteristics, sampling frames, technological infrastructure and GDP per capita. The timing of the local ‘take-off’ of marketing research is also crucial. In the USA market research moved rapidly away from personal interviewing first to postal research, then to telephone and recently is in the forefront of the use of the Internet. Europe on the other hand invariably has regarded mail surveys negatively and it was not until the late 1980’s or 1990’s that telephone surveys became acceptable for general populations of interest (3). Personal in-home interviewing is preferred in many countries as is the case in Africa, Asia and S America. The early move to the web in the USA has been decelerating. Web take-up elsewhere in the world has been slow other than in a few northern Europe countries and a couple of Asian countries with intensive Internet use. The consequence of this local methodological evolution was a variety of methods and approaches that have converged over time. A general complaint- Kish (4) -is that such methods are not written up. Reported experimental work on non-probability methods is scanty. Moser and Stuart (5) and Marsh and Scarborough (6) being rare exceptions and somewhat dated. However, there are few reasons why commercial researchers should want to write up their work. Increasingly when they carry out experimental work it is unpublished. The reason why this omission occurs is straightforward competitive advantage. Without academic career pressure to publish there is no desire in the face of weak or unenforceable IPR legislation to tell the world how one makes a better widget! The free movement of staff and ideas makes it very difficult to protect the fruits of what are increasingly expensive investments from which shareholders expect a measurable return. This desire for confidentiality is evidenced by the Black Box nature of much of the discussion elsewhere about propensity weighting systems. What market researchers will agree on is that their methods are not generally probability methods. That has not been and is not always the case. Most countries have several major commercial surveys using probability methods or carry out public sector surveys that use such methods. However, whilst marketing researchers do know how to conduct random surveys and, when required, can do so adequately, they or their clients choose not to. Why is that? 4. Inferential structure Groves summarises a number of constructs that describe differing approaches of survey researchers. To paraphrase: Describers v modellers Sampling error persons v non-sampling error persons Error measurers v error reducers To this one might add: Survey topic specialists v generalists Stand- alone v repeat Whilst he states they are not mutually exclusive the marketing researcher and the academic sampling statistician find themselves more often than not at different ends of these scales. Marketing researchers are non-sampling error; we are reducers; we are generalists; we are trackers; we are describers and modellers and increasingly use modelling to describe. Much of the literature is written as if a survey takes place in a vacuum, or is totally removed from other surveys about different subjects. Relevant administrative data rarely makes an appearance and discussion is around theoretical or actual single parameter estimates. In literature real time appears to stand still as consideration is given to the matters at hand. This state of affairs is International Statistical Institute, 55th Session 2005 not the commercial reality where on-time delivery is of prime importance. In the majority of commercial quantitative studies the survey is invariably repeated at some time interval- be it a month, a quarter, a year, bi-ennially or longer. Others surveys also will have previously asked the majority of the same questions. We are concerned with datasets and inter-relationships, with overall conclusions, with time-dependent decision contexts, with cost and resource constraints. We are concerned with checking results against a wide range of prior data, an increasing amount of which is comprehensive administrative data of some sort emanating from the client. Inference occurs within this context- it is a complex and professional task. Much of this approach is, of course, shared with survey statisticians in public sector organisations where the pressures on delivery and accuracy of trend are much the same. However, there is still a marked divide between the sectors in the use of probability and non-probability methods. I would argue that where the primary concern is to track change through time, then the use of non-probability methods provides greater control of sources of variability and bias. Whilst theoretically random methods may be better, their application is not justified unless response is so high and unbiased as to be ignorable. This unfortunately is rarely the case even in much public sector research. Within marketing research organisations, the production process is increasingly mechanical and process driven. Survey after survey goes through ‘the mill’ using the same sampling, interviewers, coding teams, CAPI equipment, weighting, back-checking etc. It is easy, particularly within the context of integrated databases, for the experienced researcher to be able to generalise aspects of the error and bias structures that exist from one survey to another. In many countries larger organisations are equal in size or larger than many public sector organisations carrying many hundreds of surveys and many hundreds of thousands of interviews each year. In parallel, the large amount of material, available around the world, collected in the same manner on the same application enables the researcher to develop relevant knowledge about data relationships. An example of this is the ‘Theory of Repeat Buying Behaviour’ developed by Ehrenberg (7). The theory establishes empirical generalisations about patterns of buying behaviour and then proceeds to model these within a coherent theory. It has been validated across a wide range of markets and choice behaviours around the world. Other law-like relationships have been derived in areas