Esomar/Wapor Guide to Opinion Polls and Published Surveys

Home , Exit poll, Opinion poll, Questionnaire, Survey (human research)

ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

All ESOMAR world research codes and guidelines, including latest updates, are available online at www.esomar.org

© 2009 ESOMAR/WAPOR. All rights reserved. No part of this publication may be reproduced or copied in any form or by any means, or translated, without the prior permission in writing of ESOMAR or WAPOR.

The ESOMAR/WAPOR Guide to Opinion Polls and Published Surveys was drafted in English and the English text is the definitive version.

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

CONTENTS

Introduction to the guide 2

1. Opinion polls and democracy – an overview 3 q q f a 2. Frequently asked questions about polls 4 f a

3. Guideline for the publication of opinion poll and survey results 14 3.1 Introduction 14 3.2 Scope 16 3.3 Requirements of the ICC/ESOMAR International Code 16 3.3.1 General requirements 16 3.3.2 Specific requirements for published public opinion surveys 17

3.3.3 Arrangements between the Research f or researchers and commissioners Organisation and its Client 21

4. Specific guidance on conducting pre-election opinion polls and polls on voting intention 24 4.1 Introduction 24 4.2 The Guidelines 25 4.2.1 Timing of fieldwork 25 4.2.2 Sample size 25 4.2.3 Sample distribution 26 4.2.4 Telephone interviewing 27 4.2.5 Online polls 27 4.2.6 Weighting 28 4.2.7 Adjustments 28 4.2.8 Survey content 29 4.2.9 Time series 29

4.2.10 Consistent design 29 f or researchers 4.2.11 Indicator variables 29 4.2.12 Unusual poll results 30

5. Guideline on conducting exit polls and election result forecasts 31 5.1. Introduction 31 5.2. Specific requirements for exit polls 31 5.2.1. Items for minimal disclosure 32 5.2.2. Election projection methods and their disclosure 34

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Introduction to the guide and published according to the ESOMAR Guideline for the Publica- Public opinion polls are regularly con- tion of Opinion Poll Results. The ICC/ ducted and published in many coun- ESOMAR International Code and this tries. They measure not only support associated Guideline are observed by for political parties and candidates, but members of ESOMAR working in more also public opinion on a wide range than 100 countries around the world. of social and political issues. They are published frequently by a variety of Second there is a section on frequently print, online and broadcast media. asked questions about how to judge the quality of an opinion poll. These The public discussion of opinion polls questions are designed to help journal- is not always well informed. However, ists, politicians, academics, the public the case for restricting the publication and other interested parties understand of polls during election campaigns is how to judge the quality of an opinion hard to support with rational argument poll. or empirical evidence. ESOMAR and WAPOR have produced the present Third there is the ESOMAR Guideline guide in order to help those interested for the Publication of Opinion Poll and in the subject of opinion polls to reach Survey Results which is intended to a more informed judgement about reduce the risk of the public being the value of such polls and the most misled by research which is inadequate appropriate ways of conducting and or badly presented or interpreted. reporting them. Fourth there is a section which pro- There are five sections in this guide. vides specific guidance for practitioners on the conduct of pre-election polls. First there is a statement on the role of opinion polls in democratic systems. Finally, there is a Guideline for the ESOMAR and WAPOR’s position on Conduct of Exit Polls and Election Re- this issue is quite clear. We believe that sults Forecasts. These last two guide- there should be no restriction on the lines are designed to help ensure that conduct or publication of opinion polls election polls are carried out to high which have been carried out according standards.They provide technical guid- to the ICC/ESOMAR International ance to researchers and background Code on Market and Social Research information but they are not intended to

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Why opinion polls are essential for democracy 3

be a “How to do it” manual. The subject of the results of opinion polls tell read- of public opinion research and the ers and listeners that their opinions are measurement of voting intention con- important, and can even sometimes be tinue to evolve and each election may more important than the opinions of the bring new circumstances the research- elite. er must address. The professional skills and previous experience of polling The democratic urge towards participa- organisations are essential components tion and the journalistic desire to ask of effective public opinion research. It is questions have merged to create the not possible to write them into guide- extensive media polling of the last 70 lines or codes of practice. years. Imagine a political system where the public is told what it thinks by its 1. Opinion polls and political leaders, where election win- democracy – an overview ners have the ability to tell voters why they voted the way they did and where Public opinion is a critical force in the government, when it looks for pub- shaping and transforming society. lic input, asks only its friends what Properly conducted and disseminated the public thinks. The alternative to survey research gives the general properly conducted polls is a public and public an opportunity for its voice to be a government exposed only to unscien- heard. Through opinion research the tific and probably inaccurate assertions public, politicians, the media and other about what people believe, in many interested groups have access to cases presented by partisan individuals accurate measures of public attitudes or organisations with a political agenda. and intentions.

“Scientific” polling is among the most successful political developments of the last century. Public opinion polls help guide policy by giving decision-makers impartial information about what the public wants. Polls also alert the public to their own hopes, desires, and political goals. They are mirrors, permitting individuals to understand where they fit into the political system. Media reports

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

4 Understanding opinion polls

2. Frequently asked Internet polls); or visits randomly- questions about polls drawn addresses or names from a list such as an electoral register (for some 1. What is an opinion poll? face-to-face surveys). The polling An opinion poll is a scientific and repre- organisation then contacts people on sentative survey designed to measure those telephone numbers or at those the views of a specific group – for addresses, using a random selec- example a country’s electors (for most tion procedure, and asks them to take political polls) or parents or trade union part in the survey. “Quota” sampling members. involves setting quota controls – for example, age and gender – and letting the 2. What makes a survey “scientific”? interviewer seek out different people The two main characteristics of sci- who, together, match those char- entific surveys are a) that respondents acteristics. Surveys based on quota are chosen by the research organisa- sampling are often used in face-to-face tion according to explicit criteria to surveys. In addition, some Internet polls ensure representativeness, rather than employ quota sampling to select repre- being self-selected, and b) that ques- sentative samples from a database of tions are worded in a balanced way. people who have already provided such For example, if the population being information about themselves. Quota sampled contains 52% who are women sampling may be used, in otherwise and 30% who are over 55, then a sci- randomly sampled telephone surveys, entific opinion poll will represent those to select the person to be interviewed groups appropriately and the ques- within the household, in order to speed tions will be balanced and not lead the up the fieldwork process. respondent towards a particular answer. 4. Do polling companies do anything else to achieve representative 3. How does a poll choose a sample samples? that is truly representative? Yes, they do. While well-conducted There are two main methods. The first random and quota samples provide is “random” sampling, the second “quo- a broad approximation to the public, ta sampling”. With random sampling, a there are several reasons why they polling organisation either uses a list of might contain slightly too many of some randomly-drawn telephone numbers or groups and slightly too few of others. email addresses (for telephone or some What normally happens is that polling

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Understanding opinion polls 5 q f a

companies ask respondents not only the kind of extra information (such as about their views but also about them- gender and age) that would allow some selves. This information is then used to judgement to be made about the nature compare the sample with, for example, of the sample. census statistics. The raw numbers from the poll may then be adjusted 6. But surely a phone-in or write-in slightly, up or down, to match the pro- poll in which, say, one million file of the population being surveyed. people take part is likely to be more If, for example, a poll finds that, when accurate than an opinion poll sample its survey-work is complete, it has 100 of 1,000? members of a particular demographic Not so. A biased sample is a biased group, but should have 110 of them (in sample, however large it is. One a poll of, say, 1,000 or 2,000), then it celebrated example of this was the US will “weight” the answers of that group Presidential Election in 1936. A maga- so that each of those 100 respondents zine, Literary Digest, sent out 10 million counts as 1.1 people. This way, the post cards asking people how they published percentages should reflect would vote, received almost 2.3 million the population as a whole. back and said that Alfred Landon was leading Franklin Roosevelt by 57-43 5. Are other kinds of surveys bound percent. The Digest did not gather to be wrong? information that would allow it to judge No. Just as a stopped clock tells the the quality of its sample and correct, or right time twice a day, unscientific “weight”, groups that were under- or surveys will occasionally produce right overrepresented. Since the Literary percentages. But they are far more Digest sent its postcards primarily to likely to be inaccurate. The most com- individuals with telephones and auto- mon forms of unscientific surveys are mobiles, their “sample” included few phone-in polls conducted by televi- working class people. A young pollster sion programmes and self-selecting called George Gallup employed a much surveys conducted over the Internet. smaller sample (though, at 50,000, it These contain two defects. First, their was much larger than those normally samples are self-selecting. Such polls used today), but because he ensured tend to attract people who feel pas- that it was representative, he correctly sionately about the subject of the poll, showed Roosevelt on course to win by rather than a representative sample. a landslide. Second, such polls seldom collect

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

6 Understanding opinion polls

7. How can you possibly tell what for each view. But there is a one in 20 millions of people think by asking chance that the poll will fall outside this just 1,000 or 2,000 respondents? range. With a sample of 2,000, the poll In much the same way that a chef can will be within 2 percentage points 19 judge a large vat of soup by tasting times out of 20. just one spoonful. Providing that the soup has been well stirred, so that the 9. You say those calculations apply spoonful is properly “representative”, to “a random poll with a 100% one spoonful is sufficient. Polls operate response rate”. Surely that’s pie in on the same principle: achieving the sky? representative samples is broadly akin Fair point. Many polls are non-random, to stirring the soup. A non-scientific and response rates are often very survey is like an unstirred vat of soup. much lower – well below 50% in many A chef could drink a large amount from countries for polls conducted over just the top of the vat, and still obtain a a few days. misleading view if some of the ingredients have sunk to the bottom. Just as the trick 10. So isn’t the real margin of error in checking soup is to stir well, rather than much larger? to drink lots, so the essence of a scientific Possibly – but possibly not. Here are poll is to secure a representative sample, two examples, at opposite extremes rather than a vast one. of this issue. Return to our example of an equally divided country. Suppose 8. But isn’t there some risk of everyone who holds view A lives in sampling error in a poll of 1,000 or the northern half of the country, while 2,000 people? everyone who holds view B lives in the Yes. Statistical theory allows us to southern half. In that case, if pollsters estimate this. Imagine a country that ensure that half of each survey is divides exactly equally on some issue conducted in the north, and half in the – 50% hold one view while the other south, then their polls should be exactly 50% think the opposite. Statistical accurate. Structuring polls in this kind theory tells us that, in a random poll of of way is called “stratification”. Properly 1,000 people, with a 100% response done, stratification can help to increase rate, then 19 times out of 20, a poll a poll’s accuracy. will be accurate to within 3 percentage points. In other words, it will record Now make a different assumption about at least 47%, and no more than 53%, our mythical, equally divided country.

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Understanding opinion polls 7 q f a

Suppose people who hold view A are a well-designed, representative survey far more likely to be members of a finds that the public divides 70-30% religious or ethnic minority who refuse on an issue, then a margin of error of to answer questions than people who even 10 percentage points cannot alter hold view B. Unless the polling com- the fact that one view is expressed far pany is aware of this bias, and knows more widely than the other. However, it how big it is, it could well produce is true that in a closely-fought election, results showing that view B is far more a polling lead (in a sample of 1-2,000) popular than view A. This is an example of less than 5 percentage points for one of a systematic error. candidate or party over another cannot To measure the “true” margin of error, be regarded as a certain indicator of we would need to take account of who was ahead at the time the survey random sampling error, and the effects was taken – let alone a guarantee of of stratification, and possible systematic who will in the days, weeks or months errors. The trouble is that it is hard, ahead. and arguably impossible, to be sure of the true impact of stratification and 12. Can I Trust an Internet Poll? systematic errors. (If the impact of all The answer depends on the people systematic errors were known, a com- being sampled and the answer is the petent survey company would adjust its same for telephone or face-to-face results to compensate for them.) interviewing as well, not just the internet. If a representative sample of 11. Doesn’t this mean that polls can’t people can be reached by any of these really be trusted at all? interviewing modes, the answer is No. Polls may not be perfect, but yes, the poll can be trusted. There are they are still the best way of measuring places and populations where it is very what the public thinks. In most difficult for an interviewer to get a rep- countries where poll results can be resentative sample using face-to-face compared with actual results (such interviewing, just as there are places as elections), well-designed polls are and populations where only certain usually accurate to within 3 percentage classes of people have telephones or points, even if they occasionally stray access to the internet. Obviously if the outside that margin of error. Moreover, interviewing mode cannot be corrected much of the time, polls provide a good by stratification and weighting to allow guide to the state of opinion, even for this bias, the interviewing mode allowing for a larger margin of error. If cannot provide a representative sample

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

8 Understanding opinion polls

and should not be trusted. In some regions, ethnic groups etc.) cases the coverage of the internet may c) They might have been conducted be sufficiently high for internet inter- using different methods. Results can viewing to be a good, or even the best, be subject to “mode effects”: that is, way of carrying out surveys, but with some people might, consciously or the level of access to the internet still sub-consciously, give different answers low and unrepresentative in most coun- depending on whether they are asked tries, internet polls of the general public questions in person by an interviewer, should be treated with caution. or impersonally in self-completion Internet surveys use panels of people surveys sent by post or email/Internet. who have given permission to be There is some evidence that contacted for surveys. The ESOMAR anonymous self-completion surveys 26 Questions to Help Research Buyers may secure greater candour on some of Online Samples provides help on sensitive issues, than face-to-face or how to assess the quality of an internet telephone surveys. panel sample. d) The polls might have asked different questions. Wording matters, especially 13. I have seen polls conducted by on subjects where many people do not different, well-regarded, companies have strong views. It is always worth on the same issue produce very dif- checking the exact wording when polls ferent results. How come? appear to differ. There are a number of possible rea- e) There might be an “order effect”. sons, beyond those issues related to One poll might ask a particular question sampling error. “cold”, at the beginning of a a) The polls might have been conduct- survey; another poll might ask the ed at different times, even if they same question “warm”, after a series are published at the same time. If the of other questions on the same topic. views of many people are fluid, and Differences sometimes arise between liable to change in response to events, the two sets of results, again when then it might be that both polls were many people do not have strong views, broadly right, and that the public mood and some people may give different shifted between the earlier and the answers depending on whether they later survey. are asked a question out of the blue or b) The polls may have used different after being invited to consider some definitions of the group that they aspects of the issue first. are representing (e.g. different age,

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Understanding opinion polls 9 q f a

14. Does the way the question is that more people will say they are asked influence the answers? concerned than if this question had There is a great deal of knowledge been asked before the others. about how questions should be worded, When using answers to questions like based on what we know about how this, it is important to be aware that people process information. But this is the questions were biased or ambigu- really a matter of common sense. It is ous and therefore the answers cannot important to look at the exact question be an accurate reflection of what the which was asked and, if possible, to people answering them really believe. check the questions asked before it. This type of questioning is particularly Questions can contain concepts within popular with pressure groups, who use them which lead the respondent in them to try and get media coverage for a certain direction e.g. “There seem their point of view. Responsible journal- to be fewer policemen on the streets ists and commentators should not and a lot of people around here are report those polls, or they should draw concerned about rising crime, do you attention to misleading questions when think the police in this area are over- reporting the results of opinion polls. stretched?” or questions which contain more than one concept but where only 15. When I read or see a report of a one answer is reported e.g. “How well poll, how can I tell whether to take it is the city council dealing with traffic seriously or not? congestion and the lack of public trans- Check the following: port?” reported as the level of concern with public transport. Questions like a) Who conducted the poll? these will not provide clear or helpful Was it a reputable, independent polling answers about what people really think organisation? If not, then regard its find- of the police or public transport. ings with caution. If you are not sure, then one test is its willingness to answer the The context in which questions are questions below. Reputable polling firms asked can obviously influence the way will provide you with the information in which people respond. If a question you need to evaluate the survey. about concern with crime is asked after a series of questions about whether b) Who paid for the poll and why people have ever felt nervous on public was it done? transport or have a relative or friend If it was conducted for a respected who has been mugged etc. it is likely media outlet, or for independent

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

10 Understanding opinion polls

researchers, there is a good chance their preference. While excluding these it was conducted impartially. If it was groups ensures that, the poll reports conducted for a partisan client, such as the opinion of the most relevant group a company, pressure group or political – “likely voters” – reported voting-in- party, it might still be a good survey tention sample size may be significantly (although readers/listeners/viewers lower than the total sample, and the should be told who the client was). The risk of sampling error therefore greater. validity of the poll depends on whether Likewise, be careful when comparing it was conducted by an organisation sub-groups – for example men and that used a scientific approach to sam- women. The sampling error for each pling and questionnaire design, whether sub-group could be significantly higher it asked impartial questions, and whether than for the sample as a whole. If the full information about the questions asked total sample is 500, and made up of and results obtained are provided. If such equal numbers of men and women, information is provided, then the quality of the margin of error for each gender the survey stands or falls according to its (counting only random errors and intrinsic merits. If such information is not disregarding any systematic errors) is provided, then the poll should be treated around 6 percentage points. with caution. In either event, watch out for loaded questions and selective findings, d) How were those people chosen? designed to bolster the view of the client, Is it clear who is included in the sample rather than report public opinion fully and who was left out? If the poll claims and objectively. to represent the public as a whole (or a significant group of the public), has c) How many people were the polling company employed one of interviewed for the survey? the methods outlined in points 2,3 and The more people, the better – although 4 above? If the poll was self-selecting a small-sample scientific survey is – such as readers of a newspaper or ALWAYS better than a large-sample magazine, or television viewers writing, self-selecting survey. Note, however, telephoning, emailing or texting in – that the total sample size is not always then it should NEVER be presented as the only relevant number. For example, a representative survey. If the poll was voting intention surveys often show conducted in certain locations but not figures excluding “don’t knows”, others, for example, cities but not rural respondents considered unlikely to areas, then this information should be vote, and those who refuse to disclose made clear in any report.

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Understanding opinion polls 11 q f a

e) When was the poll done? g) What were people asked? Events have a dramatic impact on poll Try to get a copy of the full question- results. The interpretation of a poll naire, not just the published questions. should depend on when it was con- A reputable organisation will publish ducted relative to key events. Even the the questionnaire on its website, or freshest poll results can be overtaken provide it on request. Decide if the by events. Poll results that are several questions were balanced and be cau- weeks or months old may be perfectly tious about the results if the interview valid, for example, if they concern was structured in a way which seemed underlying cultural attitudes or behav- to lead the respondent towards a iour rather than topical events, but the particular conclusion. date when the poll was conducted (as distinct from published) should always h) Are the results in line with be disclosed. The date of the fieldwork other polls? is particularly important for pre-election If it is possible, check other polls to see polls where voting intention can change if the results are similar or very differ- right up to the moment the voter ent. Surveys aiming to cover the same records their vote. topic should come to similar conclusions. If the answers are very different, f) How were the interviews con- the reasons may become apparent ducted? when the questionnaire or the sampling There are four main methods: in method is examined. person, by telephone, online or by mail. Each method has its strengths i) Was it a “push poll”? and weaknesses. Telephone surveys The purpose of “push polls” is to do not reach those who do not have spread rumours and even outright lies telephones. Email surveys reach only about opponents. These efforts are not those people with Internet access. All polls, but political manipulation trying methods depend on the availability and to hide behind the smokescreen of a voluntary co-operation of the respon- public opinion survey. In a “push poll,” dents approached; response rates can a large number of people are called by very widely. In all cases, reputable telephone and asked to participate in a companies have developed statistical purported survey. The survey “ques- techniques to address these issues and tions” are really thinly-veiled accusa- convert their raw data into representa- tions against an opponent or repeti- tive results (see points 3 and 4 above). tions of rumours about a candidate’s

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

12 Understanding opinion polls

personal or professional behaviour. The and why did candidate/party (a) or focus here is on making certain the candidate /party (b) win?. Exit polls respondent hears and understands the are characterised by an elaborate accusation in the question, not in gath- design and a much higher number ering the respondent’s opinions. “Push of interviews than pre-election polls; polls” have no connection with genuine often tens of thousands and, in some opinion surveys. The best way to guard countries, hundreds of thousands are against “push polls” is to find out who interviewed. conducted the survey. Reputable companies have nothing to do with “push Exit polls have four distinct sources of polls”, a phenomenon that has grown in error, apart from pure random error: recent years in a number of countries. • Supporters of one candidate/ j) Was it a valid exit poll? party may be more willing to disclose This question applies only at elections. their vote than supporters of another. Exit polls, properly conducted, are an This phenomenon, “differential excellent source of information about nonresponse”,is especially hard to voters in a given election. They are the judge accurately in exit polls. only opportunity to survey actual voters and only voters. They are generally • Some people may genuinely have conducted immediately after people thought they voted for a particular have voted, and are therefore able (in candidate/party, but may inadvertently theory) to report actual behaviour. Pre- have voted for someone else, or spoiled election surveys, even those conducted their ballot paper or (when using voting the day before the vote, cannot entirely machines) not have completed the avoid the danger that some people may process properly. change their mind, about whether to vote or which party/candidate to sup- • Exit polls may not have been port, at the very last minute. conducted on an absolutely representative group of polling stations. Even if Properly conducted, they are an the total sample is very large – say, excellent source of information about 5,000 or more – it may suffer from an voters in a given election. In addition to effect known as “clustering”. If, say, 50 answering the question “Who won?” polling stations are selected, and 100 they provide information to answer the voters questioned at each, the figures questions: Who voted for the winner could be wrong if the overall political

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Understanding opinion polls 13 q f a

balance of those 50 polling districts is even slightly askew.

• There may be operational difficulties in sampling voters accurately at polling places – either because of interference by local election officials or because of legal restrictions on where interviewers my stand. Reputable polling organisations go to considerable lengths to avoid these problems. Other organisations may conduct exit polls in a minimal number of voting locations using interviewers who do not have experience or specialist training in this method of polling.

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

14 How to publish opinion poll and survey results

3. Guideline for the interested groups should, through publication of opinion research, have access to accurate and poll and survey results unbiased measures of public attitudes, intentions and behaviours. We recogn- 3.1. Introduction ise concerns regarding the effects 1. Public opinion research – the study of such poll data on voting or other of people’s attitudes and beliefs and behaviour. However, the alternative is behaviours about political, social and that the public would be exposed only other issues – forms a part of the total to unscientific and probably inaccurate market and social research field. It is assertions about the situation, in many subject to exactly the same profes- cases presented by individuals or sional and ethical requirements as organisations who take an extremely other forms of survey research. These partisan or ideological approach to pre- requirements are set out in the ICC/ senting the facts. The objective of this ESOMAR International Code on Market Guideline is to reduce the risk of the and Social Research. public being misled by research which is inadequate or badly presented. 2. However, public opinion research often concerns an especially ‘sensitive’ 4. Recommendations in this Guideline area. It deals with issues which arouse are based on the previous ESOMAR greater public interest and emotion Code for the Publication of Opinion than do most market research projects. Polls. This was examined by the In addition, its findings are much more Parliamentary Assembly of the Council widely published and debated, and may of Europe which then recommended sometimes be presented in a provoca- the widespread application of the Code tive or even tendentious way. ESOMAR to govern the publication of polls. The has therefore set out specific recom- Code has since been combined with mendations about the publication of updated guideline advice. such research. 5. The validity and value of public 3. Opinion polls and surveys have a opinion surveys depend on three main valuable role to play in present-day considerations: society. Because the data from polls and surveys have far-reaching implica- (i) the nature of the research tech- tions, it is important that the general niques used and the efficiency with public, politicians, the media and other which they are applied,

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

How to publish opinion poll and survey results 15

(ii) the honesty and objectivity of the be theoretically desirable and what is researcher carrying out the study, practicable.

(iii) the way that the findings are pre- 7. All reputable research organisations sented and the uses to which they are apply the appropriate scientific methods put. and operate with professional objectivity. In doing so they conform to the This section of the Guide concentrates ICC/ESOMAR International Code on f or researchers

primarily on the second and third of Market and Social Research. There is and commissioners these issues. Specific guidance on also general agreement among them techniques and the conduct of pre-elec- on the principles which should under- tion polls and exit polls is given in the lie the publication of research results. following two sections – 4 and 5. However, normal professional practice may vary between countries and in 6. Major misunderstandings can arise certain countries additional information when opinion poll or survey find- to that specified in this Code will also ings are published and debated. It is customarily be provided as part of the unrealistic and unreasonable to expect standard key material. the media to quote the full technical background of a survey when present- Research organisations have a particu- ing its findings: they have limitations of lar responsibility in the field of space and must also hold the interest public opinion polls, to make sure that of their audience. However, there is both the client and the public have a certain basic information which must reasonable understanding of the special be provided if that audience is to have problems and limitations involved in the opportunity of judging for itself measuring attitudes and beliefs as the evidence presented and decid- distinct from behaviour. Such research ing whether or not it agrees with any frequently deals with complex and sen- conclusions drawn from the research. sitive issues about which respondents This Guideline is primarily concerned have varying degrees of knowledge and with trying to ensure that the public has interest, and where their views may reasonable access to this key informa- often be half-formed, confused and tion about the survey, and that pub- inconsistent. High professional integrity lished reports of the findings are not and skill is essential if the research misleading. The Guideline tries to strike itself is to be unbiased and meaningful, a realistic balance between what would and if the findings are to be presented

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

16 How to publish opinion poll and survey results

and interpreted clearly and accurately. 3.2. Scope It is important also that the research 1. This Guideline must be read in budget available is sufficient to carry conjunction with other ESOMAR codes, out a valid study. ESOMAR and WAPOR guidelines and principles available at fully recognise that such considerations www.esomar.org are vital if public opinion polls are to merit public confidence and support. 2. Market, social and opinion research involve the gathering and further 8. If as a result of past experience, a processing of personal data, which is researcher has reason to believe that regulated by law in many countries. In a particular client will not fairly present addition, certain countries have laws opinion poll results in the published regulating the publication of pre-elec- version of the findings, the researcher tion opinion poll results. This Guideline has a responsibility to stop carrying out does not provide a description of the polls for publication by that client. legislation in force around the world, and cannot replace the advice of legal 9. Finally, one particular type of work experts and self-regulatory bodies. which can be confused with an opinion Rather it sets minimum standards of survey is a consultation exercise or ethical conduct to be followed by mar- straw poll. In this type of study people ket research professionals and is to be reply to questions in the same way as applied against the background of any in a sample survey, but the individuals stricter standards that may be required who answer the questions select them- in a specific country. selves rather than being chosen as part of a scientific sample to represent a 3.3. Requirements of the ICC/ particular universe or population. It is ESOMAR International Code extremely important that the results of this type of work, which can provide 3.3.1. General requirements insight and engage the public in the 1. All researchers who conduct public democratic process, are never reported opinion polls and surveys must in a way which could lead to it being conform to the ICC/ESOMAR Inter- confused with a representative survey national Code. Particular attention is of opinion. drawn to the requirements of Article 1d (concerning the clear separation of research from nonresearch activi- ties), Article 3b (concerning protecting

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

How to publish opinion poll and survey results 17

respondents from harm or adverse (a) the name of the research organi- effects as a result of participating in sation carrying out the survey; a survey), Article 7 (concerning data (b) the universe effectively represent- protection and privacy) and Article 11 ed (i.e. who was interviewed); (concerning interpretation, reporting (c) the achieved sample size and its and publication of results). geographical coverage; (d) the dates of fieldwork; 2. It is important to distinguish between (e) the sampling method used (and f or researchers

the requirements which apply to the in the case of full random probability and commissioners reporting of public opinion poll results samples the response rate achieved); by a researcher to the original client, (f) the method by which the infor- and those which apply to the subse- mation was collected (face-to-face, quent publishing of any poll findings by telephone interview, internet panel etc.); that client to a wider audience. The first (g) whether weighting was used to of these situations is largely covered adjust the results and the universe by Article 11 of the International Code used for the weights which specifies reporting requirements (h) the relevant questions asked. In in detail. This Guideline is intended order to avoid possible ambiguity the to clarify certain additional require- actual wording of the question should ments which arise in connection with be given unless this is a standard ques- the wider publication of the findings, tion already familiar to the audience and therefore applies especially to the or it is given in a previously published second situation. report to which reference is made.

3.3.2. Specific requirements for 3. “Achieved sample size” is the published public opinion surveys number of interviews actually reported. 1. This section covers requirements for “Geographical coverage” should state all published public opinion surveys, which broad regions of the country Section 4 covers specific and more (national or other such as urban areas detailed requirements for surveys of only or a specific geographical area) voting intention. were represented. Additionally the number of sampling locations used 2. When public opinion survey findings should be given as an indication of the are published in print media they must adequacy of the sample design. In re- always be accompanied by a clear ferring to the number of “locations” the statement of: objective is to provide a realistic picture

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

18 How to publish opinion poll and survey results

of the extent to which the sample is the raw data collected in the field. (This widely distributed geographically. The recommendation is especially relevant best terms to use would vary by coun- in the case of non-standard weight- try – for example “Départements” ings – e.g. weighting applied to correct might be best in France, “Parliamentary for disproportionate sampling, like Constituencies” in the U.K. over-representing particularly competitive states or constituencies to improve 4. It is important for the reader to be efficiency, under-representing specific given some general indication of the ethnic or religious groups or to correct sampling method used since this may for poor coverage of the population in certain cases have very strong impli- when using online or telephone inter- cations for the likely representativeness viewing). of the sample. A published report cannot necessarily provide all the relevant 6. The guiding principle when deciding data for a technical assessment; but which question wordings are relevant even a limited reference can be helpful. to publish should be the elimination of In the case of probability sampling, the ambiguity and misunderstanding. This main objective is to identify those stud- is particularly important where the ies where an unusually low response actual wording of the question is critical rate has been achieved, for whatever to the interpretation of the findings, reasons. Although it is preferable and where the reported answers can wherever possible to quote the actual be affected by the precise form of the response rate, the main requirement question or its context – especially on is therefore to indicate if the response politically or socially sensitive issues rate is below that regarded in the (for example, attitudes towards abor- profession as “normal” for that type of tion). The description should therefore study (this is a matter for experienced help the reader understand exactly professional judgement). what was asked. In some cases this will be sufficiently clear from the text 5. It is also important to indicate itself and the actual answers reported, whether the results quoted have been but when in doubt the question wording adjusted by weighting procedures used should be reported. Certainly or other statistical methods and it is where tabular data are given, the full recommended that the raw data is question wording must be included. made available wherever the findings Experience shows that it is in practice reported differ substantially from often possible to include the questions

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

How to publish opinion poll and survey results 19

without overloading the published A further example is: report. Where the company is able to publish information on its website, the A national online survey of 2222 full question should be made available, adults interviewed online on 5th together with, as a minimum, the March 2008 by XYZ research and answers for the weighted sample in weighted to represent national voting total. These answers should include shares. Full details of survey available “Don’t know and non-response” at www.xyzresearch.” f or researchers and commissioners 7. Any recommendations for a standard 8. There are certain specific situations publication format in print must take in which it is clearly difficult to follow all account of the different styles, layouts, the recommendations listed: etc. of widely varying types of publication. One example of a suitable form of (i) where the survey reported on is wording would be: very extensive and complex and where the media report can therefore This survey was carried out by ABC provide only a relatively brief overview Research, on behalf of News Inc., of the total survey using a national quota sample of 1111 adults of voting age personally (ii) where an article summarises the interviewed in 102 locations between results of a number of surveys, when st th 1 -5 March 2008. again it would be too complicated to give all the key information for each of Another alternative is to use a ‘data the surveys referred to. box’ of the kind: Also, where a given survey is reported Survey carried out by XYZ Research, on ‘serially’ (for example in the course on behalf of News Inc. National of several consecutive issues of a survey of 1234 adults aged 18 and newspaper) it might be unnecessary to above, interviewed by telephone repeat all the technical details in every between 25th-28th February 2008 issue. using random digit dialling and a quota sample. These situations are exceptions. Most published reports on public opinion polls refer to much more limited studies than these. Even in the more complex

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

20 How to publish opinion poll and survey results

cases it should frequently be possible basic pieces of information referred to to give the key information required. in paragraph 2 above. The publication In all cases where the key informa- of this type of information on the agen- tion cannot be fully provided, the basic cies or broadcaster’s website is also a principle of fair and informative report- way of making fuller detail available. ing must be followed, and it should be made clear how and where the 10. The percentages of respondents serious enquirer can obtain fuller who give ‘don’t know’ answers (and details. The advent of the internet in the case of voting-intention studies, makes it possible to publish a consid- of those who say they will not vote) erable amount of information about a must always be given where they are survey, including details of sampling, likely to affect the interpretation of the weighting and questionnaire which findings significantly. When comparing might be difficult to provide in the origi- the findings from different surveys, nal publication or broadcast. ESOMAR any changes (other than minor ones) recommends that the survey company in these percentages must also be publishes full details of all public polls indicated. There are many occasions in on a website and that the website’s which the interpretation of particular address is given to the public by the findings will be quite different if the level media, so that people can check details of ‘don’t know’ answers is 5% or 50%. if they wish to. In the case of voting-intention studies the same consideration also applies to 9. In the case of broadcast media it ’will not vote’ answers. A researcher may not always be possible to give must apply their experience and profes- information on all these points. As sional judgement in deciding when such a minimum, points (a) through (f) in situations arise. It may not be necessary paragraph 2 above must be covered in to include all the ‘don’t know’ percent- any broadcast reference to the findings ages in any tables given, although of a public opinion poll, preferably in where this is possible it is frequently visual (written) form where practical. the best way of dealing with the issue. Some research organisations currently It may be quite sufficient, for example, arrange with their client to provide a to make a general comment such as: press release at the same time as the “the proportion of ‘don’t knows’ was broadcast report on a survey. Such a never higher than 5%” - or to comment press release can more easily include specifically on those instances where some kind of fact sheet covering all the the proportion was much higher.

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

How to publish opinion poll and survey results 21

11. Whatever information may be given It is therefore the responsibility of in the published report of the survey, the research organisation to draw its the publisher and/or the research client’s attention to this Guideline on organisation involved must be prepared Publication of Results and to use its on request to supply the information best endeavours to persuade the client described in paragraph 2 of this Guide- to follow the Code’s requirements. line on the survey method, as required by Article 11 of the ICC/ESOMAR Code. 2. The research organisation and f or researchers

Where the questions formed part of a the client each have a responsibil- and commissioners more extensive or ‘omnibus’ survey, ity in the public interest to ensure this must be made clear to any enquir- that the published report on a public er, including a general indication of the opinion poll does not misrepre- placement of the questions in the larger sent or distort the survey data. For questionnaire. There is no obligation example, misleading comments based under the Code for further information on non-significant differences must be beyond this to be supplied – although avoided. Special care must be taken to organisations will normally be prepared ensure that any graphs or charts used to discuss their research methods in do not convey a misleading impres- more detail with bona fide enquirers. sion of the current survey’s results or of trends over time. It is also important 3.3.3. Arrangements between the that the reader or listener should be Research Organisation and its Client able to clearly distinguish between the 1. In order to ensure that the require- survey findings as such and any edito- ments of the ICC/ESOMAR Code are rial or other comments based upon followed, and to avoid possible misun- these findings. Wherever feasible, the derstandings, the research organisation research organisation must approve in must make clear in advance to its advance the exact form and content of client: publication as required in Article 11b of the ICC/ESOMAR Code. (i) that the research organisation itself is bound by the requirements of the 3. In preparing material for publication, general International Code. journalists and others connected with the media themselves normally follow (ii) that subsequent wider publication their own professional codes of prac- of the research findings should be in tice and ethics. The present Guideline accordance with this guideline. is not intended in any way to substitute

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

22 How to publish opinion poll and survey results

these but rather to support them. (In misreporting or misuse of results when this context, ‘published report’ covers these are brought to its attention. non-print as well as print media.) 5. In the event that a client releases data Article 11a of the Code emphasises the from a survey which was not originally importance of distinguishing, as much intended for publication, this Guideline as possible, the results that emerge will apply to it as if it had originally been directly from the questions asked, and commissioned for publication. any commentary/interpretation based on these results. Although the dividing 6. ESOMAR strongly advises the line is not always a simple one to use of contracts between research define, in most cases the distinction organisations and their clients to between ‘fact’ and ‘comment’ is in ensure adherence to the ICC/ESOMAR practice a workable one. Code (see ESOMAR Guideline on the mutual rights and responsibilities of Article 11d of the Code requires that researchers and clients). For example, the research institute must reserve the some contracts stipulate that the right to publish the total study and not agency has the right to examine and just the technical specifications in the approve a copy of the publication based event of: on its research. Where the agency reserves the copyright of the findings • a shortened version of the publication this can also help to reduce some of distorting the analysis of the results the problems involved in misleading • an unforeseen and abridged version ‘secondary reporting’ of the findings by of the publication other people. In addition to any other • a publication which does not conform requirements it is suggested that such to the prior agreements a contract should cover:

4. The research organisation cannot • clarification of the point that the normally be held responsible for any contract binds both the survey sponsor subsequent use made of public opinion and the media involved, where these poll results by people other than the are different parties original client. It should however • some measure of control by the be ready to immediately issue such research organisation over the pub- comments or information as may be lished form of the results including necessary to correct any cases of figures and graphs

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

How to publish opinion poll and survey results 23

Certain contracts also provide that if research findings commissioned for publication are not in fact published, such findings can subsequently (after a specified period of time) be released by the research organisation itself; or alternatively the organisation is free to repeat the survey for another client. It f or researchers

is also increasingly common practice and commissioners in certain countries for data tapes from public opinion surveys to be lodged with appropriate archives for subsequent secondary research by academic researchers and others. Such steps can help to reduce the danger that polls may be thought sometimes to be used in a ‘manipulative’ way by less scrupu- lous clients.

7. Any code of practice in this area has obvious limitations, in that researchers can exercise only restricted control over how their results are presented in the media, and still less influence over any comments and interpretations (sometimes misguided and tendentious) based on the findings. A code therefore depends on spreading the use of ‘best practice’ and influencing media clients to avoid misleading presentation of survey results. ESOMAR expects its members to follow this Guideline with this objective firmly in mind.

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

24 Best practice guidance for researchers

4. Specific guidance on electors will actually vote in line with conducting pre-election their earlier stated voting intentions. opinion polls and polls People do change their mind, some on voting intention even in the second before marking their vote on the ballot slip. Polling organisa- 4.1. Introduction tions have a responsibility to electors to 1. This section concentrates on the ensure that polls, especially those polls conduct of pre-election polls and as- published in the last few days of an sumes that the requirements of the election campaign, provide reliable and ICC/ESOMAR Code described in the objective information. preceding section will be met. At first it may seem strange to have specific 3. The second objective of these guide- guidelines on pre-election polls since lines is to protect the public reputa- they are just one particular type of po- tion of market research using sample litical poll. While all opinion polls require surveys. Pre-election opinion polls high technical standards, it is pre-elec- which are published in the final days tion polls that feature most frequently of an election campaign have a major in the debate about polls, and which influence on this. While it is true that are restricted in some countries. These opinion polls are a snapshot of inten- guidelines have two main objectives – tions at a specific point of time, they to protect the interests of the voter in a are inevitably seen as predictions of the democracy and to protect the credibility election result when they are pub- of market and opinion research. lished late in the campaign. In general, pollsters have not effectively challenged 2. The first objective of these guide- this use of polling data, partly because lines is to ensure that polling organisa- the track record of the polls in “predict- tions take all possible technical steps ing” the result is good. to ensure that polls published close to the vital decision point for voters are 4. In some countries where the publi- an objective guide to the state of public cation of polls in the final stages of a opinion and voting intentions. The campaign is restricted, polls based on process of sampling cannot guarantee national samples are often conducted highly precise measurement by every on polling day, or the day before, for single poll. Also, the measurement publication within minutes of the close of stated intentions to vote is only an of the polling stations. Also carrying out estimate and cannot guarantee that all exit polls (interviewing voters as they

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Best practice guidance for researchers 25

leave the polling station) has become Polling organisations must be respon- much more common. Such polls are sible for ensuring that polls published even more likely to be seen as predic- in the very late stages of an election tion polls and the analysis of their are likely to be a fair representation of results is often used to explain why the public opinion as close as possible to election came out the way it did. Their the end of the campaign. accuracy is equally important to the public image of market research. Guideline Polling companies should try to reduce the risk of “getting it wrong” 5. Pre-election polls are a very public by minimising the time elapsed be- test of sampling theory and survey tween fieldwork and publication. A poll research in action. Polls have a good is more likely to achieve a good repre- track record for accuracy but the occa- sentative sample if the fieldwork period sional poll which appears to be wrong includes some time in the evening gets extensive media coverage. “Polls and during week-ends when potential wrong” is news and always gets major electors in full-time employment are coverage. “Polls accurate” is a headline available for interview. that is seldom written. ESOMAR hopes that these guidelines will contribute to 4.2.2. Sample size the technical education of journalists The measurement of the share of the responsible for poll reporting. However, vote for a party is subject to the normal special care must still be taken by poll- statistical confidence limits for sample ing organisations to minimise the risk surveys. Two factors affect the size of of “getting it wrong”. the confidence limit of any party share. The first is the absolute level of support 4.2. The Guidelines for a party. The closer this gets to 50%,

the wider will be the confidence limit f or researchers 4.2.1. Timing of fieldwork around the share estimate. The second Anyone attempting to prepare a critical is the size of the sample interviewed in case about polls would certainly decide order to produce the estimate. In most that the date of publication of a poll is a pre-election polls the size of the sample key piece of information. Regardless of is the more important factor. when the interviewing took place, the publication date is the important fact in In countries with a simple proportional judging the contribution of the poll to representation system, the pre-election the electoral process. poll measurement of the share of the

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

26 Best practice guidance for researchers

vote to each party is a fair indicator of 4.2.3. Sample distribution the election outcome. In other electoral In European countries there are two systems this may not be the case. main approaches to selecting samples However, the best that the polls can for face-to-face interviews. do is estimate the share of the vote to each party at a national level. The Method 1 key statistic reported in the media is Select sampling points in proportion to the gap in share of vote between the the number of electors. Then, in each leading parties, and the measurement selected area take an equal number of the gap has much larger confidence of interviews. For any given sample limits than those for an individual size, the rule for samples using this party’s share. methodology should be to maximise the number of sampling points and Polling organisations frequently repre- minimise the number of interviews sent the margin of error of their polls conducted at any one sampling point. as ± 3 percentage points. This may be This implies minimising the number accurate for a single party but is rarely of interviews conducted by any single accurate for the key media figure – the interviewer. gap between leading parties. A poll that produces a 95% confidence limit of ± 3 Guideline There is empirical evidence percentage points on the share for one that if the number of interviews per of the leading parties, could produce sampling point exceeds 20, then the an equivalent confidence limit of ± 5.7 influence of increased interviewer percentage points on the gap between variation outweighs increased sample the two leading parties. size.

Guideline Pre-election polls should Method 2 not have a sample of fewer than 1,000 Select sampling points with equal respondents. In circumstances where probability but then to take a different the gap between leading parties is number of interviews in each area expected to be small, the sample size proportional to the size of the electorate should be larger and samples of 1,500 in the area. to 2,000 should be used. Guideline Polls using this method should still aim to minimise the number of interviews conducted by any single

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Best practice guidance for researchers 27

interviewer and stick to the maximum polling organisation must use methods of 20. to correct for any under-representation of supporters of particular political Guideline For both methods, the parties. Simple demographic profile sample design should give priority adjustments may not be adequate. to maximising the number of sampling points and minimising the 4.2.5. Online Polls number of interviews conducted by Internet surveys are increasingly being each interviewer. used for opinion polls. As with telephone surveys, its viability as a method of 4.2.4. Telephone interviewing carrying out pre-election polls depends In principle telephone surveys offer to a large extent on the accessibility of high-quality, unclustered, random sam- a representative sample via the internet. ples with fast completion of fieldwork. Given the spread of the internet, there are In countries with limited telephone a number of countries where this can be ownership or availability, ownership is done, if care is taken to include people frequently correlated with voting inten- who do not have the possibility to use the tion, i.e. those who cannot be contacted Internet at home, but connect to the inter- by phone are more likely to support one net at work or in some other place. The rather than another of the parties in the ESOMAR Guideline on Research Using the election. This may also be the case for Internet contains additional guidance on those telephone owners who are not requirements including online sampling listed in the telephone directory, which and the use of access panels. is why random digit dialling is to be preferred, or those whose telephones Guideline This method should not be may not be part of the sampling frame used for pre-election polling without

(e.g. mobile/cell phone households). evidence that a representative sample f or researchers of electors can be contacted via the Guideline If telephone ownership is internet. As with telephone surveys, not high and/or likely to produce an the polling organisation must correct unrepresentative sample, this method for any underrepresentation. Simple of interview should not be used for demographic profile adjustments will pre-election polls, or should be supple- generally not be adequate. The organi- mented with data collected using sation should provide information about another mode of interviewing. If tele- sample coverage and the weighting phone samples are used for polls, the variables used to correct the data.

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

28 Best practice guidance for researchers

4.2.6. Weighting In some countries it is common to In order to conduct fast surveys with measure self-reported voting at the large samples, most pre-election opin- previous election and use this to adjust ion polls based on face-to-face inter- the voting estimate from the current views will use quota sampling methods. survey. In Denmark and France this is The application of simple demographic an important quality improving proce- weights to ensure an accurate sample dure. However, this approach does not balance is standard good practice. If work as well in other countries. some parts of the electorate have been deliberately over-sampled, weighting Experience has shown that voting inten- should be used to re-establish the tion measurements in some countries correct balance (see section 4.2.11 for need adjustments in order to provide a additional information). reliable guide to public opinion. In these countries the weighting or adjustment Guideline The demographic profile of strategy of the polling organisation pre-election polls should be checked may be confidential to that company for for representativeness and, if neces- competitive reasons. In such circum- sary, weighting should be applied to stances, where the adjustment is made correctly represent the electorate. via a stable and systematic procedure, Polling companies should ensure that the polling company may wish to with- the population profile used is that of hold full details of its method. electors eligible to vote rather than the more normal all adults profile used in Guideline Polling organisations should commercial market research. not make any adjustments to the simple poll result which cannot be documented 4.2.7. Adjustments and defined in advance of seeing the In some circumstances polling organi- results. Any adjustments must be sations may feel that the result of their capable of being repeated and justi- pre-election poll is not an accurate fied. It is good practice for pre-election guide to the likely outcome. The most polls to measure key variables such obvious example is where the voting as likelihood to vote and to consider intention of those with a high likelihood whether the simple poll result should of casting their vote is different from be adjusted. Where adjustments to the the total sample. unweighted poll findings are made, this should be noted in the publication of the poll findings.

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Best practice guidance for researchers 29

4.2.8. Survey content of its methodology for the final poll. It Pre-election opinion polls will have could be argued that there is a benefit if greater political and social value if polling organisations improve the meth- they do not confine themselves only odological quality of the design they use to measuring voting intention but for final polls. However, this reduces also explore the reasons for party the comparability of a series of polls choice and opinions on important from the same polling organization, and campaign issues. suggests that some campaign polls can be of poorer quality. However, if polls Guideline Wherever possible, pre-elec- are to make a valid contribution to in- tion polls should measure reasons forming the political process, they must for party choice or attitudes on issues all be of high quality. A two-tier quality or other aspects of the campaign. policy undermines the value of polls.

4.2.9. Time series Guideline Polling organisations should Polling organisations use different attempt to keep key elements of methodological designs that may methodology consistent throughout the distinguish one firm from another. The election campaign. This particularly meaning of a final pre-election poll is applies to sampling method, question far easier to determine if it is the latest wording and the positioning of voting of a series of polls conducted by the intention questions. It does not apply same organisation during the campaign. to sample size. Where polls are based on eligible (registered) voters early in Guideline The validity of the methods the campaign and subsequently move used by an organisation can be judged to samples of “likely” voters late in the better if they produce a series of voting campaign, this adjustment should be

intention estimates during the cam- clearly noted and distinguished. f or researchers paign. Any obvious biases will become apparent by comparison with the pub- 4.2.11. Indicator variables lished polls of other organisations. It is possible to interview a sample representative as far as age, sex and social 4.2.10. Consistent design grade are concerned, but still have a The ability to compare a final pre-elec- politically unrepresentative sample. tion poll with previous polls from For example, section 4.2.1 refers to the the same organisation is weakened if issue of representing working electors the organisation changes key aspects by ensuring that the fieldwork period

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

30 Best practice guidance for researchers

includes some evening and weekend out of line with earlier polls unless a interviewing. valid technical reason why the poll is wrong has been established. It is also It is good practice in a pre-election poll unacceptable to make a “gut feeling” to include the collection of information adjustment. The poll should be which is correlated with voting behav- published with appropriate warning iour but which is not part of the quota about the unusual poll result. The odds control. The item should have a known are 20:1 in favour of the poll being cor- distribution from alternative sources. rect and voters do change their minds Variables such as religion or language even in the polling booth. spoken may be possible indicator variables if they have not been used in the design of the survey quota controls.

Guideline Polling companies should be encouraged to develop an indicator variable for sample weighting purpos- es. In the period between elections it is possible to calibrate the electoral impli- cation of an over or under representation of the indicator variable.

4.2.12. Unusual poll results Probability theory suggests that one poll in twenty may give results outside the normal 95% confidence limits. A polling organisation may therefore find itself with a pre-election poll that is out of line with all the campaign evidence available up to that point. Usually there will be little time available between getting the final result and writing the copy for publication.

Guideline It is unacceptable to suppress a pre-election poll that looks

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Best practice guidance for researchers 31

5. Guideline on nating the results will inevitably raise conducting exit polls and questions about the legitimacy of the election result forecasts effort, especially with regard to estimat- ing the outcome of the election. If 5.1. Introduction analysis is the only purpose of the exit An increasingly popular component of poll, prompt release is less important. the television coverage of election nights is the exit poll. These are not In some countries, election laws based on nationally representative prohibit the publication of exit poll samples of voters asked their voting data until after the polls have closed. intention. Exit polls are polls of voters, WAPOR and ESOMAR oppose regula- interviewed after they have voted, and tion of the conduct and reporting of no later than Election Day. They may polls in principle. Nevertheless, state- include the interviewing before Election ments about the outcome of an election Day of postal, absentee and other early based on exit polls must not be pub- voters. In some countries Election Day lished before all the polls in the contest polls cannot be conducted at the polling have closed. In national elections, this place, but in most cases, interviewing means polls relating to election results takes place at the polling location. for elections in smaller voting units can be reported when all the polling places Exit polls can serve three different have closed in those locations, rather functions that are not mutually exclusive: than waiting until all polling places used for voting that day have closed. • predicting election results Descriptive information other than vot- • describing patterns of voter support ing behaviour may be published before for parties, candidates, and issues the polls have closed, unless this is

• supporting extensive academic prohibited by local legislation or codes f or researchers research efforts of practice.

The main difference between these 5.2. Specific requirements may be the speed with which the re- for exit polls sults are formulated and disseminated. Survey researchers in general and those conducting exit polls in particular Exit polls used for projections should need to follow certain broad principles be reported as soon as is practical after in conducting their research: the polls close. Any delay in dissemi-

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

32 Best practice guidance for researchers

1. Exit polls conducted for public ing. Limitations and weaknesses in the consumption must be impartial and design of an exit poll, its execution, and non partisan. Exit polls are designed the results must be noted in all reports to collect data from voters and report and analyses. Results must be released information on electoral outcomes. to the public and other interested They are not tools for partisan advocacy. parties through the general media and simultaneously made accessible to all. 2. Methods must be transparent, public, and well-documented. These goals can 5. The identity of respondents in exit be achieved by publicly describing the polls must be protected. Identifying methods prior to conducting the exit information (e.g. name, address, or poll and by adhering to the standards other IDs) must not be maintained with of minimal disclosure delineated in this the voter-level records, and the data Guideline. It is also recommended that set should not allow deductive disclo- when the exit poll is used for analysis, sure of respondents’ identities. To limit the data set (without individual identi- the chances of deductive disclosure, fiers) along with appropriate survey small-area geographic details such as documentation be deposited in public the specific polling place in which votes archives and/or on websites for general were cast must not be revealed. access. 6. Poll methods used must be those 3. Researchers must adopt study generally accepted as good survey designs for their exit polls that are suit- practice and must be disclosed in able for producing accurate and reliable advance of the conduct of the exit results and that follow specific proce- poll, as well as with any projection or dural and technical standards stipulated analysis or subsequent public release in this document. of the dataset.

4. When reporting results from exit 5.2.1. Items for minimal disclosure polls, researchers must observe the re- 1. These items must be disclosed with quirements for publication of surveys in any exit poll report or when any projec- the ICC/ESOMAR Code and be careful tion is made. Good practice would be to to keep their interpretations and state- disclose as much of the methodology ments fully consistent with the data. in advance as possible, particularly Speculation and commentary should those items in italics, which must be not be labelled as data-based report- disclosed before Election Day.

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Best practice guidance for researchers 33

• Sponsor of the exit poll • A description of the precision of • Name of the polling company or the findings, including estimates principal researcher; prior experience of sampling error (if any) in exit polling; and whether • Monitoring and validation the data collector has any business procedures (if any) or personal ties to political parties, • Weighting procedures candidates, political organizations or • Response rates (using one of governmental bodies. the definitions in the AAPOR/ • Name of the organization responsible WAPOR “Standard Definitions: for analysis and projections, if different. Final Dispositions of Case Codes and • Number of interviews Outcome Rates for Surveys”) and • Number of polling stations or item non-response on vote questions. sampling points if this is not where • Any known non-response bias the sampling takes place • General description of how estimates • Sampling frame are made and the kinds of variables • Geographic dispersion and coverage that are being used, and whether • How sampling points are selected adjustments for non-response have • Where and how interviews are been made conducted: at polling places, • Known design effects in person at homes, by phone, by self-administered questionnaire etc. 2. Political parties may sometimes • Any legal limits on data collection make claims about private data. These that might affect polling accuracy (e.g. claims also require documentation. Any minimum distance of interviewers public statement referring to exit poll from the polling place) results must abide by the disclosure • Time of day of interviewing principles and requirements above.

• Whether interviewers are part of f or researchers a permanent field staff or hired for 3. Those conducting exit polls must the occasion always use generally accepted statisti- • How respondent anonymity is cal methods. However, there are a guaranteed (paper questionnaires, etc.) number of good practices that apply • The interview schedule or specifically to exit polls. questionnaire and instructions • Which results are based on parts • Exit polls typically employ clustering of the sample, rather than the in their sample designs. Because of the whole sample possibilities that various groups might

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

34 Best practice guidance for researchers

attempt to influence voters and/or exit smallest voting unit level or because poll respondents, researchers are not the sampling units do not coincide with expected to disclose the actual sample voting units. points or locations. 5.2.2. Election projection methods • Exit polls must collect information and their disclosure across the whole of the polling day. 1. Election projections can be made in Probability sampling (or full census) other ways than by interviewing voters for interviews conducted at the polling as they exit the polling place. While place is the only acceptable selection most projections are based on inter- method. Quotas are not appropriate for views with voters after they have voted sampling at the polling place. at a polling place, other forecasting models may include: • A national exit poll must represent the entire country, with 95% of the target • interviews in person, by telephone, population included in the sampling or by other means of communication frame. If the sampling frame covers with voters after or before having cast less than 95% of the target population, their votes there must be an explanation for that • counts of official votes in a sample of decision. precincts, often known as quick counts • a mix of methods • Researchers must keep in mind the relationship between small units for 2. A projection is an estimate that leads which the votes are tabulated and to a conclusion about the outcome of can also serve as clusters for exit poll an election in a jurisdiction such as a interviews. One way to evaluate an exit nation, a state or a district. This may poll is to compare the actual election occur in two different situations: results with the estimates derived from the exit poll interviews for these • If the winner is based on the popular same units. This comparison of small vote for an office or a party, then a unit accuracy, typically at the precinct projection of the division of that vote or polling place level, is one of the is a projection of the outcome in the best ways to understand the exit poll’s jurisdiction. success. But there are situations where • If the winner is based on the vote in this will not be possible, either because multiple jurisdictions, such as election no tabulations are reported at the of a Parliament where votes are cast in

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

Best practice guidance for researchers 35

districts or of a President where votes about the election outcome based upon are accumulated based on victories collecting information from a subset of won in many jurisdictions, a conclusion all jurisdictions. Similar rules to those about which party has a plurality of for the provision of basic information seats in the new Parliament or which apply. The following information should presidential candidate has a winning be made available before election day: number of votes is a national projection. • Sponsor of the exit poll The projection need not reach a con- • Name of the polling company or clusion about each sub-jurisdiction. It principal researcher; prior experience need only reach a conclusion about the (if any) in projections; and whether jurisdiction outcome. the data collector has any business or personal ties to political parties, The objective of any projection is a candidates, political organizations or conclusion about an election for some governmental bodies. jurisdiction. A sample of that jurisdic- • Name of the organization responsible tion must be adequate to reach an for analysis and projections, if different. unbiased conclusion with sufficient/ • Source of the data used to make the appropriate confidence in the estimate. projections (i.e. interviews in person, by A national projection typically requires telephone, or by other means of com- the coverage of the entire country, with munication with voters after or before at least 95% of the target population in having cast their votes). the sampling frame.

There will be times that a subset of the country will be used (for example,

only competitive districts). But if the f or researchers sampling frame is used that includes something less than the entire voting population of a jurisdiction, then the researcher should define in a disclosure statement what is and is not included in the sampling frame. The researcher must also publish a rationale to justify the researcher’s ability to make an unbiased conclusion

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

“Market research, which includes social and opinion research, is the systematic gathering and interpretation of information about individuals or organisations using the statistical and analytical methods and techniques of the applied social sciences to gain insight or support decision making. The identity of respondents will not be revealed to the user of the information without explicit consent and no sales approach will be made to them as a direct result of their having provided information.” Definition of market research contained in the ICC/ESOMAR International Code

ESOMAR Eurocenter 2 Barbara Strozzilaan 384 1083 HN Amsterdam The Netherlands Tel +31 20 664 2141 Fax +31 20 664 2922 E-mail [email protected] www.esomar.org

ESOMAR WORLD RESEARCH CODES & GUIDELINES ESOMAR/WAPOR GUIDE TO OPINION POLLS AND PUBLISHED SURVEYS

ESOMAR is the world organisation for enabling better research into markets, consumers and societies. With 5000 members in over 100 countries, ESOMAR’s aim is to promote the value of market and opinion research in illuminating real issues and bringing about effective decision-making. To facilitate this ongoing dialogue, ESOMAR creates and manages a comprehensive programme of industry-specific and thematic events, publications and communications, as well as actively advocating self- regulation and the worldwide code of practice.

Founded in 1947, the World Association for Public Opinion Research (WAPOR, www.wapor.org) is dedicated to advancing the scientific study of public opinion. Its members around the world work to maintain high standards for collecting, analysing, and disseminating public opinion data. Each year, the organisation holds conferences and workshops involving academics, practitioners, journalists, and policy-makers. WAPOR sponsors the multidisciplinary research journal, the International Journal of Public Opinion Research.

ESOMAR WORLD RESEARCH CODES & GUIDELINES