www.researchassociation.org.nz

New Zealand Political Polling Code

Effective June 2020

New Zealand Political Polling Code - page 1

Introduction

This code documents best practice guidelines for public to be confident that political polls do in fact the conducting and reporting of political polls in represent the opinions of the wider public and are New Zealand. a guide as to likely voting behaviours.

The code is binding on companies that are The development of the code is in recognition of members of Research Association New Zealand the fact that reporting of polls can have an impact and on researchers that are members of the on how people vote. Research Association New Zealand. Inaccurate polls or polls that are reported The polling code was first published in 2014 and inaccurately can impact on voting attitudes and updated in 2020. behaviours and thus influence the democratic process. The code only covers “political polls”, which for the purpose of the code are polls that relate to All members of the polling and media communities public votes such as national elections, local body must treat polling responsibly. Reliable polls, elections and parliamentary referenda. rather than informal surveys, require a high degree of rigour. These guidelines are designed to ensure For each issue, the code details: that rigour is understood and applied.

• Best practice for the researcher The guidelines have been developed in the light of conducting the poll the European Society of Market Research • Best practice for the researcher in (ESOMAR) international guidelines. reporting results • Best practice for the media in publishing ESOMAR is the global authority on research best results practice and RANZ has a formal partnership agreement with ESOMAR. The term “must” indicates a requirement, while the term “should” indicates recommended best We recommend that those interested also read practice. the ESOMAR /WAPOR guide to opinion polls and published surveys at: It is intended that the code assist politicians, http://www.esomar.org/uploads/public/knowledg political scientists, journalists and members of the e-and-standards/codes-and-guidelines/WAPOR- ESOMAR_Guidelines.pdf

New Zealand Political Polling Code - page 2

The Code

Conducting Reporting Publishing

Sampling

Size Nationwide polls must use Report must include the Story must include the a sample size of at least sample size, and the sample size 500. sample size of “decided” voters Note that for a maximum margin of error of 3 Due warning must be Due warning should be percentage points, the added that results for added that results for sample size should be at small subsamples (e.g. small subsamples (e.g. least 800 what over 65-females what over 65-females think) are based on small think) are based on small For regional polls or numbers and should be numbers and should be electorate-wide polls treated with caution. treated with caution (Which way is Wellington Central leaning? What are The maximum margin of The maximum margin of the results from the Māori error at a 95% level of error at a 95% level of seats?) the minimum confidence must also be confidence must also be sample size must be at reported alongside reported alongside least 250. sample size. sample size.

Sampling Method Should be either Report must disclose the Story should include the “random”, “stratified sampling method sampling method sample” or “quota” Randomness Researchers should aim to Report should disclose Story should include how maximise the true how the randomly respondents were randomness of the survey selected respondents reached. sample. Although less were reached. Landline, relevant today, one mobile, on-line, door to method is to conduct door multiple call-backs to avoid systematically Report should disclose ‘missing’ hard to reach that multiple call-backs people. have occurred

Avoid self-selection Polls should invoke the Report must state that polls principle of random the poll is of a sampling. Self-selecting representative sample of polls (for example on New Zealanders websites) do not even come close to passing the test of true randomisation. Representativeness The sample should Report should disclose Story should include the represent either those the population the population the sample self-identified as eligible sample represents represents to vote or those likely to vote

New Zealand Political Polling Code - page 3

Conducting Reporting Publishing Likely Voters Unless voting is Report should consider Reporting should consider compulsory (as in excluding those unlikely results in terms of Australia) then the to vote from the analysis reflecting the intentions likelihood of voting is very of voting behaviour. of likely voters. much part of the equation in compiling poll results. The report should include a definition of Those who say they are how it was determined ineligible or unlikely to someone was likely to vote should be excluded vote from the analysis

Response Rates Establishing a good The report should response rate is one include the response rate means of having data you and discuss the can rely on – though the weighting method and importance of this extent of weighting. measure has been challenged. There is no guarantee that a survey with a 40% response rate is significantly more reliable than a survey with a 20% response rate. In addition, the use of panels makes these measures less relevant. However, the benefits include the ability to identify ‘hard to reach’ segments of the population. Response rates also act as a general reliability/confidence indicator. A low response rate may indicate a systematic bias.

New Zealand Political Polling Code - page 4

Conducting Reporting Publishing

Collection Method

Landline Phone When employing random Report must disclose The story should disclose probability sampling, both how a respondent is if calls were to landlines the household dialled, and selected. only the respondent selected in the household should be The report should random. When employing disclose if calls were to

quota sampling, the landlines only, and any

household dialled should impact this may have

be randomly selected, but had on the poll

the person in each household may be selected to achieve specific quota requirements.

Mobile Phone Polling based on Report must identify Published stories should respondent contact via whether one or other or indicate whether the mobile phones either both landline and mobile polling was conducted come from random devices were used as door to door, via landline, dialling or use of a part of the random via mobile or using mixed research panel. (See survey design. methodology below). This should be specified. Online – opt-in The chief challenge for Report should disclose The story should disclose panels panels is to ensure that the use of panel if there were any major participants in a poll are recruitment, the size and platforms that the poll representative of the make-up of the panel was not accessible on wider public. being employed.

No panel member should The report must state be asked to complete the whether the use of the same poll question more panel complies with the than once every six ESOMAR guideline for months, unless the poll is online research. longitudinal and deliberately of the same Not all online devices set of respondents as a successfully handle previous poll, and online surveys. Usually reported as such. this is a very small percentage. But the The final panel sample report should disclose if should reflect a true cross- there were any major section of eligible New platforms that the poll Zealand voters, which may was not accessible on. be fully achieved by screening or by weighting.

The panel should stay open for at least 72 hours.

Researchers should try

New Zealand Political Polling Code - page 5

and minimise people signing up to their panel, just to participate in political polls as such self- selection can bias the result.

The panel should be managed in line with the ESOMAR guidelines for online research Robo-polling This involves auto-diallers Robo-polling results Be very wary of publishing and the use of automated exhibit bias. If such a robo-poll results. Any voice interviews. Overall poll is to be published, such story should disclose robo-polling has been the report must explain that these results will in found to be less accurate that the results were all probability differ to than live interviewer polls gathered using robo- more scientifically due to non-response bias polling methods. designed polls. and self-selection by participants. Quality controls – Researchers must check data cleaning data for (and eliminate,) ‘straight-liners’ and incomplete responses

Omnibus If the political questions The report must disclose The story should disclose are part of a longer if the questions were if the questions were part omnibus poll, they should part of an omnibus of an omnibus survey. be asked early on. survey. Question Order Principal voting behaviour The report must disclose The story should disclose question should be asked the order of questions any other questions which before all other questions asked and any political may have impacted the questions asked before responses to the principal the principal voting voting behaviour question behaviour question. Questionnaire Scientific polls must use Reports must use the Reporters should quote wording language that ensures fair actual wording used in the actual question and unbiased results. the poll. wording.

New Zealand Political Polling Code - page 6

Conducting Reporting Publishing

Weighting

Weighting Researchers should attempt to make their samples representative to obviate the need for weighting. However, weighting provides a final adjustment in attempting to match the sample universe. Weighting Method A random sample poll The report must confirm Story must include that should be weighted using the sample was the sample is weighted. an industry accepted weighted. weighting methodology to correct for the probability of selection and/or non- response.

RIM Weighting should exclude variables that bear a strong relationship with each other.

Weighting Variables A minimum of gender and Report should disclose age should be weighted. which variables the poll Other typical weighting is weighted on variables include region, ethnicity and household income.

When weighting to correct for demographic non- response, the calculated sample weights should be based on known or estimated population characteristics (for example, from Statistics New Zealand or the Electoral Commission). Weighting Extreme weighting can limitations distort poll results. An Report should disclose indication of the spread of the extent of the weights is desirable- e.g. weighting – suggest the 95% of weights are 95% range of weights be between 0.8 and 1.4 utilised as standard

New Zealand Political Polling Code - page 7

Conducting Reporting Publishing

Margin of Error

Maximum Error The maximum sampling The report must disclose The story must include margin of error must be the maximum margin of the maximum margin of no more than ±4.5 error at a 95% level of error at a 95% level of percentage points and confidence. confidence. should be no more than ±3.5 percentage points for national polls, at the 95% confidence level. Maximum errors for The report should Stories should avoid breakdowns disclose the sample size reporting breakdown and maximum margin of results from small error for demographic samples breakdowns. Significance The report should Stories should focus on highlight results that are changes that are statistically significant. statistically significant This includes trend changes, not just from the previous poll. Errors for results < The report should Media should be careful 50% include the maximum not to assign the margin of error for maximum margin of error results below 50%, such to low results. as for 10%. Other Errors Care should be taken to Again, the report must eliminate sources of error state the maximum not associated with the margin of error for a sampling process simple random sample, at the 95% confidence level. Other sampling errors or assumptions should be reported on if deemed significant.

Conducting Reporting Publishing

Timing

Data Collection The final poll before an The report must disclose The story should disclose Dates election should be the dates the data the dates the data conducted as close as collection occurred poll collection occurred possible to the reporting results date Median Date As more responses often The median date of occur earlier in the poll, collection should be the date the median included in the report response was collected should be calculated

New Zealand Political Polling Code - page 8

Conducting Reporting Publishing Influential events Potentially influential Reporting should include events that occurred mention of significant during the conduct of the events that may have poll should be noted. influenced the results. No claim should be made unless there is clear evidence that those events actually drove voter opinion. Results

Undecideds The poll script should The report must state The story should include probe initially undecided the number and the percentage that were voters as to a lean or percentage of undecided undecided preference. and refused Trends Reports should highlight Stories should focus on statistically significant statistically significant trends trends, which may not be just between the current and last poll, but over a number of polls Seats In NZ there is a close fit Reports should include Stories should include between overall election seat projections, and any analysis of likely “bloc” results – seats won under assumptions used for results (not just individual MMP – and credible poll electorate seats. Care party results) as the results. However, a small should also be taken to highest polling party may complication arises if a explain that polls may not be most likely to form Party is polling close to indicate the final result – a Government. Story the 5% threshold. They but do not predict. should state that polls do may – or may not - attain Voters are still capable of not predict – they the 5-6 seats available changing their minds. measure a point in time. Reports The agency must prepare The online version of a report suitable for stories should link to the publication with full full report, as quickly as results and practical. methodology. The report must include the wording of the voting questions and all the other information required by this code.

The report must be published on the conducting agency’s website.

New Zealand Political Polling Code - page 9

Term Definition Terminology A subset of individuals (a sample) chosen from a larger set (a population). Each individual is chosen randomly and entirely by chance, such that each individual has Random sampling the same probability of being chosen at any stage during the sampling process, and each subset of k individuals has the same probability of being chosen for the sample as any other subset of k individuals. When subpopulations within an overall population vary, it could be advantageous to sample each subpopulation (stratum) independently. Stratification is the process of dividing members of the population Stratified random sampling into homogeneous subgroups before sampling. The strata should define a partition of the population. That is, it should be collectively exhaustive and mutually exclusive: every element in the population must be assigned to one and only one stratum. In quota sampling, a population is first segmented into mutually exclusive sub-groups, just as in stratified sampling. Then judgment is used to select the subjects or units from each segment based on a specified Quota sampling proportion. For example, an interviewer may be told to sample 200 females and 300 males between the age of 45 and 60. This means that individuals can put a demand on who they want to sample (targeting). Surveys where the sample selects itself (for example, Self-selection poll when a website seeks views from those who are browsing it). A political poll that has a hidden agenda of tilting the Push poll respondent toward one party or another. Characterised by loaded questions.

New Zealand Political Polling Code - page 10

Describing polls Polling Best Practice Guidelines for Media A sample standard paragraph or box which media can use in stories which conforms to 1. If possible, get a copy of the full poll this code is: report and do not rely on a media release. This poll was conducted by Agency Name for 2. The story should include the name of Media Name. It is a sample method poll of the company which conducted the sample size sample populations and is poll, and the client the poll was done weighted to the overall adult population. It was for, and the dates it was done. conducted by method between dates, has a 3. The story should include, or make maximum margin of error of +/- x% and y% available, the sample size, sampling were undecided on the party vote question. method, population sampled, if the The full results are at URL. sample is weighted, the maximum margin of error and the level of An example would be: undecided voters. 4. If you think any questions may have This poll was conducted by Kiwi Research for impacted the answers to the principal Kiwi Media. It is a random poll of 1,000 adult voting behaviour question, mention New Zealanders likely to vote and is weighted this in the story. 5. Avoid reporting breakdown results to the overall population of adult New from very small samples as they are Zealanders. It was conducted by phone unreliable. between 15 and 19 June, has a maximum 6. Try to focus on statistically significant margin of error of +/- 3.2% and 8.5% were changes, which may not just be from undecided on the party vote question. The full the last poll, but over a number of polls. results are at 7. Avoid the phrase “This party is below www.kiwiresearch.com/polljune2020.pdf. the margin of error” as results for low polling parties have a smaller margin of error than for higher polling parties. 8. It can be useful to report on what the electoral results of a poll would be, in terms of likely parliamentary blocs, as Compliant polls the highest polling party will not necessarily be the Government. 9. In your online story, include a link to Polls following the code are entitled to use the the full poll results provided by the emblem below to signal their compliance. polling company, or state when and where the report and methodology will be made available. 10. Only use the term “poll” for scientific polls done in accordance with market research industry approved guidelines, and use “survey” for self-selecting surveys such as text or website surveys.

New Zealand Political Polling Code - page 11

New Zealand Political Polling Code - page 12