Web Survey Methodology Mario Callegaro, Katja lozar Manfreda & VaSja VehoVar

00_Callegaro et al_Prelims.indd 3 4/21/2015 4:00:16 PM 6 CONTENTS

6.1 Broader methodological context 232 6.1.1 Web survey mode revised 232 6.1.2 Total survey error and survey data quality 233 6.1.3 Evaluation of the web survey mode effect 235 6.1.4 Cost-error optimization 236 6.1.5 Beyond the core web survey process 237 6.1.6 Towards interactive fieldwork design 240 6.2 Web surveys within the project management framework 242 6.3 The web survey profession 244 6.3.1 Related disciplines 245 6.3.2 Legal issues 246 6.3.3 General ethical concerns 246 6.3.4 Formalized certificates, codes and standards 247 6.3.5 Informal guides, recommendations and other material 248 6.4 Web survey bibliography 249 Notes 252 252

06_Callegaro et al_Ch-06.indd 230 4/21/2015 4:00:33 PM Broader context of web surveys

06_Callegaro et al_Ch-06.indd 231 4/21/2015 4:00:33 PM WEB SURVEY METHODOLOGY

In previous chapters we focused on the essen- features and procedures of the measurement stage. tial methodological aspects of the core web These attributes define the basic mechanisms of survey process (Chapters 2, 3, 4) and on communication, including information transfer related implementations (Chapter 5), while in between the respondent and questionnaire. While this chapter we will discuss the broader meth- different authors (Biemer & Lyberg, 2003; Couper, odological (Section 6.1), managerial (Section 2011; Groves et al., 2009; Tourangeau & Brad- 6.2) and professional (Section 6.3) contexts, burn, 2010) identified different dimensions of the which are all very important for understand- survey mode, recently Berzelak (2014) defined the ing and managing the process of web surveys. survey mode by six inherent mode characteristics. We also overview the web survey bibliography These are the characteristics that are all under a (Section 6.4). researcher’s control, are stable across actual sur- vey implementation and have an impact on the response process. We briefly outline them below, building on our discussions in Chapter 1, together 6.1 BROADER METHODOLOGICAL with their options, where we underline those that CONTEXT relate to the basic web survey mode:

We first address aspects that have considera- a Interviewer involvement: interviewer ble indirect methodological relevance for web administered, self-administered without the surveys and provide a closer look at the defi- presence of an interviewer, self-administered nition of the web survey mode (Section 6.1.1), with the passive presence of an interviewer data quality (Section 6.1.2), web survey mode (e.g. CASI, see Section 1.3.4). effect (Section 6.1.3) and costs (Section 6.1.4). b Usage of computer technology: interactive In addition, we review the related activities that computerized questionnaire, non-interactive extend beyond the core web survey process computerized questionnaire, P&P recording (see Section 1.1 and Section 1.3.2). (Section 6.1.5): the steps of preliminary research c Information transmission medium (i.e. activities and advanced analysis, processing delivery of the completed questionnaire and valorization, as well as mixed-modes and from the respondent to a researcher): F2F, mixed methods. We conclude with a discussion telephone, mail, email attachment, email on adaptive, responsive and interactive designs embed, automatic online delivery (e.g. via the (Section 6.1.6). Internet or some other electronic network, see Section 1.3.1). d Question presentation input channel: auditory, visual, both (see Section 1.3.4 and Section 1.1). 6.1.1 Web Survey Mode Revised e Response output channel: oral, manual hand- Our initial definition (Section 1.1.1) of the basic writing, manual electronic (mouse, keyboard, pointer, stylus, touch screen) (see Section web survey mode served as an introduction 1.3.4 and Section 1.1). to this mode, but now we can outline it more f Closeness of the interviewer interaction: precisely. Firstly, let us say again that we con- physical, remote audio or video, pre-recorded ceptualized the survey mode as being entirely audio or video, virtual audio or video separate from the sampling and also from the (e.g. virtual interviewer), no interaction. recruitment process. When we defined it, we thus considered only factors with a direct impact Moreover, there are also other survey mode char- on the measurement stage, where respondents acteristics, related to various implementation fill in the questionnaire. and contextual specifics, which are mostly out The literature typically defines the survey of a researcher’s control and vary across imple- mode with certain attributes (dimensions, fac- mentations, interviewers and respondents. As tors, characteristics) that are related to the such they do not define the survey mode, but

232

06_Callegaro et al_Ch-06.indd 232 4/21/2015 4:00:33 PM BROADER CONTEXT OF WEB SURVEYS

rather the specific implementations in a partic- inherent mode characteristics – self-­ ular survey. Examples include the pace of the administration (a), interactive computerization interview, cognitive requirements, technology (b), online delivery of responses (c) and (e.g. device type, screen resolution, sound absence of interviewer interaction (f) – define quality, etc.), environmental distractions, an even broader family of online CSAQ sur- multitasking of respondents, sense of imper- vey modes, where voice technologies are also sonality, and characteristics of the interviewer involved. When the web is used specifically and the respondent. Implementations may also as the transmission medium for the automatic differ across various other technical aspects, online delivery of responses, then we speak which often have no or negligible impact on about the web-related survey modes. This is the survey process, such as the location of the case with some mobile survey applica- the server (e.g. SaaS), web survey software, tions, where the web serves only for receiving as well as client-side (e.g. surveys run as the questionnaire and sending it back to the mobile applications) vs server-based surveys. server. Correspondingly, the use of smartphones and The above clarifications demonstrate the tablets for answering web questionnaires does large variety of possible modes and show that not introduce a new survey mode, and when the case of the basic web survey mode is in fact a respondent is using a desktop/notebook, only a very specific example. Nonetheless, it tablet or smartphone to answer the web ques- bears repeating that the vast majority of contem- tionnaire, we still speak about the web survey porary web-related surveys are conducted with mode. This does not mean that the mobile this mode. More detailed elaboration, examples device does not cause any specific device and graphical presentations of survey modes effects (see Section 5.1.3), or that we do not can be found in the Supplement to Chapter 6, need to consider also the case of intentional Survey modes (http://websm.org/ch6). device-specific web survey implementations (e.g. online panels using mobile survey appli- cations). The six dimensions define hundreds 6.1.2 Total Survey Error and Survey of different survey modes. The underlined Data Quality options in (a)–(f) above specify a survey mode, which has the characteristics of the basic web The notion of total survey error (TSE) is an survey mode, but in fact relates to a broader umbrella term covering all types of survey family of basic online CSAQ (which we intro- errors. Certain classification and terminological duced in Section 1.3.2) survey modes, where differences exist with respect to the treatment of the input (d) is restricted to the visual option TSE by various authors, as discussed in Groves and the output (e) to a manual electronic out- & Lyberg (2010), and continuous revisions also put. Within this context, the basic web survey appear, for example Smith (2011) provided a mode is only a specific subtype of basic online much elaborated typology of TSE components. CSAQ, with the following three sub-charac- Nevertheless, the following components of sur- teristics: the online transmission medium (c) vey errors are most often exposed: sampling, is the Internet; server-based communication is coverage (frame), nonresponse, specifica- used for interaction; and the interface is based tion, measurement and processing (Biemer & on web browsers. Thus, the basic web sur- Lyberg, 2003). Each type of error can have vey mode excludes surveys which run within systematic and random components, expressed mobile applications, independent of brows- as bias and variance respectively. The TSE is ers and the interaction with the server (see defined and conceptually elaborated on only Section 5.1.5). within a probability sampling context; how- When we remove the restrictions for input ever, in practice the approximations are also (d) and output (e) channels, the remaining four used, as outlined in Section 2.2 on sampling.

233

06_Callegaro et al_Ch-06.indd 233 4/21/2015 4:00:33 PM WEB SURVEY METHODOLOGY

The mode effect is a specific subtype of and where smaller sampling variance then out- measurement error and also a subcompo- balances the potential bias, which often appears nent of the TSE. It appears as a result of the in web surveys due to higher nonresponse and/ influence of survey mode on estimates. The or non-coverage. corresponding differences in response distri- While survey errors integrated into the TSE butions are caused by inherent characteristics relate to the statistical characteristics of the of the specific mode. They are obtained when estimates (e.g. accuracy), the notion of survey administering the same question in different quality is much broader (Lyberg, 2012). It includes survey modes (e.g. F2F vs the web). Biemer additional aspects of survey outcomes (statistical & Lyberg (2003) used the more precise term results), such as relevance, comparability, pure mode effect to contrast a broader notion coherence, timeliness, accessibility and clarity. of the mode system effect, which also includes An even broader framework of data quality other related error sources potentially accom- involves trust and the perception of the users, as panying certain survey modes (e.g. sampling, well as the characteristics of the related production nonresponse, recruitment, incentives, research process (Ehling & Körner, 2007). Similar design context). For example, these mode sys- processing components are included in the ISO tem effects refer to situations when a specific (2012) standard on the survey process. Various survey mode is not assigned to the units ran- general quality management approaches can also domly, but is chosen by respondents through be applied here, such as Six Sigma statistical an unknown selection mechanism, which is process control and total quality management then difficult to separate from the pure mode (TQM). Dedicated survey data organizations (e.g. effect. statistics offices) typically develop their own – Each component of the TSE can damage often very elaborate – systems to monitor the the statistical quality of the estimates used to data quality via standardized quality reports (e.g. infer the population parameter. Most frequent Eurostat, 2009). Nevertheless, the specific mode indicators of these deviations are related to effects are a component of these general systems sampling precision (random oscillations, on which rather little and weak elaboration is expressed with sampling variance), bias (dif- available in the literature, though we believe this ference between the true population value will change with the increased usage of mixed- and the expected value of the estimate), reli- mode surveys. ability (consistency of responses in repeated Particularly for large and complex surveys measures) and validity (degree to which we as well as for decision support surveys, these actually measure the concept we truly wanted broader survey quality aspects are very impor- to measure). tant. For example, a failure to deliver the results Very often we also use the notion of accu- at a certain time (e.g. at a decision point) means racy, expressed by the mean squared error the entire survey was useless, regardless of its (MSE), which estimates various components accuracy and otherwise high quality. Researchers of the TSE (Dietrich, 2008). The MSE is the with a very narrow focus on the methodological sum of random variances and squared biases details of the web data collection process may of the related components included in the TSE. sometimes overlook these additional and very However, it is difficult to deal with the random important components of data quality, and con- and systematic errors of all TSE components. sequently forget to look at the big picture. Thus, In practice the discussion is most often reduced it is advisable to include aspects of data quality to the sampling variance and the biases arising explicitly in the conceptualization, operation- from nonresponse or non-coverage. Minimiz- alization and planning of web surveys, as well ing the MSE can justify the use of web surveys as in the regular managerial monitoring of the which can afford inexpensive larger samples, process.

234

06_Callegaro et al_Ch-06.indd 234 4/21/2015 4:00:33 PM BROADER CONTEXT OF WEB SURVEYS

6.1.3 Evaluation of the Web Survey (CAPI, n = 7,204) of the Dutch general popu- Mode Effect lation. They found considerably higher drug usage in the online panel (e.g. 39% vs 28% for The complexities of the survey mode and its ever using cannabis in the age group 15–24). interaction with other types of survey errors However, it is unclear whether these differences make the empirical evaluation of mode effects come from the mode (respondents admit more very demanding. As a consequence, there are drug use in the web vs F2F survey mode) or numerous studies in which we cannot truly iso- from differences with respect to nonresponse, late the pure mode effect. The mode effect can self-selection, sampling, non-coverage or dif- also be contaminated with a poorly designed ferent populations involved. The mode effect questionnaire (e.g. incorrect scale format, in many studies is often confounded by frame excessive colours or pictures, etc.), respondent problems (Burr, Levin, & Becher, 2001; Tomsic, failures (e.g. low computer literacy), technical Hendel, & Matross, 2000), different samples problems (e.g. device failure) and other mistakes (e.g. Braunsberger, Wybenga, & Gates, 2007; in the web survey implementation. Most impor- Chang & Krosnick, 2009; Roster et al., 2004), tantly, pure mode effects often interact with the availability of email addresses (Deutskens, nonresponse, non-coverage and sample selec- de Ruyter, & Wetzels, 2006; Griffis, Goldsby, tion mechanisms, which are then very difficult & Cooper, 2003), and sequential mixed-mode to separate. designs for nonresponse reduction (Carini, When comparing survey modes, one mode Hayek, Kuh, Kennedy, & Ouimet, 2003). These is usually declared (or implicitly assumed) to uncontrolled differences can be wrongly attrib- be the accurate one (i.e. the gold standard). uted to pure mode effects. Very often this is F2F. We need to be very A standard approach towards isolating the careful about such situations and rather speak pure mode effect is to use experiments and about the between-mode effect or, better, randomly assign units to subsamples with between-mode difference, unless we know different implemented modes. Nevertheless, the true value of the responses (e.g. via aux- nonresponse and considerable random errors iliary data). The between-mode differences still cannot be separated from the mode effect are usually observed as simple differences of (Vannieuwenhuyze & Loosveldt, 2013). Pop- estimates of means among alternative imple- ulation weighting might help, but only to a mentations, such as experimental groups. limited extent which is usually unknown (see Further, mode effect studies can also focus Section 4.1.4). Some specific approaches (Van- on correlations or measurement equiva- nieuwenhuyze, Loosveldt, & Molenberghs, lence (Revilla, 2013), reliability (Buchanan 2010, 2012) which use single mode groups & Smith, 1999; Chang & Krosnick, 2009; within the mixed-mode context may partially Hertel et al., 2002; Mangunkusumo et al., reduce this problem. 2006; Miller et al., 2002), latent struc- An even better research design is randomi- tures (Buchanan & Smith, 1999; Deutskens zation in a laboratory, where the participants et al., 2004; Roster et al., 2004) and so-called are randomly assigned to experimental groups concurrent validities (Linchiat Chang & Kro- only after arrival at the laboratory, as in Chang snick, 2009; Hertel et al., 2002). & Krosnick (2010). Similarly, Jäckle, Rob- A typical example of a study comparing sur- erts, & Lynn (2006) reported an experiment vey modes, but not isolating the pure mode where participants were recruited on the street effect, is the Spijkerman, Knibbe, Knoops, and, after they agreed to cooperate, they were Mheen, & Eijnden (2009) study comparing randomly assigned to one of four modes. results from a non-probability online panel With this approach we can additionally elim- (n = 57,125) with a probability F2F survey inate the potential differences in nonresponse.

235

06_Callegaro et al_Ch-06.indd 235 4/21/2015 4:00:33 PM WEB SURVEY METHODOLOGY

However, external validity issues arise; that is, On the other hand, specific web survey mode whether the findings in artificial (laboratory) effects may sometimes appear in the negative experimental conditions also hold in real-life direction, predominantly due to various forms conditions. of shortcutting: satisficing, excessive item non- An alternative approach to measuring the pure response (or don’t know) selection, speeding, survey mode effect is re-surveying (test–retest or inconsistent responses, etc. Most often this is a repeated measurement), which can effectively consequence of the absence of the interviewer, control for sample, coverage and nonresponse whose presence is sometimes indispensable for differences between groups. In addition, this motivation (e.g. long surveys) and clarification sharply reduces the sampling variance, because (e.g. complicated questions). Nevertheless, a we measure the same units in both waves, and careful combination of incentives and interac- dramatically increases the power of discovering tive questionnaire design may compensate for potential differences (e.g. Alwin, 2007). How- the absence of the interviewer to a considerable ever, practical implementation is complicated extent. and expensive, as we can infer from the lim- ited number of studies with this implementation (Mangunkusumo et al., 2006; E. T. Miller et al., 6.1.4 Cost-Error Optimization 2002; Rivara, Koepsell, Wang, Durbin, Jaffe, Vavilala, … Temkin, 2011). In addition, respond- When discussing differences in survey modes ents may remember the same question from and survey data quality in general, cost aspects the first wave and some external effects/events are very often omitted. The question of whether between the two measurements may change the it is accurate and fair to compare the web sur- value of the target variable. To control for these, vey mode to the alternatives without controlling another experimental group would be needed (i.e. for costs then arises. That is, only comparisons an ‘after’ control group without treatment of the under a fixed budget – and not under the usual first wave). An additional problem may arise due practice of fixed sample sizes (e.g. n = 1,000 for to the effect of the order in which survey modes each mode) – would possibly show that web sur- are applied; to control properly for it, yet another veys allow us to afford much larger sample sizes. experimental group would be needed with the Alternatively, the comparative savings from web opposite order of treatments. surveys may be invested in incentives, mixed- Many studies demonstrate that the web survey modes and other improvements, which can mode is usually superior, or at least equivalent substantially increase data quality and change to conventional interviewer-administered (F2F, the results of comparisons. The absence of cost telephone) and also to P&P self-administered considerations in the literature is even more sur- modes (e.g. Börkan, 2010; Braunsberger et al., prising, because the cost is very often the most 2007; Buchanan & Smith, 1999; Chang & Krosnick, essential factor when we decide on which survey 2009; Hertel et al., 2002; Mangunkusumo et al., mode to use. 2006; Roster et al., 2004). More specifically, a We have already mentioned certain spe- meta-study by Tourangeau, Conrad & Couper cifics (Section 1.3.5) of fixed costs in web (2013) showed that with respect to sensitive surveys, which are often relatively independ- behaviour, which is usually under-reported in ent of sample size, while variable costs per unit traditional survey modes, respondents tend are comparatively low and depend on sample to answer more realistically in web surveys, size. The computation of total costs per minute where no interviewer is present. We have of respondent time is a great tool to compare already asserted (Section 1.3.5) that self-­web surveys with traditional alternatives. For administration is advantageous for other example, general population probability-based reasons (e.g. respondents select their own time F2F surveys, such as general social surveys, and pace when filling in the questionnaire). have typical per minute costs of a few euros/dollars

236

06_Callegaro et al_Ch-06.indd 236 4/21/2015 4:00:34 PM BROADER CONTEXT OF WEB SURVEYS

in developed countries, sometimes even above following components, which we present in a 5 or 10, while probability online panels often simple, unstructured and non-exhaustive list: have total per minute costs roughly in the interval of 1–2 euros/dollars. Web survey responses from • clarifications of general issues, such as initial non-probability samples can have much lower research ideas, the broader (research, business, costs per minute, which strongly varies, depend- administrative) framework, basic paradigmatic and conceptual settings, problem formulation, ing on the quality. The costs need to be balanced goal identification and research problem defi- with data quality, which is a very complex prob- nition, together with elaboration of the relation lem, and can be additionally complicated when between the research problem (e.g. how to modes are mixed. Conceptually, these issues were measure customer satisfaction) and the even- addressed relatively early (e.g. Groves, 1989), but tual decision problem (e.g. how to increase in practice we still lack effective operational cri- customer satisfaction); teria and procedures. An approach to optimizing • further elaboration of research design aspects, cost and accuracy simultaneously – which means such as a more specific outline of the key con- minimizing the product of costs and MSE – has cepts, operationalization of the constructs, been discussed by Vehovar et al. (2010). A specific development of research hypotheses, speci- approach was presented by Roberts, Vandenplas, fication of research methods used, definition of the target population and sometimes the & Stähli (2014) and Vannieuwenhuyze (2014), selection or elaboration of key questions; while Schouten, Calinescu, & Luiten (2013) dis- • ethical considerations, which can also involve cussed cost optimization within the context of the potential formal approval process; responsive design, where cost-error optimization • key administrative, management and process directly affects sequential fieldwork procedures. quality issues, including essential aspects of timing, costs and other resources; • in the case of complex research designs, 6.1.5 Beyond the Core Web Survey certain forms of preliminary research activ- Process ities are conducted here, particularly desk (secondary) research, qualitative studies and So far we have observed the core web survey expert consultations; process as being entirely isolated and within • additional activities appear whenever the role a simplified flow of pre-fielding, fielding and of a researcher – whom we have treated so post-fielding. In reality this is rarely the case – far as one entity – is split across the sponsor, web surveys are usually more complicated and the client, research provider, the software supplier, etc. (see Section 1.2), which then nested in the larger research context, so we review greatly complicates the preliminary research a few of the most essential aspects below. activities and requires additional actions, interactions and processes. 6.1.5.1 The Step of Preliminary Research Activities Although not all of the above activities appear in every survey, the majority of them do, at least We defined preliminary research activities in in some unstructured and informal way. Even in Section 1.2 (Figure 1.1) as preceding the core simple DIY research, certain decisions and activ- web survey process. What separates them from ities are needed for the majority of these issues to pre-fielding is that they are not directly related or find corresponding solutions. In more complex involved in the specifics of web surveys. Instead, research projects, the methodological aspects they are concerned with general preparations, are usually elaborated in a formal research which are independent of any survey mode. Nev- design, which specifies the research methods ertheless, they can be very important for the entire used, together with the stages, sequence and context of web surveys. These preliminary research other specifics of the related research activities. activities typically – but now always – include the In general, various qualitative and quantitative

237

06_Callegaro et al_Ch-06.indd 237 4/21/2015 4:00:34 PM WEB SURVEY METHODOLOGY

research methods can be involved, according to and Sue & Ritter (2012), which do provide some the exploratory, descriptive or causal nature of insight into the specifics of the tendering process. research. The role of the web survey needs to be determined within the general research design. Of course, some of the above preliminary 6.1.5.2 The Step of Advanced Analysis, research activities can overlap with pre-fielding Processing and Valorization to a certain extent, particularly the early elabo- We have already defined this step in Section ration of methodology (e.g. the questionnaire is 1.2 (Figure 1.1) as the one belonging to broad already predetermined by the client). research activities, not directly related to the Another type of formalization relates to a specifics of web surveys. Its activities thus fol- research proposal, which outlines all essential low the core web survey process and we also aspects of the proposed research, not just the specified the corresponding separation from the methodological aspects. Examples of good prac- post-fielding step in Chapter 4. tical elaborations can be found in the book by The corresponding activities predominantly Iacobucci & Churchill (2015). Very often this relate to advanced analyses, which include proposal then undergoes a certain approval pro- reporting, statistical procedures, variance esti- cess from management, client, sponsor or other mation, substantive interpretations and the funding body. preparation of reports and presentations. A research proposal is particularly relevant in This step also includes various advanced situations where the role of the research is split forms of post-survey data processing, such among various entities and their relationships as integration into certain management and need to be specified. In this context, various decision-making procedures within broader forms of tenders or research grant calls are very administrative or business processes. Survey frequently involved, requiring additional and data can be subject to further manipulations, careful administrative elaboration (application from data merging, record linking, statistical forms, specification of processes, legal aspects, matching (i.e. data fusion) and several other uses. selection process and criteria). In addition to the These manipulations are increasingly becoming price, quality indicators of the web survey pro- an important element in advanced research and cess (e.g. response rate) should play a decisive administrative or business processes. role whenever a web survey is involved. The valorization component includes further Since preliminary research activities are exploitation of results (usage and reusage in other typically very complex, structured and survey research or decision processes) and dissemina- specific, the literature rarely discusses them tion of results by public archiving of the datafile as one entity. Instead, various components or and by distribution, promotion and publication of aspects are discussed separately (e.g. research the findings (in newsletters, briefs, documenta- paradigms, research methods, procurement tion, full reports, as well as in papers and books). procedures, legal aspects). For a more informed view of the methodological aspects of prelimi- nary research activities, the reader can consult 6.1.5.3 Mixed-modes the general literature on survey methodology (e.g. Blair, Czaja & Blair 2014; Groves et al., 2009), We introduced the notion of mixed-modes in social science methodology (e.g. Babbie, 2013) Section 2.1 as a combination of survey modes or discipline-specific literature (e.g. for market- in the measurement stage, and we used the term ing research; Iacobucci & Churchill, 2015). Other mixed-mode system to also encompass multi- aspects are rarely elaborated specifically for sur- mode contacts in the recruiting stage, as well as vey contexts. Exceptions are the publications by other aspects and consequences of using more Iacobucci & Churchill (2015), Iarossi (2006), communication channels in the fielding step.

238

06_Callegaro et al_Ch-06.indd 238 4/21/2015 4:00:34 PM BROADER CONTEXT OF WEB SURVEYS

In addition to this, we address certain broader ESS mixed-mode experiment in Estonia (Ainsaar implementation aspects. et al., 2013) and the UK (Villar, 2013). Firstly, note that, with business surveys, the Let us present a typical example which illus- web survey mode and the P&P (or telephone) trates well the problems with mixed-modes. A survey mode are often routinely combined, recent tobacco, alcohol and drug survey (aver- typically in a standard sequential manner, pro- age length 40 minutes) from the Slovenian ceeding from the inexpensive (web) mode to the Institute of Health1 used a standard sequential more expensive (mail, telephone) one. mixed-mode survey approach (web → telephone Secondly, we should point out that, despite → F2F) and obtained a cumulative response rate various claims and indices of the growing use of 20% after the web, 45% after the telephone of mixed-modes – understood as a combination and 60% after the F2F wave. Response rates and of survey modes in the measurement stage – this cost savings were a big success compared with in general is not the case, at least for marketing the standard alternative of F2F as well as the research and for the DIY segment. In marketing inclusion of segments which would not cooper- research (Macer & Wilson, 2013, p. 40), mixed- ate in traditional survey modes. However, mode mode surveys have a relatively low share of effects remained unclear. For example, canna- revenue (6% for mixed-mode compared with bis consumption (ever using cannabis) – after 52% for web surveys). In addition, the share has weighting for socio-demographic­ population already been very stable for the last seven years for each mode segment – was 25%, 10% and and no future increase is foreseen. This is not 9% for the web, telephone and F2F mode seg- surprising, since high-quality mixed-modes are ments respectively. The problem is that it is now complicated and typically very expensive, com- impossible to isolate mode effects from selection pared with the stand-alone alternatives of web, (wave) of the mode, differential nonresponse mail or telephone surveys, which currently dom- bias, stage of survey process and differences inate in the field of marketing research. in survey population. The future impact of the The situation is different with expensive increasing percentage of responses collected probability-based F2F household surveys. Here, via web surveys on trends is thus unclear. If the due to problems related to sampling, recruiting, estimates of cannabis consumption increase, we non-coverage and nonresponse, web surveys might not know whether this is because people cannot simply replace traditional surveys modes, truly consume more, or only because more and but can enter into the survey process through more people respond on the web, where they mixed-modes. However, as discussed in Sec- may admit more consumption. tion 2.1, this is methodologically complicated. A possible solution to this problem would Nevertheless, it seems that – after initial imple- require parallel experiments where units would mentations (e.g. Beukenhorst & Wetzels, 2009) – be surveyed only F2F, so that pure mode and mixed-modes involving a web survey mode are between-mode effects could be estimated. Alter- slowly becoming mainstream in these types of natively, re-interviewing the units with a different surveys (Buelens & van den Brakel, 2014). survey mode can be used. These approaches were An important mixed-mode challenge in official studied in experiments run by Klausch, Hox, & statistics relates to the Labour Force Survey, where Schouten (2014), where various mode effects continuous progress can be observed in terms of were then isolated. A possible solution is also the introducing web surveys as a mode of data col- involvement of advanced weighting approaches lection, as in Körner & van der Valk (2011) or the (Buelens & van den Brakel, 2014). It seems that DCSS project (Blanke & Luiten, 2014). we need to use both approaches, the involvement In academic surveys we can also observe pro- of some modelling assumptions based on exter- gress in considering mixed-mode data collection, nal estimates (e.g. obtained by reinterviewing) of but with a high degree of caution, as shown in the mode effects and also advanced weighting.

239

06_Callegaro et al_Ch-06.indd 239 4/21/2015 4:00:34 PM WEB SURVEY METHODOLOGY

6.1.5.4 Mixed Methods Mixed methods are particularly convenient within the context of so-called crowdsourcing Mixed methods refer to the research design where and various other approaches in online commu- we combine quantitative methods of primary data nity studies, where corporations and marketing collection (e.g. surveys) with certain qualitative research companies establish and maintain an methods of primary data collection (e.g. in-depth online community of the target population for interviews, focus groups, expert interpretations, their qualitative and quantitative data collection. story-telling, action research, ethnography and brainstorming). As with surveys, qualitative research techniques are increasingly moving to 6.1.6 Towards Interactive Fieldwork the web (e.g. online in-depth interviews (via chat) Design or online focus groups). There exist extensive paradigmatic, concep- According to the initial scope of discussions tual and operational discussions about these (Section 1.2), we predominantly discussed issues, and sometimes mixed methods even cross-sectional web surveys in previous chap- stand as the third research paradigm, in addition ters, relying on a single measurement session. to qualitative and quantitative research (Johnston, The fieldwork process is relatively simple in 2007). In recent years we can also observe an such situations, particularly when we have list- expansion of mixed methods in the literature and based web surveys, using an email invitation and in practice (Morgan, 2013). a reminder that is sent to all units. We should add that within the web survey On the other hand, complexities can arise from process researchers already rely on qualitative more sophisticated designs. Survey data can be insights for various methodological reasons collected in a series of replications of an identi- when formulating a problem, developing a ques- cal survey on independent samples, but also with tionnaire or evaluating survey questions. For the sequences of complementary surveys on inde- substantive aspects researchers typically involve pendent samples, and of course in various types qualitative research in the interpretations, using of longitudinal designs. In addition, different expert evaluation or qualitative feedback from external data can be iteratively incorporated into the target population. the web survey process, namely administrative The web is extremely suitable for the com- (e.g. using a certain form or enquiry of a citizen), bination of approaches. With traditional survey business (e.g. commercial transactions of con- modes, the combination of methods beyond sumers) or technical (e.g. log file information of two steps (e.g. focus group → survey) very certain actions performed by the user on the web). often requires a lot of time and resources. In Another level of complexity appears when a contrast, the web enables easy, inexpensive and mixed-mode system is involved and particularly rapid mixing of qualitative and quantitative when various interactive fieldwork strategies methods. Lobe & Vehovar (2009) showed that replace more or less fixed recruitment plans with sequences of qualitative (online chat) and quan- limited implementation variations, as discussed titative (web survey) methods can iteratively in the recruitment stage (Section 3.1). sharpen the depth and quality of the research Adaptive design (Schouten et al., 2013) builds and can provide a comparable advantage: four on the fact that survey fieldwork in general faces steps (qualitative → quantitative → qualita- many uncertainties, which could be reduced by tive → quantitative) may thus outperform one using the data from the fieldwork itself. That step (either qualitative or quantitative) and also is, very often all units in the sample receive the more standard two- (qualitative → quantita- same treatment, which means the same number tive) or three-step (qualitative → quantitative of reminders, the same modes of invitation and → qualitative) research designs within the the same survey measurement mode. Certain resources provided. auxiliary data (such as age, region, cost structure,

240

06_Callegaro et al_Ch-06.indd 240 4/21/2015 4:00:34 PM BROADER CONTEXT OF WEB SURVEYS

estimates of response rates) are used to structure Figure 6.1, updated from Vehovar & Batagelj and optimize the sampling or the research design (1996), conceptualizes the potential processes in (e.g. stratification, multi-stage sampling, dou- interactive fieldwork design within the mixed- ble sampling, repeated survey, mixed-modes). mode setting. The flow refers to information and However, this is done in advance and does not data communication processes at the unit level, incorporate feedback from the fieldwork. which is different from the conceptualization Adaptive design is often discussed within the of the general web survey process in Figure 1.1 narrower context of responsive design (Groves & (Section 1.2). The centralized processing and Heeringa, 2006), which additionally requires that monitoring interactively links the two essential the survey is structured in phases, where data from fieldwork activities: recruitment and measurement the previous phase are then used to improve the (i.e. questionnaire completion). Different survey next phase (e.g. Wagner, West, Kirgis, Lepkowski, modes can be used to recruit respondents (i.e. Axinn & Kruger-Ndiaye, 2012). Typically, certain multimode contacting) and contact attempts can subgroups may report low response rates, so with be repeated until it is decided that a response is not responsive design additional resources (e.g. more possible (e.g. we learn that the unit is ineligible), contact) are allocated in the next phase. the fieldwork time has finished, or it is not rea- It is surprising that adaptive and responsive sonable to try any further to contact a certain unit. designs are mostly discussed with F2F or tele- The decision on the latter can depend on simple phone surveys (Schouten et al., 2013), while rules (e.g. a maximum of five contact attempts), implementations with web surveys are rather on the nonresponse context (e.g. minimal response rare, despite the fact that rich paradata, flexi- rate or minimal number of responses in a certain bility, ICT support and elaborate monitoring segment), on statistical parameters (e.g. required actually provide a much more convenient setting precision of certain estimates) or on certain com- (Bianchi & Biffignandi, 2014). plex cost-error criteria, which optimize the data We extend here the notion of adaptive design to quality by cost (e.g. product of costs and MSE). even more general interactive fieldwork design, Different decisions can be applied to different which means any usage of data generated during units in order to optimize timing, costs and data (a) fieldwork at the (b) unit level – namely, survey quality. For example, if there were units with no data, paradata and external data (technical, admin- response after two email invitations, intelligent istrative, business) – to improve the (c) real-time centralized processing – which disposes with all optimization of the fieldwork procedures related auxiliary information about the unit (e.g. age, to the recruitment and measurement stages. gender) and also all fieldwork data, paradata The line between implementation variability and external data – might automatically decide, adaptations discussed in Section 3.1 and inter- based on some optimizing algorithm, on the active fieldwork design can sometimes be very optimal timing, mode and format for the next thin, as, for example, in the case where we pre- contact attempt. It might be optimal to send the specified the exact criteria in the recruitment plan, next email reminder in a week or two, or rather when low response rates would require activation to send a mail reminder. Similarly, the most of a supplementary sample, sending additional appropriate survey mode can be a P&P question- reminders or launching an additional promotion. naire instead of a web questionnaire. The main distinction is that interactive design The centralized ICT-supported processing also includes interventions at the unit level, not can strongly improve integration, interactivity just at the level of segments. For example, it and optimization, particularly because it can is not that a mail reminder is simply sent to all incorporate experiences from past surveys and nonrespondents who have already received three involve statistical and decision models. We may email invitations, but that additional information further optimize the structure of the measure- at unit level is used to treat each unit separately ment session at the unit level, which can be split and decide specific actions at that level. into more sessions.

241

06_Callegaro et al_Ch-06.indd 241 4/21/2015 4:00:34 PM WEB SURVEY METHODOLOGY

Centralized processing of eldwork

Questionnaire Decision for completion recruiting Unsuccessful recruiting outcome attempt

Recruitment Measurement Researcher contacts Respondents complete the potential respondents web survey questionnaire (or individually (e.g. email or Successful other CASIC mode or mail, mail, web, telephone, recruiting telephone, F2F mode) F2F mode) attempt

Figure 6.1 Unit-level processes within an interactive mixed-mode fieldwork design for list-based samples In the above discussion and in Figure 6.1, we 6.2 WEB SURVEYS WITHIN have talked about list-based web surveys, but THE PROJECT MANAGEMENT in the case of non-list-based web surveys, this FRAMEWORK would change slightly: instead of direct individ- ual invitations, a researcher interacts with the In Section 2.6 we addressed management aspects units using variations in the promotion strategy that were specific to the core web survey pro- (e.g. banner ads), so the focus of interaction cess, while in this section we discuss the broader would be on modifying the online recruitment context of project management. efforts (Section 2.2.3). For example, a researcher Web surveys can be treated as any other can decide to decrease direct banner advertising project, where general project management and increase promotion in online social media. principles apply: a researcher needs to organ- Interactive framework designs need to be con- ize, plan and coordinate various activities to sidered in pre-fielding and can complement or reach the goals within certain time and resource replace the recruitment plan, where we additionally constraints. Professional project management extend the set of design features related to timing, standards were developed for these types of tasks costs and data quality, as well as the set of mon- and work, particularly when it is done once, for itoring indicators, which then point to situations the first time, or repeated with different content. requiring action or modification. As mentioned Project management helps formalize the elab- previously, real-time calculations can be integrated oration of description of work (DoW), structuring and expanded with intelligent ICT support. of the tasks and activities into work packages Some aspects of the general optimization in a (WP) and the optimal allocation of resources. It mixed-mode setting can be found in the literature also deals with planning and timing, including (e.g. Schouten et al., 2013; Vannieuwenhuyze, the specification of milestones (i.e. key dates) 2014; Vehovar et al., 2010). Of course, the iter- and deliverables, which is the term that denotes ative fieldwork design can be further expanded reports, results and other measurable outputs of beyond the involvement of mixed-modes by the the project in the project management context. incorporation of mixed methods and with the The provision of control is also very important interaction of external administrative, business here, as well as supervision, coordination and and/or technical processes. interventions.

242

06_Callegaro et al_Ch-06.indd 242 4/21/2015 4:00:34 PM BROADER CONTEXT OF WEB SURVEYS

A researcher needs to coordinate many activ- complexity of the web survey requires a ities during the web survey project. We can project team, where additional interactions, deduce from the structure of the web survey communications and complications arise. A process, illustrated in Figure 1.1 (Section 1.2), typical web survey project, which includes that the activities of all 12 stages interact among certain substantive preliminary investiga- tions, implementation of the web survey and themselves. These stages can also be directly a final report, may already require a small transformed into the structure of work packages team and a few hundred hours spanning within the project management plan. Consid- a few weeks or months. All of this perhaps erable complexity arises already within the also requires formalized project management questionnaire preparation process (Figure 2.1 in tasks, at least at a minimal level. Section 2.3.6), where testing and questionnaire • With growing complexity the resources development are linked to a complex manage- required for project management typically ment process. Additional complications come expand rapidly, while at the same time the from preliminary research activities and other core web survey process itself often takes up interactions with the environment that is external an increasingly smaller part. Running web sur- to the core web survey process. veys with budgets approaching or exceeding six digits (e.g. $100,000) is already a serious Thus, there is a lot to plan, a lot to coordinate enterprise which requires formalized project and also a lot that could go wrong. The mana- management. gerial skills required can easily become equally • With online panels, longitudinal studies, important – or even more important – than the research conducted across many coun- methodological skills for the successful running tries and research where partners from of web surveys. We can illustrate the project different organizations and sectors are management tasks with a few examples related involved, the importance of project man- to levels of complexity: agement increases additionally. Within international research, the translation and back-translation­ alone is already a serious • In a post-event evaluation survey – where we project management challenge. seek feedback from participants whose emails we are also disposing of – project manage- ment can be a straightforward sequence of Professional data collection organizations, preparing the questionnaire, arranging emails, regardless of the sector (commercial or public), sending invitations plus a reminder and then typically have their own elaboration of project generating a preliminary report, which is dis- management rules. On the other hand, organi- seminated online. All this may take only a few zations not dedicated to survey data collection, minutes for an experienced researcher with particularly in the academic and non-profit sec- suitable software and with a standardized tor, are also involved in conducting web surveys, questionnaire at hand. and sometimes their researchers are not prepared • In the above case, if we add new questions, for the management of complex web survey which need to be developed and pre-tested, projects because management issues cannot be this involves more activities, so the entire proj- ect may require a few hours of a researcher’s dealt with using only general knowledge and time. In any case, this example fully demon- common sense. Below we outline typical proj- strates the power of web surveys: namely, the ect management issues that can cause problems extreme speed and convenience, especially if when they are treated with insufficient resources respondents reply immediately via a mobile and attention: device. The time and cost efficiency cannot be compared with any traditional survey mode • Adequate format of project team meetings alternative. (i.e. structure, length, frequency), together • Things get more complicated when we with strict recording of minutes, their conduct a survey for a client or when the monitoring and reporting.

243

06_Callegaro et al_Ch-06.indd 243 4/21/2015 4:00:34 PM WEB SURVEY METHODOLOGY

• Clear (i.e. written) separation of responsibilities sections on survey administration. A more struc- among team members and the client, including tured discussion of project management issues in the communication strategy (e.g. can the client surveys – not specifically geared to web surveys, talk directly to the web programmer?). but to surveys in general – can be found in Iarossi • Quality assurance criteria and corresponding (2006) and Kennedy, Tarnai, & Wolf (2010). evaluation procedures, which need to be formal- ized in advance in a stand-alone document, and where it is also useful to involve external expert monitoring or a steering committee, which reg- 6.3 THE WEB SURVEY ularly oversees the project (this is particularly relevant for new, lengthy or complex projects). PROFESSION • Elaboration in advance of the essential manage- Conducting web surveys requires familiarity ment procedures, together with initial planning, not only with survey methodology, but also contingency plans, risk management, crisis management and conflict resolution schemes. with ICT, statistics, management, legislation • Formalized monitoring of the core web survey and with the broader social context, including process, especially fielding (e.g. response ethics. Related competences are found at the rates, data quality). intersection of various professions. In addition, • Regular control of resources, including a question of whether we can talk about a sepa- hours spent, and monitoring of the timing rate web survey methodology profession arises. via milestones, deadlines, critical paths and Generally, we talk about a profession when Gantt charts, which is a standard monitoring the following elements exist: formal educa- approach in project management. tion programmes, explicit occupation profiles, • Platform for documenting, backing up, professional journals, dedicated international archiving and communication, which pref- associations and events, as well as certain gen- erably goes beyond email attachments and enables online access to the most recent mate- eral codes and standards. The majority of these rial (e.g. draft questionnaire). attributes are missing for web survey meth- • Clear regulation (e.g. contracts) regarding odology. However, they do exist for survey ownership of the data, intellectual property methodology in general, which is the field to rights, and penalty clauses in the case of which web survey methodology predominantly delays or mistakes in data collection. belongs. On the other hand, various professional activi- All of the above hold true for any survey research ties specific to web survey methodology do exist, project, while with web surveys there are many from regular events, dedicated courses (particu- additional opportunities to shorten, simplify and larly in various summer schools), numerous optimize management tasks by using integrated research projects, professional networks, spe- web survey software and other online services. cialized standards and textbooks. We may also The essential professional resources are related mention dedicated conference tracks, thousands to specific management approaches where of scholarly papers, presentations and other mate- standards and certificates are also elaborated, rials, including numerous online resources, as for instance PMIbook,2 IPMA3 and PRINCE24. well as hundreds of web survey software pack- There also exist many project management ages, which often include related methodological textbooks and guides, but there are surprisingly guidelines. WebSM (http://www.websm.org) pro- few dedicated treatments dealing directly with vides an exhaustive overview of these aspects. surveys (Stouthamer-Loeber & van Kammen, From this point of view, we can talk about a 1995) or indirectly (Lyberg et al., 1997). Like- certain professional knowledge and the corre- wise, survey methodology textbooks rarely have sponding elements of the profession, of which explicit chapters covering project management, this book is a testimony. In the remainder of this although they may partially cover these issues in section, we first review the scientific disciplines

244

06_Callegaro et al_Ch-06.indd 244 4/21/2015 4:00:34 PM BROADER CONTEXT OF WEB SURVEYS

where web surveys are applied and methodolog- account when implementing web surveys; see the ical research is conducted (Section 6.3.1). We relevant textbooks: Iacobucci & Churchill (2015) also address the related legal (Section 6.3.2) and for marketing research; Eysenbach & Wyatt ethical issues (Section 6.3.3), formal codes and (2002) for health studies; Bocarnea, Reynolds, standards (Section 6.3.4), as well as informal & Baker (2012) for organizational research. guides and other materials (Section 6.3.5). A very natural methodological environment of web surveys is the social science methodol- ogy, which provides the broader methodological framework for web surveys (e.g. Babbie, 2013). 6.3.1 Related Disciplines In particular, this holds true for the issues related Web surveys are formally nested within the to problem formulation, research steps, rela- framework of survey research and within the tionship of decision vs research problem, the broader context of social science methodology. conceptualization and operationalization pro- In the narrowest sense, social science method- cess, as well as for general familiarity with the ology often relates predominantly to sociology implementation of qualitative and quantitative and to a somewhat lesser extent also to political research methods. More specifically, the survey science and communication studies. In any case, research context is particularly important here. implementations of web surveys in these areas are Throughout this book we have referred to the highly relevant for web survey methodology. standard textbooks in this field, of Groves et al. In an even broader sense (e.g. OECD, 2002), the (2009) and Biemer & Lyberg (2003). social sciences – and the implementation of web On the other hand, for various specific surveys – also include three other large streams: aspects, other disciplines need to be consulted. psychology, economics and educational science, For example, survey data analysis requires sta- as well as geography, urban planning, demog- tistical knowledge (e.g. Heeringa et al., 2010), raphy, management, public administration, law, while web usability involves expertise from HCI management, organization science, and various (e.g. Roe, 2008). specific areas such as evaluation research, social According to the above overview of the anthropology and linguistics. Marketing research, related disciplines, the aspects of web survey in particular, nested within the broader field of methodology are very scattered, which is also a economics and business, is especially important characteristic of survey methodology in general. for web surveys, because it is there that the major- Therefore, we can find these aspects within var- ity of professional web surveys are conducted. A ious professional associations, mostly related very important area of web survey implementa- to survey research (AAPOR, CASRO, ESRA, tion is also the specific cross-disciplinary context ESOMAR), but also to Internet research (AoIR, of so-called online research and Internet studies. AAAS, APA), social science methodology (ISA Besides the social sciences, web surveys RC33), statistics (ISI, IASS, AMSTAT, RSS) appear in other sciences, especially in the and marketing research (ESOMAR, AMA, humanities and medicine (health research), but MRS), as well as areas related to computer sci- also in various natural and technical sciences. ence and informatics (ASC, ACM, IFIP-HCC). The areas of statistics, computer science and Large organizations (e.g. Eurostat) may also run informatics (e.g. HCI) are particularly relevant specific activities related to new technologies for web survey methodology. in data collection (e.g. the New Techniques and To understand the role of web surveys in Technologies for Statistics (NTTS) conference). the various contexts, we should be aware that None of the above professional organizations is many disciplines have developed their own dedicated exclusively to web surveys, and the methodological approaches. Accordingly, the same is also true for the corresponding activi- corresponding professions should be taken into ties and events. Nevertheless, dedicated and

245

06_Callegaro et al_Ch-06.indd 245 4/21/2015 4:00:34 PM WEB SURVEY METHODOLOGY

continuous events related to web survey method- copyright for certain questions established ology exist: namely, the biannual Internet Survey and protected? Methodology Workshop5 and the annual Interna- • Archiving: Whose responsibility is it if data dis- tional Workshop on Internet Survey and Survey appear? What are the data archiving procedures? Methodology6. • Software: Whose responsibility is it if a soft- ware bug causes damage? • Illegal content: Whose responsibility is it when illegal content is uploaded by a researcher or 6.3.2 Legal Issues respondents to the web questionnaire? • Disclosure: Who is responsible if tables or Web surveys are closely interwoven with leg- data that enable identification of personal islation, which can become very complicated information are released? particularly when we conduct international • Security: Whose responsibility is it if an unau- research. This is in general a very complex thorized person accesses the data during the issue, and in our brief overview we will pinpoint (non-)encrypted answering process, via a stolen and illustrate only the most essential aspects. account, or directly by breaking into the server? More specific information and guidance can be obtained from local marketing, statistical or sur- When web surveys run as a hosted service (SaaS), vey research associations. the cancellation policy and taxes can also become Despite the trends towards global harmoni- an issue, as well as the relationship between the zation, legal settings vary considerably across respondent, a researcher and the supplier of the countries, particularly with respect to criminal web survey service. Additional complications issues, privacy, intellectual property, telecom- arise when the hosting service is separated from munication rules and consumer protection. Let the software supplier, not to mention situations us present a few of the crucial legal issues: when all these subjects and services are in different countries with different legislation. • Unsolicited email invitations: Can they be sent It is not surprising that the terms of use and to private or business email addresses, which the related legal clarifications of web survey soft- are public on the web? ware are becoming increasingly lengthy, often • Cookies: How does legislation regulate them? with separate sections on privacy, spam, cookies, • Spam: What are the specific spam regulations? security, incentives, cancellation policy, etc. A • Incentive: What type and level of incentives good illustration of this complexity are the pop- are allowed and what administrative proce- ular web survey software services, where we can dures accompany them? find up-to-date examples of elaborations on legal Lotteries or sweepstakes: How are lotteries or • issues (e.g. terms of use, privacy issues). However, sweepstakes arranged from a legislation point of view? such detailed treatments are still not standard. A • Privacy, confidentiality, anonymity: How are WebSM study (Vehovar, Čehovin et al., 2012) these issues legally regulated (for technical found that only 60% of suppliers include a privacy discussions, see Section 2.4.3)? How does one statement on their main website, while only 44% deal with situations where researchers take list conditions of use. advantage of collected responses (e.g. for marketing purposes)? • Paradata: Who has the right to capture, pro- cess, analyse, publish and archive them? 6.3.3 General Ethical Concerns • Minors: How can children be involved in a web survey and what is the lowest age for Ethical dilemmas are present in all steps and their participation? stages of the web survey process. Generally • Intellectual property rights: Is permission speaking they address the question of whether required to use a certain question? How is the data, procedures, actions and non-actions related

246

06_Callegaro et al_Ch-06.indd 246 4/21/2015 4:00:34 PM BROADER CONTEXT OF WEB SURVEYS

to the web survey process may potentially harm ensure the protection of safety, rights and welfare other persons or groups. By harm we mean cer- of the respondents. The American Association tain negative physical, psychological or societal for Public Opinion Research (AAPOR) main- experience and related negative feelings. tains a special section on its website dedicated to A typical problem, yet still a relatively light IRBs and survey research practices. one, is whether to disclose the expected length We recommend familiarity with ethical codes of the web questionnaire in advance – when we before the start of a web survey project, in order know that it is above 30 minutes – and then run to identify and address potential problems. The the risk of a decline in cooperation rates (Craw- recommendation of which ethical code to read ford et al., 2001; Galesic, 2006). Likewise, we is less straightforward. Groves et al. (2009) pro- have mentioned that seeking explicit consent that vide a good starting point for the general survey paradata can be captured substantially decreased research context. cooperation (Couper & Singer, 2013), which is problematic for the entire informed consent approach (Losch, 2008) that requires respond- 6.3.4 Formalized Certificates, Codes ents explicitly to agree – before undertaking the survey – that they are aware of the purpose, and Standards rights and potential harm of the research. In previous sections we mentioned the general A more extreme example would be a web legal and ethical issues that need to be taken respondent reporting various problematic issues in into consideration when conducting web sur- a web survey, from illegal activities (e.g. stealing, veys. More specific elaboration is usually found drug trafficking, illegal gun ownership) and even- in various codes and standards for conducting tual critical inner states (e.g. depression, suicide research, which have been established by profes- attempts) to serious crime issues (e.g. attempted sional research associations within the context murder, paedophilia practices). In such cases, keep- of formal certificates and standards. Since the ing the promise of anonymity and non-disclosure methodological, legal, ethical and professional can become a very serious ethical dilemma for a aspects are usually interwoven in such codifica- researcher. We provide some illustration of typical tion, we treat them all together in this section. ethical dilemmas in web surveys in the Supple- With respect to project management, we have ment to Chapter 6, Examples of ethical dilemmas already mentioned specific standards (e.g. PMI- in web surveys (http://websm.org/ch6). book, IPMA and PRINCE2), and other quality The treatment of ethical aspects varies across management approaches and certificates (e.g. Six countries to a lesser extent compared with the Sigma TQM). General management, administra- variation of legal issues. However, a larger var- tion and processing standards (e.g. ISO 9001) can iation in ethical aspects is seen across different be obtained from professional certification organ- institutions and associations. Instead of inspec- izations (e.g. ISO7 or BSI8). Some more specific tions and juridical systems, various ethical standards may appear relevant for web surveys, committees and boards monitor the rules and as we mentioned in the related discussions on usa- may impose profession-specific sanctions (e.g. bility, security or archiving. Similarly, we referred formal warning or even expulsion from a pro- to two survey-specific ISO standards: fessional association). Sometimes (e.g. in health research) explicit pre-approval is required from • ISO 20252 Market, opinion and social research. an ethical body before the start of the survey. In Vocabulary and service requirements (ISO, the United States, for example, research in most 2012). institutions (academic, public and sometimes • ISO 26362 Access panels in market, opinion private) is subject to Institutional Review Boards and social research. Vocabulary and service (IRBs), and surveys are examined in order to requirements (ISO, 2009).

247

06_Callegaro et al_Ch-06.indd 247 4/21/2015 4:00:35 PM WEB SURVEY METHODOLOGY

Obtaining and maintaining these certificates codes (AAPOR, 2011), as well as the annual confirms that an organization complies with review of new books (English language only) in certain standards and may also help to qualify a public opinion, survey methods and survey sta- research organization for big clients. However, tistics (Callegaro, 2014a). this is expensive (e.g. ISO or BSI). With respect to general statistical aspects, In addition to these certificates, which are Eurostat’s professional code is very informa- commercialized to a considerable degree, various tive (Eurostat, 2011). Various other professional other codes, standards and guides – directly or codes and standards exist, particularly within indirectly related to web surveys – exist and are national associations and large organizations; an typically freely available in the context of profes- overview can be found at WebSM. sional associations and organizations. These may We may also recall here the importance of be stand-alone documents dealing exclusively general W3C technical standards on the web with web surveys or just explicit sections in some (e.g. HTML or CSS code) and particularly the broader documents. The corresponding focus is compliance with web accesibilty standards (e.g. typically coloured with the mission and specifics of the US Rehabilitation Act, Section 50810). the organization/association (e.g. survey methodology, marketing or statistics). A monitoring and/or appeal body may also exist to overview and clarify issues. 6.3.5 Informal Guides, In terms of specific international treatment Recommendations and related to web surveys, ESOMAR is particularly Other Material a very rich source that provides several guide- lines, including ‘Guideline for online research’, Despite the certificates, codes and standards ‘Guideline for social media research’, ‘Guide- discussed above, in practice there is still a con- line for conducting mobile marketing research’, siderable dearth of practical guidelines about as well as ‘Questions to help buyers of online how to run web surveys. This gap is filled in samples’. Similar guidelines are also provided part by various informal guides, which predomi- by the Marketing Research and Intelligence nantly appear on the web. Association in ‘Ten questions to ask your online The main stream of these materials origi- survey suppliers’ (MRIA, 2013). nates from web survey software suppliers. They Ethical decision making and Internet research understand that selling the software and related are also fully elaborated within the context of Inter- services represents only one part of their busi- net research surveys (AoIR, 2012). International ness and that providing methodological guides, aspects of professional ethics are also provided by hints, training and best practice material from the Global Research Business Network (GRBN), the broader methodological field – and not which published a study comparing codes of pro- merely support for specific software features – is fessional conduct in Australia, Canada, Germany, an important part of their mission. These materi- the UK and the United States, with a specific sec- als range from simple how-to-do checklists and tion dedicated to Internet research.9 systematic tutorials to special user conference Specific web survey aspects are discussed in events and profound methodological materials, the CASRO ‘Code of standards and ethics for sur- including white papers and even books (e.g. vey research’ (CASRO, 2011) and in the Council Bhaskaran & LeClaire, 2010). of German Market and Social Association’s According to a WebSM study (Vehovar, ‘Directive for online surveys’ (Rat der Deutschen Čehovin et al., 2012), out of 377 software Markt- und Sozialforschung eV, 2007). suppliers, 67% offer basic methodological We should also mention the resources pro- documentation on how to conduct surveys. vided by AAPOR with respect to standards and 30% offer specific white papers, while 17% ethics, including the elaboration of disposition include extensive documentation and 9% have

248

06_Callegaro et al_Ch-06.indd 248 4/21/2015 4:00:35 PM BROADER CONTEXT OF WEB SURVEYS

methodological documentation in a formalized process is routinized and relies on a regular over- PDF file. An overview of their recommen- view of around a hundred sources and also on dations is summarized in the Supplement to an elaborated strategy to deal with new entries Chapter 6, List of web survey software meth- detected by search engines. odological recommendations (http://websm. In WebSM the bibliographic entries are org/ch6). included according to three criteria: (a) the Another stream, which fills the gap in pop- source is publicly available; (b) at least the ularizing the best practices in web survey abstract exists; and (c) the entry is based on methodology, is represented by the blogs and research with relevance to web survey method- other regular postings on the web. There might ology. Since a methodological research focus is be some promotional noise in these postings, but essential for establishing this relevance, various much valuable insight can be obtained on the descriptive research project reports or marketing current state of affairs in web survey methodol- research conference papers are beyond the scope ogy. The WebSM organizes blogs into four main of the database. On the other side, some more categories: general bibliographic entries are also included, because of their high indirect relevance for web • Expert blogs, which are maintained by survey survey methodology. They span from general methodology professionals and marketing social science methodology, general survey professionals. methodology, and CASIC, through usability • Association and media blogs, which strive to research, HCI, online qualitative methods, Inter- discuss contemporary survey methodology net studies, research on online communities, objectively and are not directly involved in e-learning to important case studies from dif- selling survey software or providing method- ferent substantive fields where web surveys are ological advice. used to collect data. This indirect criterion is • Web survey software supplier blogs, which are a part of the supplier’s marketing and sales more arbitrary, but we believe the approach has strategy, but also offer relevant information been consistent through the years. because they must be interesting to a broader The bibliography is structured according audience to be successful. to scholarly entries (journal papers, books, • Research organization blogs from the pri- edited books, book chapters) and other types vate, public and non-profit sectors which deal (magazine papers, newsletters, conference extensively with web survey methodology. proceedings, conference material, theses and diplomas, reports, business material, etc.). Each In 2014, the WebSM database11 contained around bibliographic entry is described with stand- a hundred blogs and other online sources, which ard codes for publication type, title, author(s), more or less regularly address issues related to source, year, abstract and source database for web survey methodology. full text, and also with specific codes for scien- tific field, topic, country, target population and related topics. 6.4 WEB SURVEY BIBLIOGRAPHY We provide here some insights into the WebSM bibliographic database structure for We introduced the WebSM website http://www. publications dating up to 2013. The scholarly websm.org in the Preface and referred to it in all ones amount to 2,888 out of 6,890 entries, the the chapters, particularly with respect to supple- remaining ones belonging predominantly to mentary materials, the web survey software list, various conference materials, proceedings and blogs and other information. However, the core other documents. mission of WebSM is its bibliographic database, The 10 journals with the largest num- established in 1998. The selection and inclusion ber of entries (journal papers) in the WebSM

249

06_Callegaro et al_Ch-06.indd 249 4/21/2015 4:00:35 PM WEB SURVEY METHODOLOGY

database – each journal having more than 10 Regarding the trends shown in Figure 6.2, entries – can be sorted in descending order as there seems to be a steady increase of new entries follows: Quirk’s Marketing Research Review, over two decades, following the initial publica- Public Opinion Quarterly, Social Science tion year of 1994, when the very first scholarly Computer Review, Journal of Official Statis- publication directly related to web survey meth- tics, International Journal of Market Research, odology appeared (i.e. Pitkow & Recker, 1994). Survey Practice, Behavior Research Methods, After 2004, the increase in annual input has Computers in Human Behavior, Journal of slowed down to around 500 entries per year, of Medical Internet Research and Marketing News. which 150–200 are scholarly entries. An unex- These journals essentially reflect the structure plained decline occurred in 2006. It is interesting of the related scientific fields and disciplines to note that similar observations on trends were discussed in Section 6.3.1. The majority of the found by a meta-study of related publications on entries are from the social sciences, followed by web survey research in China (Zhang, Shao, & marketing, health, Internet research, psychol- Fang, 2008). ogy and education, with a strong overlap with As for authors, there were 8,323 in the database general social science methodology, computer by the end of 2013, of whom 4,741 contributed science and statistics. Of course, this structure some scholarly input. In both cases the largest differs greatly from the structure of actual web part (76% and 84% respectively) contributed survey projects, as well as from the structure of only one entry. We might say that the entry level daily completed web questionnaires. for publishing in this area is relatively low, but With respect to region, US-related research this is more likely due to the fact that the main contributes the majority of entries (53%), while, research focus of the authors is outside web sur- for conferences, AAPOR has by far the most vey methodology. entries, followed by ESRA (European Sur- In Figure 6.3, we illustrate a co-authorship vey Research Association) and GOR (General network of authors in the field of web survey Online Research). methodology using the Pajek tool for network

700

600

500

400

300

200

100

0 1 12 994 995 996 997 998 999 10 11 13 1 1 1 1 1 1 2000 200 2002 2003 2004 2005 2006 2007 2008 2009 20 20 20 20

All types Scholarly All journals Top 10 journals

Figure 6.2 Bibliographic entries in the WebSM database according to publication year (1994–2013) and publication types

250

06_Callegaro et al_Ch-06.indd 250 4/21/2015 4:00:35 PM BROADER CONTEXT OF WEB SURVEYS

analysis (de Nooy, Mrvar & Batagelj, 2011). web survey methodology and that it excludes even- For this purpose we run additional analysis on tual other scientific bibliography of the authors. November 2014 WebSM bibliographic data- It is interesting to note that 52 of the 72 base, including only scholarly publications with authors in the figure present the largest com- direct relevance to web survey methodology. In ponent in the network (the largest sub-network Figure 6.3 only 72 authors with five or more of connected authors). Thus, this is a rela- scholarly bibliographic entries appear and all tively small network and as we (the authors of together they produced 528 entries included this book) personally know almost all of these in this analysis. Each author is presented with researchers, we observe that web survey meth- a circle and its size reflects the number of their odology is a dominant research focus only for entries in the WebSM database, for example a few of them. The great majority are involved Couper with 53, Reips with 42, and Dillman in general social science methodology, general with 31 entries. Altogether, there are 25 authors survey methodology and various substantive with more than 10 entries. The weight of the fields (e.g. sociology, psychology). lines linking the authors reflects the number of The list of all bibliographic entries included scholarly entries written in co-authorships, for in the above figure, together with some further example Couper-Tourangeau with 14 entries statistics on the bibliographic entries (research written in co-authorship. topic, language, sources, etc.), can be found on We should not forget that this illustrates only WebSM as the Supplement Statistics of WebSM bibliographic entries that are directly relevant for bibliography database (http://websm.org/ch6).

Christian, L. M.

Link, M. W. MacElroy, B. Macer, T. Heerwegh, D. Loosveldt, G. Stern, M. J. Smyth, J. D. Pitkow, J. E. de Leeuw, E. D. Israel, G. D. Im, E. O. Chee, W. Biffignandi, S. Dillman, D. A. Matzat, U. Faas, T. Krantz, J. H. Hox, J. Fuchs, M. Pratesi, M. Buchanan, T. Dennis, J. M. Kypri, K. Lozar Manfreda, K. Callegaro, M. Reips, U.-D. Vehovar, V. Fang, J. Batagelj, Z. Bethlehem, J. Joinson, A. N. Singer, E. Baker, R. P. Malhotra, N. Scherpenzeel, A. McCabe, S. E. Toepoel, V. Thomas, R. K. Krosnick, J. A. Bremer, J. van Soest, A. Lugtig, P. J. Taylor, H. Stieger, S. Boyd, C. J. Schonlau, M. Goeritz, A. Das, M. Couper, M. P. Birnbaum, M. H. Crawford, S. D. Terhanian, G. Bosnjak, M. Batinic, B. Conrad, F. G. Kaczmirek, L. Tuten, T. L. Roster, C. A. Tourangeau, R. Peytchev, A. Smith, S. M. Braun, M. Bandilla, W. Albaum, G. Yan, T.

Galesic, M. Behr, D. Whitcomb, M. E. Porter, S. R. Ritter, L. A. Sue, V. M.

Figure 6.3 WebSM co-authorships for authors with five or more scholarly entries with direct relevance to web survey methodology, November 2014

251

06_Callegaro et al_Ch-06.indd 251 4/21/2015 4:00:36 PM WEB SURVEY METHODOLOGY

We can conclude that professional activi- methodology context, as well as into the gen- ties related to web survey methodology have eral management framework. The borders with stabilized at a certain level. In future we survey methodology will become increasingly expect that web surveys will become even blurred, which we further discuss in Chapter 7 more integrated into the general social science on future trends.

NOTES

1 See pages 20–21 in Institute of Public Health presentation on data aspects, available at http://www.websm. org/nijz13 2 http://www.pmi.org/PMBOK-Guide-and-Standards.aspx 3 http://ipma.ch/resources/ipma-publications/ipma-competence-baseline/ 4 http://www.prince2.com/prince2-methodology 5 http://workshop.websm.org 6 http://kostat.go.kr/iwis/index.html 7 ISO standards: http://www.iso.org/iso/home/standards.htm 8 BSI certification: http://www.bsigroup.com/ 9 http://www.websm.org/grbn12 10 http://www.websm.org/508 11 http://www.websm.org/blogs

252

06_Callegaro et al_Ch-06.indd 252 4/21/2015 4:00:36 PM