Field Experiments in Organizations 93 OP04CH05-Eden ARI 25 February 2017 7:14
Total Page:16
File Type:pdf, Size:1020Kb
OP04CH05-Eden ARI 25 February 2017 7:14 ANNUAL REVIEWS Further Click here to view this article's online features: • Download figures as PPT slides • Navigate linked references • Download citations Field Experiments in • Explore related articles • Search keywords Organizations Dov Eden Coller School of Management, Tel Aviv University, Tel Aviv 6997801, Israel; email: [email protected] Annu. Rev. Organ. Psychol. Organ. Behav. 2017. Keywords 4:91–122 field experiment, experimental control, randomization, quasification, First published online as a Review in Advance on December 21, 2016 quasi-fields, little data The Annual Review of Organizational Psychology and Abstract Organizational Behavior is online at orgpsych.annualreviews.org Field experimentation, although rare, is the sterling-gold standard of orga- https://doi.org/10.1146/annurev-orgpsych- nizational research methods. It yields the best internally valid and general- 041015-062400 izable findings compared to more fallible methods. Reviewers in many psy- Copyright c 2017 by Annual Reviews. chology specialties, including organizational psychology, synthesize largely All rights reserved nonexperimental research, warn of causal ambiguity, and call for experi- mental replication. These calls go mostly unheeded. Practical application is a raison d’etreˆ for much organizational research. With the emergence of Access provided by Vrije Universiteit - Millennium Amsterdam on 11/11/17. For personal use only. Annu. Rev. Organ. Psychol. Behav. 2017.4:91-122. Downloaded from www.annualreviews.org evidence-based management, field experiments enable us to deliver the most actionable tools to practitioners. This review explicates the role of experi- mental control and randomization and enumerates some of the factors that mitigate field experimentation. It describes, instantiates, and evaluates true field experiments, quasi-experiments, quasi-fields, combo designs, and tri- angulation. It also provides practical tips for overcoming deterrents to field experimentation. The review ends describing the merging of new technolo- gies with classical experimental design and prophesying the bright future of organizational field experimentation. 91 OP04CH05-Eden ARI 25 February 2017 7:14 INTRODUCTION Field experimentation should be the method of choice for studying the many independent variables in organizational research for which field-experimental treatments can be devised. Organizational scholars too often overlook field experimentation as a design option, even when investigating treatable (aka, manipulable) independent variables. It is crucial that organizational research es- tablish causality. To be applicable it must be both internally valid and generalizable. Reviewers’ ubiquitous calls for experimental replication (Bass & Bass 2008, Kluger & DeNisi 1996, Stajkovic et al. 2009), specifically for field experiments (Scandura & Williams 2000), evidence unquenched thirst for organizational field experimentation. Social psychologists prefer laboratory experiments over field experiments. For example, Deci et al. (1999) expressly excluded field experiments from their review of experimental research on the effects of reward on motivation because they are “noncomparable with the laboratory experiments” (p. 635). This preference is inappropriate for organizational researchers, with our commitment to creating actionable knowledge. Fortunately, the many field experiments reviewed here—and many others not cited here—are evidence that editors and reviewers of top journals do appreciate well-done field experiments. This is not a review on method. Numerous publications explicate experimental method thoroughly (Cook & Campbell 1979, Lipsey & Cordray 2000, Shadish et al. 2002), including field experimentation in organizations (Aguinis et al. 2009, Dipboye 1990, Grant & Wall 2009, Highhouse 2009, King et al. 2012, Stone-Romero 2011). Boruch & Wothke (1985) discuss randomization in field experiments in detail. Distinctions and confusion regarding external va- lidity, ecological validity, generalization, and the role of mundane realism (Aronson & Carlsmith 1968, Highhouse 2009) are not discussed here. The review shows what has been done—and how much more could be done—applying the field-experimental method in organizations to advance organizational research. Neighboring fields are experiencing rapid growth of field experimentation. These include marketing (e.g., Ein-Gar & Steinhart 2011, Petersen & Kumar 2015) and behavioral economics (e.g., Azar et al. 2013, Bandiera et al. 2011, De Paola et al. 2014, Pelligra & Stanca 2013, Shavit et al. 2014, Stoop 2014). There is also expanding methodological literature on field experimentation in economics (e.g., Harrison & List 2004, Levitt & List 2009). Organizational psychology and organizational behavior (OPOB) may soon have to play catch-up. If the review is successful, the reader (a) when planning new research will consider field ex- perimentation as an option; (b) when reading research reports, will contemplate how field exper- imentation might better explore the same relationships; and (c) when writing discussion sections about nonexperimental research, will suggest field-experimental follow-up. The sidebar Major Topics Covered lists the key themes discussed below. The prime importance of causality in organizational research, and the unique appropriateness of field experimentation Access provided by Vrije Universiteit - Millennium Amsterdam on 11/11/17. For personal use only. Annu. Rev. Organ. Psychol. Behav. 2017.4:91-122. Downloaded from www.annualreviews.org to provide such causal knowledge, is a focus. The review discusses briefly experimental treat- ment, control, and randomization, then provides a description of some of the earliest attempts to conduct experiments in organizations. Distinctions among true experiments, quasi-experiments, field experiments, and quasi-fields are elaborated. Laboratory experimentation on organizational variables (see Weick 1967) is not reviewed. The review then discusses the factors that militate against field experimentation and dispels some misconceptions. These include presuming that field experiments are too difficult to do, that samples are too hard to obtain, and that ran- domization is impossible in organizational settings. The variety of published and unpublished organizational field experiments, including field experiments in organizations, laboratory-like ex- periments in the field, large- and small-scale field experiments, and in-house, problem-focused 92 Eden OP04CH05-Eden ARI 25 February 2017 7:14 MAJOR TOPICS COVERED Centrality of causality Rarity of field experimentation Experimental treatment and control Randomization: friend and foe Pioneers: the earliest organizational field experiments Quasi-experiments, quasification, quasi-fields, and combo designs Lab-like experiments in the field Generalization Big and small field experiments “Little data”: unsung field experimenters Factors mitigating field experimentation Triangulation and mixed methods Overcoming deterrents to field experimentation: utilize management indifference when randomization and fairness align invert the treatment piggyback on naturally occurring events transform delicate data use emerging technologies Approaching the ideal design A model field experiment The mother-in-law test (MILT) The future of organizational psychology and organizational behavior field experimentation practitioner-conducted field experiments, is instantiated. Experience-based “do’s” and “don’ts” are listed. The review closes with optimistic prophecy regarding the future of field experimentation in organizations. CENTRALITY OF CAUSALITY It is hard to refrain from causal thinking. Kahneman’s (2011) System 1 thinking, which he dubs fast thinking, has us automatically inferring causality: “We are evidently ready from birth to have impressions of causality which do not depend on reasoning about patterns of causation” (p. 76). We are wired to assume that relationships evidence causality. Kahneman describes this Access provided by Vrije Universiteit - Millennium Amsterdam on 11/11/17. For personal use only. Annu. Rev. Organ. Psychol. Behav. 2017.4:91-122. Downloaded from www.annualreviews.org as a major bias with potential for erroneous intuition. Organization scholars are not immune to fast thinking and fallacious causal inference. The experiment’s relative causal certainty justifies the causal inference that we so covet and elevates it above other methods, including the novel and ever more sophisticated analyses designed to infer causality from observational data. Highhouse (2009) stated it most cogently: “Randomized experiments are the most potent research design for determining whether or not x causes y” (p. 554). Unfortunately, most organizational research lacks causal persuasiveness. Aguinis & Vanden- berg (2014) described it as a central shortcoming: Better decisions presumably stem from findings for which strong confidence exists as to the cause and effect of a phenomenon. The reality, though, is that such confidence does not exist in the majority of www.annualreviews.org • Field Experiments in Organizations 93 OP04CH05-Eden ARI 25 February 2017 7:14 cases because of how the study was originally designed. ...[T]he most frequently used research designs are passive observation studies in which we observe whether the rank order of values in one (or more) variable(s) is associated with the rank order of another variable(s). Causal inferences are improbable under those circumstances. (pp. 584–85) Testing causal hypotheses