Psychological Bulletin Copyright 2004 by the American Psychological Association 2004, Vol. 130, No. 6, 959–988 0033-2909/04/$12.00 DOI: 10.1037/0033-2909.130.6.959 The Role of Representative Design in an Ecological Approach to Cognition Mandeep K. Dhami Ralph Hertwig City University University of Basel Ulrich Hoffrage Max Planck Institute for Human Development Egon Brunswik argued that psychological processes are adapted to environmental properties. He proposed the method of representative design to capture these processes and advocated that psychology be a science of organism–environment relations. Representative design involves randomly sampling stimuli from the environment or creating stimuli in which environmental properties are preserved. This departs from systematic design. The authors review the development of representative design, examine its use in judgment and decision-making research, and demonstrate the effect of design on research findings. They suggest that some of the practical difficulties associated with representative design may be overcome with modern technologies. The importance of representative design in psychology and the implications of this method for ecological approaches to cognition are discussed. There is little technical basis for telling whether a given experiment is During his career, Brunswik developed a comprehensive theo- an ecological normal, located in the midst of a crowd of natural retical framework referred to as probabilistic functionalism and a instances, or whether it is more like a bearded lady at the fringes of corresponding methodological innovation called representative reality, or perhaps like a mere homunculus of the laboratory out in the design (for reviews, see Hammond, 1966; Hammond & Stewart, blank. (Brunswik, 1955c, p. 204) 2001a; Postman & Tolman, 1959). For Brunswik (1952), an or- If a student of psychology or philosophy had a time machine and ganism’s behavior is organized with reference to some end or could choose when and where to study, early 20th-century Vienna objective that it has to achieve in an inherently probabilistic world. would be a good choice. Its sparkling intellectual atmosphere To study these organism–environment relations, stimuli should be produced and brought together great thinkers such as Rudolf sampled from the organism’s natural environment so as to be Carnap, Herbert Feigl, Kurt Go¨del, Otto Neurath, Moritz Schlick, representative of the population of stimuli to which it has adapted and Ludwig Wittgenstein—all united in their interest in questions and to which the experimenter wishes to generalize (Brunswik, of epistemology and the philosophy of science. One individual 1956). As the opening quote of this article illustrates, Brunswik who was fortunate enough to have been there at the time was Egon stressed that experimenters should first avoid oversampling highly Brunswik (Leary, 1987). Brunswik was trained in Karl Bu¨hler’s improbable stimuli in a population and, second, avoid using stim- Vienna Psychological Institute, and he participated in discussions uli that do not exist in the population. with members of the Vienna Circle, who promoted the ideas of Whereas initially Brunswik’s ideas were largely ignored (Gig- logical positivism. It was to Bu¨hler and Schlick whom Brunswik erenzer, 2001), they were later silently—insofar as it happened submitted his doctoral thesis in 1927. In the minds of numerous without acknowledgment of their source—absorbed into main- people, Brunswik was to become one of the most outstanding and stream psychology (Hammond & Stewart, 2001b). Today, for creative psychologists of the 20th century. instance, the notion that cognitive functioning is adapted to the structure of the environment or the demands of the task is com- monplace. However, there appears to have been little acceptance of the method of representative design. Of this method, James J. Mandeep K. Dhami, Department of Psychology, City University, Lon- Gibson (1957/2001) wrote that don, England; Ralph Hertwig, Institute for Psychology, University of Basel, Basel, Switzerland; Ulrich Hoffrage, Max Planck Institute for Hu- [Brunswik] realized more than his contemporaries that the problems man Development, Berlin, Germany. of perception, as of behavior, cannot be solved by setting up situations We are grateful to Michael Birnbaum, James Cutting, Gerd Gigerenzer, in the laboratory which are convenient for the experimenter but Ken Hammond, David Mandel, Gary McClelland, James Shanteau, and atypical for the individual. He asks us, the experimenters in psychol- Tom Stewart for many constructive comments on ideas presented in this ogy, to revamp our fundamental thinking. It is an onerous demand. article. We also thank Tina-Sarah Auer, Lydia Lange, and Carsten Kissler Brunswik imposed it first on his own thinking and showed us how for their assistance in the literature searches and Anita Todd for editing a burdensome it can be. (p. 246) draft of this article. Correspondence concerning this article should be addressed to Mandeep Gibson expressed this view almost 50 years ago, shortly after K. Dhami, Department of Psychology, City University, Northampton Brunswik’s death. Since then, psychology has developed consid- Square, London EC1V 0HB, England. E-mail: [email protected] erably, and its methodology has evolved. The main goal of the 959 960 DHAMI, HERTWIG, AND HOFFRAGE present article is to explore whether the notion of representative & Schneider, 1966). It has since been featured in textbooks on stimulus sampling is of relevance to present-day research practice experimental psychology and become synonymous with the ex- in cognitive psychology. We focus on research conducted within perimental method per se. One example is Woodworth’s (1938) the field of judgment and decision making, because it is here that classic textbook, Experimental Psychology, also known as the Brunswik’s ideas have had the greatest impact. “Columbia Bible,” which espoused systematic design as the ex- We believe the notion of representative design will eventually perimental tool. become an important instrument in our methodological toolbox. Brunswik opposed psychology’s accepted experimental method Our assertion lies partly in the impact that representatively de- (Kurz & Tweney, 1997). He questioned both the feasibility of signed studies have already had in shaping policy and, conse- disentangling variables and the realism of the stimuli created in quently, human lives; partly in the rising concern with the limited doing so. In what he considered to be the simplest variant of external validity and generalizability of research findings; and systematic design, the one-variable design, Brunswik (1944, partly with the inclusion of ecological factors in contemporary 1955c, 1956) pointed to the fact that variables were “tied,” thus cognitive theories. However, we do not advocate a dogmatic making it impossible to determine the exact cause of an observed approach to methodology. The problem of how to conduct exper- effect. For example, in the Galton bar experiment, observers are iments such that they enable substantive inductive inferences has presented with the task of estimating the physical size of differ- no universally accepted solution. We believe that informed judg- ently sized objects. The distance between the objects and the ment, instead of dogmatism, should aid decisions about when to observers is held constant; consequently, physical size and retinal use representative design. size are artificially “tied,” because a small object will project a In Part 1 of this article, we review the development of the smaller retinal size and a large object will project a larger retinal method of representative design. In Part 2, we analyze a present- size. Brunswik (1944, 1955c, 1956) further argued that in the day research practice, namely, that of researchers who have de- sophisticated variant of systematic design, namely, factorial de- clared their commitment to Brunswik’s ideas and methods. In Part sign, variables are artificially “untied.” Here, the range of the 3, we examine whether representative stimulus sampling matters variables is arbitrary and is divided into k levels. The levels of one for the results obtained in psychology experiments. Finally, in Part variable are combined with the levels of another variable, exhaust- 4, we discuss the future of representative design. ing all possible combinations. All combinations have equal fre- quency, and the natural covariation among variables is eliminated. Part 1: Theoretical Foundations and Evolution of Factorial designs may, therefore, yield artificial combinations that Representative Design are impossible in the real world. Such stimuli may inhibit a researcher’s ability to examine how psychological processes func- Brunswik’s methodological ambition was grand. He proposed tion and thus limit the generalizability of research findings. an alternative to systematic design, which was the methodological dictate of the time. In this design, experimenters select and isolate Cognitive Psychology as a Science of one or a few independent variables that are varied systematically Organism–Environment Relations while holding extraneous variables constant or allowing them to vary randomly. Experimenters then observe the resulting changes Brunswik’s methodological innovations were closely inter- in the dependent variable(s). This design places a strong emphasis twined with his theoretical outlook (see Kurz & Hertwig, 2001). In on the notion of internal