What Should Our Scientific Discourse Look Like? ObserverVol. 30, No. 1 January 2017
FIVE Myths About the Role of Culture in Psychological Research
a publication of
www.psychologicalscience.org/observer Integrative Science Symposia 23-25 March 2017 | Vienna, Austria Our Social Brain: Neurobiology Emotions in Context Bridging the Lab and the of Human Interactions Ralph Adolphs, Division of the Real World Christian Keysers, Social Brain Lab, Humanities and Social Sciences, Karen E. Adolph, Department of Early Price Registration | 15 February Netherlands Institute for Neuroscience, California Institute of Technology, USA Psychology, New York University, USA Register Now to Get the Best Rates and Department of Psychology, University Iris M. Engelhard, Department of Rick Dale, Department of Cognition of Amsterdam, The Netherlands Psychology, Utrecht University, & Information Sciences, University of Brian D. Knutson, Department of The Netherlands California, Merced, USA Psychology and Neuroscience, Stanford Klaus R. Scherer, Department of Susan Goldin-Meadow, Department University, USA Psychology, University of Geneva, of Comparative Human Development, KEYNOTE SPEAKERS Rebecca Saxe, Department of Brain and Switzerland (Discussant) The University of Chicago, USA Cognitive Sciences, Massachusetts Institute Jeanne L. Tsai, Department of Emiliano Macaluso, Impact Team, of Technology, USA Psychology, Stanford University, USA Lyon Neuroscience Research Center, Frank H. Wilhelm, Department of France Who’s In, Who’s Out? Loneliness, Clinical Psychology and Psychotherapy, Exclusion, and Integration University of Salzburg, Austria In Sync: The Dynamics of Taciano L. Milfont, School of Psychology Social Coordination Victoria University of Wellington, New Zealand The Push and Pull of Values Nick Chater, Behavioural Science Frosso Motti-Stefanidi, Department and Behavior Group, Warwick Business School, of Psychology, National and Kapodistrian Scott Atran, School of Anthropology United Kingdom University of Athens, Greece and Museum Ethnography, University Shaun Gallagher, Department of Alan Teo, Department of Psychiatry and of Oxford, United Kingdom Philosophy, University of Memphis, USA Cognitive Evolution: How Infants Break Genetic and Brain School of Public Health, Oregon Health Chi-yue Chiu, Department of Marco Iacoboni, Department of People Are Animals Too Into Language Diversity in AutismS & Science University, USA Psychology, The Chinese University of Psychiatry and Biobehavioral Sciences,
Hong Kong, China University of California, Los Angeles, USA W. Tecumseh Fitch Linda B. Smith Thomas Bourgeron Department of Cognitive Biology Department of Better Minds: Understanding Hazel R. Markus, Department of Department of Department of Human Genetics Andrzej Nowak, University of Vienna, Austria Psychological and Brain Sciences Cognitive Enhancement Psychology, Stanford University, USA Psychology, University of Warsaw, and Cognitive Functions Indiana University Bloomington, USA Pasteur Institute, France Daphne Bavelier, Department of Heidi Keller, Department of Poland and Florida Atlantic University, USA Psychology, University of Geneva, Switzerland Psychology, Osnabrück University, Natalie Sebanz, Department of Germany Arthur F. Kramer, Department of Cognitive Science, Central European Psychology, University of Illinois at Walter Mischel, Department of University, Hungary Urbana-Champaign, USA Psychology, Columbia University, USA E. Glenn Schellenberg, Department of (Discussant) Psychology, University of Toronto, Canada Ilina Singh, Department of Psychiatry University of Oxford, United Kingdom
The Science of Successful Aging Monica Fabiani, Department of Psychology, University of Illinois at Urbana-Champaign, USA Denise C. Park, Center for Vital Longevity ICPS SPONSORS The University of Texas at Dallas, USA #icps17vie Karl A. Pillemer, Department of Human Development, Cornell University, USA Lorraine K. Tyler, Department of Psychology, University of Cambridge, United Kingdom www.icps2017.org Integrative Science Symposia 23-25 March 2017 | Vienna, Austria Our Social Brain: Neurobiology Emotions in Context Bridging the Lab and the of Human Interactions Ralph Adolphs, Division of the Real World Christian Keysers, Social Brain Lab, Humanities and Social Sciences, Karen E. Adolph, Department of Early Price Registration | 15 February Netherlands Institute for Neuroscience, California Institute of Technology, USA Psychology, New York University, USA Register Now to Get the Best Rates and Department of Psychology, University Iris M. Engelhard, Department of Rick Dale, Department of Cognition of Amsterdam, The Netherlands Psychology, Utrecht University, & Information Sciences, University of Brian D. Knutson, Department of The Netherlands California, Merced, USA Psychology and Neuroscience, Stanford Klaus R. Scherer, Department of Susan Goldin-Meadow, Department University, USA Psychology, University of Geneva, of Comparative Human Development, KEYNOTE SPEAKERS Rebecca Saxe, Department of Brain and Switzerland (Discussant) The University of Chicago, USA Cognitive Sciences, Massachusetts Institute Jeanne L. Tsai, Department of Emiliano Macaluso, Impact Team, of Technology, USA Psychology, Stanford University, USA Lyon Neuroscience Research Center, Frank H. Wilhelm, Department of France Who’s In, Who’s Out? Loneliness, Clinical Psychology and Psychotherapy, Exclusion, and Integration University of Salzburg, Austria In Sync: The Dynamics of Taciano L. Milfont, School of Psychology Social Coordination Victoria University of Wellington, New Zealand The Push and Pull of Values Nick Chater, Behavioural Science Frosso Motti-Stefanidi, Department and Behavior Group, Warwick Business School, of Psychology, National and Kapodistrian Scott Atran, School of Anthropology United Kingdom University of Athens, Greece and Museum Ethnography, University Shaun Gallagher, Department of Alan Teo, Department of Psychiatry and of Oxford, United Kingdom Philosophy, University of Memphis, USA Cognitive Evolution: How Infants Break Genetic and Brain School of Public Health, Oregon Health Chi-yue Chiu, Department of Marco Iacoboni, Department of People Are Animals Too Into Language Diversity in AutismS & Science University, USA Psychology, The Chinese University of Psychiatry and Biobehavioral Sciences,
Hong Kong, China University of California, Los Angeles, USA W. Tecumseh Fitch Linda B. Smith Thomas Bourgeron Department of Cognitive Biology Department of Better Minds: Understanding Hazel R. Markus, Department of Department of Department of Human Genetics Andrzej Nowak, University of Vienna, Austria Psychological and Brain Sciences Cognitive Enhancement Psychology, Stanford University, USA Psychology, University of Warsaw, and Cognitive Functions Indiana University Bloomington, USA Pasteur Institute, France Daphne Bavelier, Department of Heidi Keller, Department of Poland and Florida Atlantic University, USA Psychology, University of Geneva, Switzerland Psychology, Osnabrück University, Natalie Sebanz, Department of Germany Arthur F. Kramer, Department of Cognitive Science, Central European Psychology, University of Illinois at Walter Mischel, Department of University, Hungary Urbana-Champaign, USA Psychology, Columbia University, USA E. Glenn Schellenberg, Department of (Discussant) Psychology, University of Toronto, Canada Ilina Singh, Department of Psychiatry University of Oxford, United Kingdom
The Science of Successful Aging Monica Fabiani, Department of Psychology, University of Illinois at Urbana-Champaign, USA Denise C. Park, Center for Vital Longevity ICPS SPONSORS The University of Texas at Dallas, USA #icps17vie Karl A. Pillemer, Department of Human Development, Cornell University, USA Lorraine K. Tyler, Department of Psychology, University of Cambridge, United Kingdom www.icps2017.org Observer ISSN: 1050-4672 © 2017 Association for Psychological Science Published 10 times per year by the Association for Psychological Science, Federal ID Number: 73-1345573 the Observer educates and informs the Association on matters affecting All rights reserved. PUBLISHER Sarah Brookhart the research, academic, and applied disciplines of psychology; promotes EDITOR Scott Sleek the scientific values of APS Members; reports and comments on issues of ASSISTANT MANAGING EDITOR Mariko Hewer EDITORIAL ASSISTANT Christopher Collins national interest to the psychological scientist community; and provides a SENIOR GRAPHIC DESIGNER Candy Ferguson vehicle for the dissemination of information on APS. GRAPHIC DESIGNER Navaré Carter
APS Board of Directors PRESIDENT REPRINT PERMISSION: Photocopying OBSERVER FORUM: The Observer Susan Goldin-Meadow – The University of Chicago Observer content for classroom use is welcomes your comments and feedback. PRESIDENT-ELECT permitted at no charge. Students may not For consideration in the Observer Forum, Suparna Rajaram – Stony Brook University be charged more than the actual cost of letters should be sent to apsobserver@ IMMEDIATE PAST PRESIDENT producing the photocopy. Source citation psychologicalscience.org. Unless otherwise C. Randy Gallistel – Rutgers University must indicate that the materials are from the indicated, all correspondence received SECRETARY Observer, a publication of the Association for will be considered for publication. The Gün R. Semin – ISPA–Instituto Universitário, Portugal; and Psychological Science. Observer does not guarantee publication Utrecht University, The Netherlands and reserves the right to edit all content APS does not assist in copyright requests TREASURER for length and style. Roberta L. Klatzky – Carnegie Mellon University involving artwork obtained from a third party. For all other permissions inquiries, EXECUTIVE DIRECTOR SUBMISSIONS: The Observer accepts including requests to republish material Sarah Brookhart – Association for Psychological Science original manuscripts, announcements, and in another work, please contact Copyright calendar items for publication; manuscripts MEMBERS-AT-LARGE Clearance Center at 222 Rosewood Drive, Deanna Barch – Washington University in St. Louis must be original and should not have been Danvers, MA 01983, call +1 978.750.8400, Dorthe Berntsen – Aarhus Universitet, Denmark published or scheduled to be published or visit www.copyright.com. Thomas H. Carr – Michigan State University elsewhere. The Observer editorial calendar Michelle R. (Mikki) Hebl – Rice University is at the discretion of the editor, and the ADVERTISING: The Observer accepts Simine Vazire – University of California, Davis Observer does not guarantee publication. Cindy Yee-Bradbury – University of California, Los Angeles employment and commercial advertis- Submissions should be sent via email to ing. Information, rates, and calendar are [email protected]. available by calling Michelle Davis at FOUNDING EXECUTIVE DIRECTOR The Observer reserves the right to edit all Alan G. Kraut – Association for Psychological Science +1 202.293.9300 or going online to content for length and style. www.psychologicalscience.org/jobs. Past Presidents Nancy Eisenberg 2014-2015 John Darley 2001-2002 Elizabeth A. Phelps 2013-2014 Robert A. Bjork 2000-2001 Joseph E. Steinmetz 2012-2013 Elizabeth D. Capaldi 1999-2000 Send address changes to: Douglas L. Medin 2011-2012 Elizabeth Loftus 1998-1999 Association for Psychological Science Mahzarin R. Banaji 2010-2011 Kay Deaux 1997-1998 1800 Massachusetts Ave, NW Suite 402 Washington, DC 20036-1218 USA Linda Bartoshuk 2009-2010 Sandra Scarr 1996-1997 . Walter Mischel 2008-2009 Richard F. Thompson 1995-1996 [email protected] John T. Cacioppo 2007-2008 Marilynn B. Brewer 1993-1995 www.psychologicalscience.org/members (click Update Profile) Morton Ann Gernsbacher 2006-2007 Gordon H. Bower 1991-1993 Michael S. Gazzaniga 2005-2006 James L. McGaugh 1989-1991 Robert W. Levenson 2004-2005 Janet T. Spence 1988-1989 Henry L. Roediger, III 2003-2004 Charles A. Kiesler 1988 Susan T. Fiske 2002-2003
APS Staff Executive Director Sarah Brookhart Deputy Director Tracy L. Waldeck Senior Director for Operations Aime M. Ballard-Wood Director of News and Information Scott Sleek APS Postdoc Exchange Assistant Director of News and Information Anna Mikulak Manager, Digital and Social Media Lucy Hyde Science Writer Alexandra Michel The free and searchable APS Postdoc Exchange Senior Graphic Designer Candy Ferguson Graphic Designer Navaré Carter offers a better way to advertise postdoctoral openings to Assistant Managing Editor, Observer Mariko Hewer Editorial Assistant Christopher Collins candidates across all of psychological science and related disciplines. Assistant Director for Policy K. Andrew DeSoto Manager of Policy and Programs Gabriela M. Galeano Post your ad for free and find the best and brightest in psychological Director of Publications Todd Reitzel Senior Managing Editor, Journals Torrance Gloss science and related fields around the globe. Managing Editor Christine E. Browne Copy Editor Brian P. Winters Copy Editor Rodney Atkins Peer Review Manager Amy Drew Senior Science Writer, Journals Claire Cole Peer Review Coordinator Kristen Medeiros Director of Meetings Nathalie L. Rothert Postdoctoral Research Fellowship Senior Meetings Manager Jermaine W. Ivy Meetings Manager Yasmine Kuttab Meetings Manager Jennifer Alexander Meetings Assistant Derica Clarke Program Assistant Mari Hicks Director of Membership Patti Martin Membership Manager Dion P. Dears Membership Coordinator Lisa Lockman Marketing and Advertising Manager Sarah L. Schroeder Marketing Assistant Michelle J. Davis Managing Editor, Website Asa Boy Finance and Administration Coordinator Marilyn Talley Finance and Administration Assistant Siobhan A. DeLoatch Association for Psychological Science +1 202.293.9300; Fax: +1 202.293.9350 www.psychologicalscience.org/postdoc-exchange [email protected] www.psychologicalscience.org/observer Observer Contents January 2017 Volume 30, Number 1
FEATURES
FIVE Myths About the Role of Culture in Psychological Research Psychological scientists will need to work with increasingly diverse participant pools to ensure that their findings cover the full spectrum of human experience. Cornell University researcher and APS Fellow Qi Wang dispels some of the assumptions that researchers make about integrating cultural factors into their investigations. 20
Finding Common Ground What Do We Want Our Between Classic Learning Scientific Discourse to Theories Look Like? In an APS-Psychonomic Society Psychological scientist W.K. & K.W. Estes Lecture, APS Alison Ledgerwood Past President and US National curates a discussion on Medal of Science Laureate what scientific discourse Gordon H. Bower delivers a has become in the age of 60-year retrospective on his social media and how it attempts to integrate the might evolve to be more learning theories of his late productive. mentor William K. Estes with those of the influential learning theorist Clark L. Hull. 18 13 Contents (cont.) Observer January 2017
Presidential Column 5 Robust Science Depends on Understanding the Science of Humans APS Fellow Howard C. Nusbaum serves in a leadership position at the National Science Foundation. From this vantage point, he devotes a guest column to discussing how even the most robust science is still vulnerable to human error.
Science in Policy 14 Harnessing the Wisdom of Crowds to Improve Hiring Incorporating psychological research on implicit bias in hiring, the United Kingdom’s Behavioural Insights Team is investigating collaborative ways to help companies select the most qualified candidates for job openings.
Videos Share the Psychological Study of 25 Language With support from the APS Fund for Teaching and Public Understanding of Psychological Science, researchers at Cleveland State University have developed a video series focused on the psychology of language, with presentations by APS President Susan Goldin-Meadow and others.
Departments Teaching Current Directions in Observations 7 Psychological Student Notebook 31 Science Featured articles: Members in the News 33 “When Anxiety Doesn’t Add Up: Understanding and Announcements 37 Preventing Math Anxiety” and “Should You Trust Your Unconscious When Judging Lying? Probably Not!” 35 27 5 PRESIDENTIAL COLUMN Robust Science Depends on Understanding the
Guest Columnist Science of Humans Howard C. Nusbaum National Science Foundation My colleague Howard C. Nusbaum is on leave from the University of Chicago Department of Psychology, serving as the Director for the Division of Behavioral and Cognitive Sciences in the National Science Foundation (NSF) Directorate for Social, Behavioral, and Economic Sciences. I invited him to weigh in on the goals of reliability, validity, and replicability from his vantage point at the NSF, and to discuss the special role that psychological science can play not only in achieving these goals, but also in understanding why they are so difficult to achieve. -APS President Susan Goldin-Meadow
cience is a method of generating knowledge and test- electrical charge, which he did by selective reporting. In the ing beliefs; it trumps authority by empirical evaluation end, though, replication wins out as the natural scientific Sand depends on reliability and validity to uphold that corrective process. knowledge. But science also is a human enterprise: Whether Psychological science has always been especially mind- in psychology or neuroscience, physics or chemistry, studies ful of the tools of reliability and validity as a consequence are designed, conducted, and reported by people. Even with of our intellectual history. Psychology moved from analytic computer-controlled experiments, humans bear responsibility introspection to intersubjective testability to develop a sci- for the findings. When humans are involved, errors will oc- ence relying on objective and systematic methodology. This cur. Some errors result from cognitive biases in decisions and methodology puts psychological science on the same objective judgment (Gilovich, Griffin, & Kahneman, 2002; Tversky & footing as research in physical sciences. Acceptance of this Kahneman, 1974), including confirmation bias (e.g., Nickerson, regimen is why we object to the false distinction of putatively 1998); others occur by accident, oversight, or carelessness; still hard (e.g., physics) and soft (e.g., psychology) sciences. The sci- others may be motivated. entific method establishes parity, and the target of understand- A “motivated” error occurs when results are at odds with ing phenomena that are not directly observable — whether reality and are produced with the intention of distorting or states of mind or dark matter — certainly does not cleave the fabricating the analysis for reasons independent of objective sciences apart. evidence, whether because of conviction or gain (Broad & Although we’re well aware of the controversies over repli- Wade, 1982). Diagnosing motivated error is difficult. Allegra cation in psychological science, it is important to remember Goodman’s novel Intuition (2006) illustrates how personal and that all sciences suffer the same issues. Physics has dealt with professional motivations can muddy the waters of scientific controversies over cold fusion and faster-than-light particles, knowledge when error occurs. Selective data reporting is a but ultimately scientific theory and replication led to clar- turning point in the novel. In much the same way, William ity. However, replication is not always the answer. Consider Broad and Nicholas Wade (1982) discussed how Robert A. Prosper-René Blondlot’s 1903 discovery of N-rays (Broad & Millikan received the Nobel Prize for demonstrating quantal Wade, 1982). This discovery, a physical phenomenon, was replicated by a physical process in hundreds of papers (Simon, APS Fellow Howard C. Nusbaum’s research focuses on the basic cognitive and neural processes of communication, learning, thought, 2014; Tretkoff, 2007), but in spite of replications, there were and wisdom. Nusbaum is Director of the Chicago Center for Practical skeptics. A skeptical physicist visited Blondlot and by simple Wisdom, which supports basic research and facilitates discussions of the intervention showed that the only real phenomenon was importance of wisdom research and understanding wisdom. He can be observation bias. contacted via [email protected]. Association for Psychological Science January 2017 — Vol. 30, No. 1 6
When theory and knowledge impel replication, they can laboratory experiments in economics. Science, 351, 1433– redress scientific error, but reliability and validity are not 1436. the same. Statistical analyses raise the question of whether Cook, F. L. (2016). Dear Colleague Letter: Robust and reliable some results are too good to be true (Francis, Tanzman, & research in the social, behavioral, and economic sciences. Retrieved from https://www.nsf.gov/pubs/2016/nsf16137/ Matthews, 2014). This kind of analysis essentially questions nsf16137.jsp whether reported results are statistically plausible. The results Francis, G. (2014). Too much success for recent groundbreaking do not indicate error as such but simply flag what appear to be epigenetic experiments. Genetics, 198, 449–451. improbable findings that might not be replicated. Francis, G., Tanzman, J., & Matthews, W. J. (2014). Excess This is not a problem for psychology alone and has arisen in success for psychology articles in the journal Science. PLOS genetics (Francis, 2014). Gregor Mendel’s data were also too good One, 9, e114255. to be true (Broad & Wade, 1982; Gelman, 2012). But Mendelian Gelman, A. (2012). Gregor Mendel’s suspicious data [Blog post]. genetics withstand the test of time. Statistics bolster an argument Retrieved from http://andrewgelman.com/2012/08/08/gregor- but do not represent the whole truth of a result. Given that there mendels-suspicious-data/ are many ways for errors to distort research and derail progress, Gilovich, T., Griffin, D., & Kahneman, D. (Eds.). (2002). we need to understand how social, cultural, and psychological Heuristics and biases: The psychology of intuitive judgment. forces work in science. Cambridge, UK: Cambridge University Press. Science must generate believable and robust knowledge. Goodman, A. (2006). Intuition. New York, NY: The Dial Press. The NSF Advisory Committee to the Social, Behavioral, Kurose, J. (2016). Dear Colleague Letter: Encouraging & Economic Sciences Directorate (SBE) established a Sub- reproducibility in computing and communications research. Retrieved from https://www.nsf.gov/pubs/2017/nsf17022/ committee on Replicability in Science. Their report (Bollen et nsf17022.jsp al., 2015) defined robust findings as reproducible, replicable, and Nickerson, R. S. (1998). Confirmation bias: A ubiquitous generalizable with clear definitions. This report was a call to support phenomenon in many guises. Review of General Psychology, new robust and reliable science, and SBE has posted a “Dear Col- 2, 175–220. league Letter” (DCL; Cook, 2016) announcing support for research Simon, M. (2014). Fantastically wrong: The imaginary radiation on failures of robustness, methods to improve robustness, training that shocked science and ruined its ‘discoverer.’ Wired. to enhance robustness of research, and support for replications/ Retrieved from https://www.wired.com/2014/09/fantastically- generalizations of key SBE studies. Given that replicability concerns wrong-n-rays/ extend to other social sciences (Camerer et al., 2016), SBE is com- Tretkoff, E. (2007). This month in physics history: September mitted to improving robustness of SBE sciences. A DCL from the 1904: Robert Wood debunks N-rays. American Physical Society. Retrieved from https://www.aps.org/publications/ Directorate for Computer and Information Science and Engineering apsnews/200708/history.cfm (Kurose, 2016) announces support for reproducibility in computing Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: and communications research. Heuristics and biases. Science, 185, 1124–1131. Science depends on credibility. There are, however, many ways that findings can fail replication, and not all compromise validity. Science requires humility given uncertainty and unknowns outside current research. The SBE DCL supports increasing robust research and greater reflection about robust science generally and hopes to lead to a wiser approach to research. The problem of robust sci- ence is not unique to the social and behavioral sciences; it inheres in all sciences and does so because all science is conducted by Featuring: scientists — physicists and geneticists are human and thus subject to Jennifer A. Richeson the social and psychological forces that can lead research astray. That Yale University alone creates a unique and important responsibility for understanding the conduct of robust science in the social and behavioral sciences. One the field’s foremost researchers on the psychological phenomena of cultural References diversity reflects on her career and her Bollen, K., Cacioppo, J. T., Kaplan, R. M., Krosnick, J. A., Olds, future research plans. J. L., & Dean, H. (2015). Report of the Subcommittee on the Replicability in Science Advisory Committee to the National Interviewed by: Wendy Berry Mendes Science Foundation Directorate for Social, Behavioral, & University of California, San Francisco Economic Sciences. Retrieved from https://www.nsf.gov/ sbe/AC_Materials/SBE_Robust_and_Reliable_Research_ See these and the rest of the collection of interviews with leaders Report.pdf in psychological science at www.psychologicalscience.org/itps Broad, W., & Wade, N. (1982). Betrayers of the truth: Fraud and deceit in the halls of science. New York, NY: Simon & Schuster. Camerer, C. F., Dreber, A., Forsell, E., Ho, T.- H., Huber, J., Johannesson, M., … Wu, H. (2016). Evaluating replicability of
January 2017 — Vol. 30, No. 1 Association for Psychological Science OBSERVATIONS 7 Scientists Explore How Nutrition May Feed Mental Health
Good nutrition has long been viewed as a cornerstone of physi- cal health, but research increasingly is showing diet’s effect on mental health as well. A special section in Clinical Psychological Science highlights the different approaches that psychology researchers are taking to understand the many ways in which nutrition and mental health intersect. Decades of research have shown the importance of proper nutrition in preventing and treating the ill effects of inflammation and stress, physiological processes that are inti- mately linked with mental health. Despite this clear connection, diet and metabolism typically do not feature in studies that examine aspects of psychological well-being. “Nutrition is not mainstream within the sciences that study These findings raise the question of whether children with ADHD mental health and illness,” psychological scientist Alan Kazdin, process nutrients in ways that differ from those of other children. who organized the Special Section during his tenure as Clinical Focusing on potential nutritional interventions for Psychological Science Editor, notes in his introduction. “Standard obsessive-compulsive disorder (OCD), Jerome Sarris (The Uni- coursework in training and exposure to the scientific literature versity of Melbourne, Australia) and colleagues examined effects in the traditional mental health professions omit even a morsel. of treatment with an amino acid agent called N-acetyl-cysteine A single series of papers cannot redress that. Yet we can make (NAC) in a randomized controlled trial with 44 participants. The salient key questions and convey there are answers.” data showed no overall difference between NAC and placebo in The aim of this collection of articles is to “showcase the reducing OCD symptoms, but subgroup comparisons indicated diversity of studies being conducted in a new, rapidly emerg- that younger participants and those who had been diagnosed for ing field of nutrition and mental health,” write guest editors a shorter period of time were more likely to show improvement Julia J. Rucklidge (University of Canterbury, New Zealand) in response to NAC. The researchers suggest that further studies and Bonnie J. Kaplan (University of Calgary, Canada) in their with larger samples are necessary to determine the utility of NAC introduction to the special section. as an adjunct to OCD treatment. The five articles included in the special section investigate Joanna S. Lothian, Neville M. Blampied, and Julia J. the intersection of nutrition and mental health from varying Rucklidge, all of the University of Canterbury, New Zealand, ex- levels and perspectives. plore broad-spectrum micronutrients (i.e., vitamins and miner- Building on previous research showing the beneficial effects of als) as treatment for insomnia, a condition that is associated a Mediterranean-style diet — rich in fruits and vegetables, healthy with a variety of mental health problems. In an 8-week trial, the fats, nuts, and fish — Almudena Sánchez-Villegas (University researchers investigated outcomes associated with the use of a of Las Palmas de Gran Canaria, Spain) and colleagues examine commercial micronutrient supplement in a group of 14 adults outcomes associated with a broader Mediterranean lifestyle that who reported symptoms of insomnia. Participants reported includes diet, physical activity, and social activity. Looking at improvements with insomnia symptoms, mood, stress, and data from 11,800 individuals participating in a university-based anxiety over the course of the trial. The researchers note several longitudinal study, the researchers found that all of these variables limitations — including the fact that participants were aware of independently predicted a lower risk of depression. The article the treatment and the study lacked a control group — that should highlights the importance of examining the combined effects of be addressed in future research. nutritional and other lifestyle factors on mental health outcomes. Investigating the relationship between inflammation and Jane Pei-Chen Chang (King’s College London, United depression, Tasnime N. Akbaraly (INSERM, France) and col- Kingdom) and colleagues focus their research on another men- leagues examine data on diet and depressive symptoms collected tal disorder: attention-deficit/hyperactivity disorder (ADHD). from 4,246 adults over a period of 5 years. The researchers found Examining data from 21 children with ADHD and 21 children that diets that rated high on the “dietary inflammatory index” without ADHD, the researchers find complex relations among were associated with increased risk of depressive symptoms, children’s food consumption, physical symptoms, and cognitive but only among women. Specific biomarkers of inflammation performance. Although children with ADHD showed no dif- did not explain this association, despite their association with ference in essential fatty acid (EFA) intake compared with their dietary inflammatory index scores at baseline. non-ADHD peers, they did show signs of EFA deficiency. At the Please visit the Special Section on Nutrition and Mental same time, children who had lower EFA intake and symptoms Health at http://cpx.sagepub.com/content/current#SpecialSeries- of EFA deficiency were likely to show greater ADHD symptoms. NutritionandMentalHealth.
Association for Psychological Science January 2017 — Vol. 30, No. 1 8 OBSERVATIONS Linehan Receives Grawemeyer Award for Psychology
APS James McKeen Cattell Fellow Marsha M. Linehan, whose groundbreaking research has focused on developing interventions for teens at high risk for suicide and support networks for their families and friends, has won the 2017 University of Louisville Grawemeyer Award for Psychology. Linehan’s award-winning Dialectical Behavior Therapy (DBT) is an evidence-based cognitive behavioral program that helps patients focus on mediating their behavior with coping skills. Previous recipients of the Grawemeyer Award for Psychology include APS Past Presidents Elizabeth F. Loftus, Walter Mischel, and James L. McGaugh; APS Past Board Member Anne M. Treisman; APS William James Fellow and APS James McKeen Cattell Fellow Albert Bandura; APS James McKeen Cattell Fellows Aaron T. Beck and Irving I. Gottesman; APS William for suicide. Linehan has made it her mission to increase the amount James Fellows Daniel Kahneman, Mortimer Mishkin, Michael of empirical research dedicated to patients with mental disorders I. Posner, Amos Tversky, David E. Rumelhart, and Leslie G. by using robust study methods and by training graduate students Ungerleider; and APS Fellows Marcus E. Raichle, Steven F. to work with difficult cases early in their careers. Maier, Antonio Damasio, Steven E. Petersen, and Lynn Nadel. The psychological scientist has been open about how her own Linehan, who directs the Behavioral Research & Therapy Clin- suicidality and depression have influenced her work. “At a young ics, Center for Behavioral Technology, at the University of Wash- age, I vowed to get myself out of hell and then to go back and get ington, has worked for decades to develop rigorous clinical trials others out,” she said in the press release announcing the award. to study suicidal behaviors. In her award address at the 2015 APS In addition to undertaking this daunting mission, Annual Convention in New York City she noted that, as of 2013, Linehan has published seven books, including a widely employed only 83 randomized clinical trials had been conducted to measure handbook on DBT that has been translated into 10 languages. suicidal behavior. Furthermore, 23% of those trials excluded many To see Marsha Linehan’s 2015 APS Award address, visit www. of the individuals who most needed treatment, such as those with psychologicalscience.org/r/Linehan. depression, anxiety, or bipolar disorder as well as those at high risk
Major Change in NIH Policy for Clinical Trials Applications
In a significant departure from current practices, the National clinical trial applications through Parent Announcements (broad Institutes of Health (NIH) has issued new policies relating to grant funding opportunity announcements for investigator-initiated ap- applications involving clinical trials, including one (http://grants.nih. plications) or other FOAs that are not specifically designed to accept gov/grants/guide/notice-files/NOT-OD-16-147.html) mandating clinical trials. Each NIH Institute and Center (IC) will be required that all applications involving clinical trials must be in response to a to publish its own clinical trials FOA, which will emphasize that IC’s Funding Opportunity Announcement (FOA) specifically designed scientific area(s) of interest. for clinical trials. The changes will affect applications submitted on While NIH is making these changes in an effort to “to identify or after September 27, 2017. more easily clinical trial applications and ensure that key components This policy change will have a clear impact on the application of clinical trial information are included and uniformly considered process for psychological scientists, given that the NIH definition of in review,” vigilance from the scientific community is needed to a clinical trial (see http://grants.nih.gov/grants/guide/notice-files/ ensure that all areas of science are represented in the IC-specific NOT-OD-15-015.html) includes all research involving one or more FOAs involving clinical trials research. human subjects who are prospectively assigned to one or more inter- For more information on NIH policy announcements, check out ventions (that may include placebo or other control) to evaluate the the information posted at the NIH Office of Extramural Research, effects of those interventions on health-related biomedical or behav- Notices of NIH Policy Changes site (https://grants.nih.gov/policy/ ioral outcomes. Investigator-initiated applications involving clinical notices.htm). For information on the implications of the NIH clini- trials will no longer be accepted by NIH; instead, applicants seeking cal trials policies on behavioral and social sciences, see the Office of to conduct clinical trial research must find an FOA that specifically Behavioral and Social Sciences Research Director’s Voice blog from allows the submission of clinical trials and covers an area of scientific October 18, 2016, at https://obssr.od.nih.gov/new-nih-clinical-trials- interest relevant to the proposed research. NIH will no longer accept policies-implications-for-behavioral-and-social-science-researchers.
January 2017 — Vol. 30, No. 1 Association for Psychological Science OBSERVATIONS 9 Loftus Receives 2016 John Maddox Prize
APS Past President Elizabeth F. Loftus has been awarded the 2016 John Maddox Prize, which honors scientists who have shown courage in promoting science on a matter of public inter- est in the face of difficulty or hostility. The award, named after the late Nature Editor Sir John Maddox, recognizes Loftus for her groundbreaking studies on false memory and the unreliable aspects of eyewitness testimony. Loftus’s research, which in the 1990s challenged many people’s claims of repressed memories of childhood abuse, resulted in efforts to undermine her career and even threats to her life. “Standing up for psychological science in general, and research on memory in particular, has brought a good deal of hostility my way,” Loftus acknowledges in her comment explained the three words she would use to describe human on the prize’s webpage. “Receiving this honor helps to erase memory: “suggestive, subjective, and malleable.” the pain of insults, death threats, and lawsuits. And I love Loftus has published 23 books and over 200 articles on her that idea that, forever, my CV will contain the name of research, with titles such as “The Myth of Repressed Memory: the late Sir John Maddox, whom all respect for his tireless False Memories and Allegations of Sexual Abuse,” “Witness defense of science.” for the Defense: The Accused, the Eyewitness and the Expert Loftus’s 40-plus years of research have gained her some Who Puts Memory on Trial,” and “Eyewitness Testimony: Civil of the field’s most prestigious accolades and garnered her and Criminal.” She is an APS William James Fellow and APS popularity unheard of for most psychological scientists. She James McKeen Cattell Fellow, and her other honors include appeared on The Oprah Winfrey Show; her TED Talk, “How the Grawemeyer Award, the Distinguished Contribution to Reliable is Your Memory?,” has more than 3 million views; and Basic and Applied Scientific Psychology Award, the William she has served as an expert witness in hundreds of court cases, T. Rossiter Award for “exceptional global contribution to the including the high-profile criminal trials of Martha Stewart, field of forensic mental health,” and many more. Ted Bundy, and O. J. Simpson. To go Inside the Psychologist’s Studio with Elizabeth Loftus’s pioneering work on the “misinformation effect” has Loftus, please visit http://www.psychologicalscience.org/video/ illuminated for the field aspects of human memory that many elizabeth-loftus-itps.html#.WEgsiLIrJaQ. were unwilling to accept. In a Frontline interview with PBS, she
New Books Liberation Psychology, Technologies of Mind Management and Self Actualization by Denis Carville; Denis Carville, September 20, 2016.
Payoff: The Hidden Logic That Shapes Our Motivations by Dan Ariely; TED Books, Simon & Schuster, November 15, 2016.
Take Pride: Why the Deadliest Sin Holds the Secret to Human Success by Jessica Tracy; Houghton Mifflin Harcourt, September 20, 2016.
To submit a new book, email [email protected].
Association for Psychological Science January 2017 — Vol. 30, No. 1 10 OBSERVATIONS Two Priming Effects to be Examined in APS Registered Replication Reports With Combined Protocol
APS announces two new Registered Replication Report studies have not found moral or religious priming to have an (RRR) projects. Data for these two projects will be collected effect on subsequent deceptive behaviors. The proposers of the concurrently as part of a single protocol, and participating replication, Bruno Verschuere (University of Amsterdam, the laboratories will be listed as authors on both reports. These Netherlands), and Ewout Meijer (Maastricht University, the reports will be published in APS’s new journal, Advances in Netherlands), hope that a large, multicenter direct replication of Methodologies and Practices in Psychological Science, and they this study will help to provide clarity regarding moral priming. will replicate Experiment 1 from: The task in Mazar, Amir, and Ariely’s study was adminis- • Mazar, N., Amir, O., & Ariely, D. (2008). The dishon- tered as a part of a larger battery of tests in a large classroom esty of honest people: A theory of self-concept mainte- setting, conditions that must be met by the RRR study as well. nance. Journal of Marketing Research, 45(6), 633–644. This provides an opportunity to conduct another RRR as part and Experiment 1 from: of that battery. The second RRR will examine Thomas K. Srull • Srull, T. K., & Wyer, R. S. (1979). The role of category (University of Illinois at Urbana-Champaign) and Robert S. accessibility in the interpretation of information about Wyer’s (Hong Kong University of Science and Technology, persons: Some determinants and implications. Journal Hong Kong) seminal 1979 study. of Personality and Social Psychology, 37(10), 1660–1672. In that study, the authors examined the so-called assimi- In the 2008 study, Nina H. Mažar (University of Toronto, lation priming effect by first asking subjects to descramble Canada), On Amir (University of California, San Diego), and sentences that described either hostile or neutral behaviors. Dan Ariely (Duke University) showed that a quick and simple When subjects subsequently read a vignette about a man moral reminder significantly reduced cheating. Participants whose behaviors were ambiguous in their hostility, those who were given a problem-solving task and an incentive to perform had been primed with more hostile sentences judged these well. Those participants in the critical “cheat” condition were ambiguous behaviors as more hostile and rated the man as given an opportunity to report solving a greater number of more hostile as well. problems than they actually did, with no risk of being caught. Although the effects of “hostile priming” have been When those participants were primed with a moral reminder conceptually repeated in many experiments, recent concerns (to recall the Ten Commandments) versus a neutral reminder over the replicability of some social priming studies — includ- (to recall 10 books they read in high school) before complet- ing ones using sentence descrambling as a prime — inspired ing the task, the morally primed subjects reported solving Randy J. McCarthy and APS Fellow John J. Skowronski, 28% fewer problems. According to the authors, “the level both of Northern Illinois University, to propose a large-scale of dishonesty dropped when people paid more attention to replication to measure the true size of this effect. honesty standards” (p. 642). Researchers can learn more about the project at its Open Several labs have conducted studies extending this concept Science Framework page (https://osf.io/vxz7q/), which of religious and moral priming, and the influence of prim- includes the full protocol and all of the experimental ma- ing on honesty has been examined in a variety of contexts. terials. If you have any questions about this RRR or the However, a direct replication of the Mazar, Amir, and Ariely RRR process in general, you can email the RRR editors at (2008) study has never been published, and some recent [email protected].
Minds for Business A Blog on the Science of Work and Leadership www.psychologicalscience.org/minds
January 2017 — Vol. 30, No. 1 Association for Psychological Science CONGRATULATIONS, NEW APS FELLOWS
Patricia A. Areán June Gruber Jenae M. Neiderhiser University of Washington University of Colorado Boulder Pennsylvania State University Janet L. Barnes-Farrell Lawrence M. Hanser Helen Lisbeth Nielsen University of Connecticut RAND Corporation National Institute on Aging Dorthe Berntsen Mark L. Hatzenbuehler Kristina R. Olson Aarhus University, Denmark Columbia University University of Washington Irene V. Blair Jutta Heckhausen Sheryl L. Olson University of Colorado Boulder University of California, Irvine University of Michigan, Ann Arbor Adriel Boals Colette R. Hirsch Michael W. Otto University of North Texas King’s College, London, United Kingdom Boston University Richard E. Boyatzis Jennifer L. Hudson Franco Pestilli Case Western Reserve University Macquarie University, Australia Indiana University, Bloomington Gary L. Brase Lauri A. Jensen-Campbell Keith J. Petrie Kansas State University University of Texas at Arlington University of Auckland, New Zealand Joshua W. Buckholtz Cheryl R. Kaiser Stephen A. Petrill Harvard University University of Washington The Ohio State University Darrell L. Butler Kerry Kawakami Jonathan A. Plucker Ball State University York University, Canada Johns Hopkins University Deborah M. Capaldi Barbara Krahé Sara E. Rimm-Kaufman Oregon Social Learning Center University of Potsdam, Germany University of Virginia Regina M. Carelli Daniel A. Krauss Amanda J. Rose University of North Carolina at Chapel Hill Claremont McKenna College University of Missouri Pim Cuijpers Robin J. Lewis Karen Salmon VU University, Amsterdam, the Netherlands Old Dominion University Victoria University of Wellington, New Zealand Rudi De Raedt Chiang-Shan R. Li Arthur G. Samuel Ghent University, Belgium Yale University Stony Brook University, The State Tracy A. Dennis Filip Lievens University of New York Hunter College, The City University of New York Ghent University, Belgium Ronald Seifer Lisa M. Diamond Christopher J. Lonigan Brown University University of Utah Florida State University Deryn Strange Melanie M. Domenech Rodríguez Richard E. Lucas John Jay College of Criminal Justice, The City University of New York Utah State University Michigan State University Iroise Dumontheil René Marois Jennifer L. Tackett Northwestern University Birkbeck, University of London, United Kingdom Vanderbilt University Hillary Anger Elfenbein Mara Mather Jessica L. Tracy University of British Columbia, Canada Washington University in St. Louis University of Southern California Edgar Erdfelder Ian McGregor Brian E. Vaughn Auburn University University of Mannheim, Germany University of Waterloo, Canada Francesca Gino James M. McQueen Qi Wang Cornell University Harvard University Radboud University Nijmegen, the Netherland Rebecca L. Gómez Rashmita S. Mistry Michaela Wänke University of Mannheim, Germany University of Arizona University of California, Los Angeles Mark T. Greenberg Robert L. Munroe Chen Yu Indiana University Bloomington Pennsylvania State University Pitzer College
For a complete list of APS Fellows, visit www.psychologicalscience.org/r/Fellows. Presents Improving the Reproducibility of Our Research Practices Using Open Science Framework
With: APS Fellow Brian Nosek University of Virginia Executive Director, Center for Open Science Courtney Soderberg Statistical and Methodological Consultant Center for Open Science
In this six-part workshop, APS Fellow Brian Nosek and Courtney Soderberg of the Center for Open Science review laboratory and personal research practices to improve reproducibility. Topics in- clude project and data management, preregistration, managing collaborations, and getting the most out of the Open Science Framework for private and public laboratory operations. The workshop was recorded at the 28th APS Annual Convention in Chicago in 2016.
Chapter 1: Introduction Chapter 2: Setting Up a Collaborative Research Space Chapter 3: Pre-Registration and Pre-Analysis Plans Chapter 4: Documenting Your Research Project Chapter 5: Sharing Your Work Chapter 6: Incentives for Behavior That Research Can Take Advantage Of
Available now at www.youtube.com/user/PsychologicalScience.
Presented with support from SAGE Publications 13 APS–Psychonomic Society W.K. & K.W. Estes Lecture Finding Common Ground
Presents Between Classic Improving the Reproducibility of Our Research Learning Theories Practices Using Open Science Framework PS Past President Gordon H. Bower is a pioneering Estes posited that stimulus With: psychological scientist and a National Medal of Science aspects arise from various fac- Awinner — but even luminaries like Bower have their tors such as the experimental APS Fellow Brian Nosek mentors. In the APS–Psychonomic Society W. K. & K. W. Estes apparatus used, subjects’ internal University of Virginia Lecture at the 2016 APS Annual Convention in Chicago, Bower states, and extraneous stimuli spoke about his relationship with the late William K. Estes and not associated with any outcome. Executive Director, Center for Open Science his work to build on Estes’s theories, a journey of both personal Over the course of his and scientific gains. talk, Bower described in great Estes himself was a pioneer in mathematical psychology, a depth the apparent differences Courtney Soderberg National Medal of Science winner, and the Founding Editor of between the two theories and Statistical and Methodological Consultant APS’s flagship journal, Psychological Science. His wife, Kather- how, via a process of careful Center for Open Science ine “Kay” W. Estes, was the Founding Managing Editor of the translation, the two theories journal. give rise to virtually identi- During his address, Bower described time he spent relating cal mathematical equations, In this six-part workshop, APS Fellow Brian Nosek Estes’s research to that of influential learning theorist Clark L. bridging the gap between the Bower and Courtney Soderberg of the Center for Open Hull. Bower discussed his time in graduate school at Yale Uni- two theories, and, according to Bower, putting Estes’s stimulus versity, where Hull’s stimulus–response theory was regarded as sampling theory “on an equal footing with the theories of Hull Science review laboratory and personal research the major theory of learning and memory at the time. But when and [Edward] Tolman that were so dominant in those waning practices to improve reproducibility. Topics in- Bower met Estes during a workshop at Stanford University, he days of the grand learning theories.” clude project and data management, preregistration, managing collaborations, and getting brought back to Yale a newfound appreciation for Estes’s theory of Eventually, Bower and his colleagues moved on from his learning, called the stimulus sampling theory. Bower soon became theory-bridging efforts to other topics. the most out of the Open Science Framework for private and public laboratory operations. committed to bridging the gap between these two theories. “Although such speculations might have been noteworthy at The workshop was recorded at the 28th APS Annual Convention in Chicago in 2016. According to Bower, Hull’s theory arose from psychology’s the time, I never published any of it and Bill published only a little behaviorism movement as an attempt to determine fundamental bit … in retrospect, the speculations are very much confined to principles of learning arising from hundreds of studies being their historical time capsule. Moreover, beginning in 1960, Bill Chapter 1: Introduction conducted on animal conditioning. Hull believed that research- and I both joined the cognitive revolution and essentially moved ers could summarize the effect of learning conditions to obtain a on to studies of human memory and category learning.” Chapter 2: Setting Up a Collaborative Research Space measure called degree of conditioning. Hull’s degree of condition- Bower ended his talk by reflecting on the personal importance ing was defined as a product of habit strength due to past training, of the time he spent working with Estes, noting that Estes’s men- Chapter 3: Pre-Registration and Pre-Analysis Plans the distinctiveness or intensity of the conditioned stimulus, the torship fostered his strong interest in mathematical psychology. animal’s relevant motivation level, the magnitude and the quality “Not only was [Estes] a superbly creative scientist, he had an Chapter 4: Documenting Your Research Project of the reward, and the delay of the reward following the response. enormous impact on our field. I feel lucky to have known him “Hull always viewed his set of principles not as an end in so well for so many years,” he finished. Chapter 5: Sharing Your Work themselves but as the basis for deriving explanations for many To watch Bower’s full APS–Psychonomic Society W. K. & other behavioral observations,” said Bower. K. W. Estes Lecture, please visit www.psychologicalscience. Chapter 6: Incentives for Behavior That Research Can Take Advantage Of On the other hand, Estes’s theory was more abstract, Bower org/r/bower. To watch Bower’s “Inside the Psychologist’s Studio” said. According to Estes, learning gave rise to “a large popula- interview, please visit http://www.psychologicalscience.org/ tion of dozens of little stimulus aspects … called stimulus observer/gordon-bower-itps. To read a remembrance of William elements.” On any given learning trial, only a sample of these K. Estes, please visit www.psychologicalscience.org/observer/ Available now at www.youtube.com/user/PsychologicalScience. elements are noticed — kind of like drawing marbles from a remembering-william-k-estes. bag. When a majority of these marbles are associated with a Lastly, if you wish to donate to the W. K. & K. W. Estes Fund, certain response, the response occurs. Learning increases as please visit www.psychologicalscience.org/fund-new/index. these stimulus elements become more and more associated cfm?FundID=WKKW. Presented with support from SAGE Publications with certain responses. -K. Andrew DeSoto
Association for Psychological Science January 2017 — Vol. 30, No. 1 14 Science in Policy Harnessing the Wisdom of Crowds to Improve Hiring By Alexandra Michel
t least as far back as the 19th century, statisticians The team’s first set of experiments on innovating hiring found that groups of people are capable of making practices was inspired by a nearly 200-year-old statistics experi- A a more accurate decision than any single individual. ment. The English statistician Francis Galton famously asked Yet organizations rarely take advantage of this “wisdom of the 800 townsfolk at a county fair in Plymouth, England, to guess crowd” to improve operations. the exact weight of an ox down to the pound. Guessers wrote In a new program, behavioral scientists at the Behavioural their estimates on bits of paper, which Galton then analyzed. Insights Team (BIT) in the United Kingdom are harnessing Galton had hypothesized that “oxen experts” such as farmers and the decision-making power of groups to improve the way that butchers would have the best estimates, but the crowd proved organizations conduct hiring. Originally commissioned under this assumption wrong: The group’s estimate was more accurate Prime Minister David Cameron in 2010, the BIT is a leading than the individual guesses from experts. example of government testing of public and organizational Galton was shocked by the results: The median of all 800 policy interventions through evidence-backed collaborations guesses was extremely close to the exact weight of the ox. The with behavioral scientists. Across a series of experiments, the BIT group estimate was 1,197 pounds, and the actual weight of the has been investigating how findings from behavioral science can ox was 1,198 pounds — a difference of just 0.08%. be used to help organizations, including the BIT itself, improve This “wisdom of the crowd” demonstrated that, under the hiring practices. right conditions, groups of people can make more insightful “Organisations spend eye-watering sums trying to attract the decisions than individuals can, sometimes even besting the best talent because in many industries, the difference between experts. Glazebrook and colleagues suspected that integrating the best and the good has real implications for the bottom line,” more people into the hiring process could similarly improve BIT behavioral scientists Kate Glazebrook, Theo Fellgett, and the chances of picking out the best possible candidates from the Janna Ter Meer write in a BIT blog post. pool of applications. Many hiring decisions come down to superficial criteria, Alexandra Michel is a science writer at the Association for such as choosing to interview only graduates from certain Psychological Science. She can be reached at apsobserver@ universities or unconsciously favoring candidates based psychologicalscience.org.
January 2017 — Vol. 30, No. 1 Association for Psychological Science 15 on traits like gender, ethnicity, or sexual orientation. By about a time when you used your initiative to resolve a difficult focusing on these perfunctory traits, organizations may miss situation.”). The reviewers were given a set of guidelines, similar out on highly qualified candidates. Additionally, these kinds of to those given to conduct a structured interview, to help them homogenous hiring practices can lead to situations in which assess the quality of responses. employers miss out on the benefits that come from a diversity The 400-person crowd had a clear favorite and easily identi- of perspectives. When everyone addresses problems in the fied the best candidate response. same way (i.e., “groupthink”), teams can end up missing major “We took our data and ran statistical simulations to estimate concerns altogether. the probability that different groups could correctly select Although companies are increasingly aware of the benefits the best candidate,” Glazebrook and colleagues explain. “We of a diverse workforce, actually translating these goals into created 1,000 combinations of reviewers in teams of different hiring practices has been a challenge. The BIT wanted to find sizes, ranging from one to seven people. We then pooled them out whether they could build better, more diverse teams by by the size of the group and averaged their chance of selecting adopting a hiring strategy that could take advantage of the the right candidate.” wisdom of crowds. When there was a gap in quality between the best and “One area of crucial importance to almost all organisations second-best responses, an individual picked the less qualified is recruitment, but research shows that a whole host of implicit person approximately 16% of the time. However, with a group biases result in suboptimal hiring decisions,” the BIT explains of three decision-makers, the odds of choosing the lesser can- in their 2016 report. “Studies have shown that organisations didate dropped to 6%, and with a five-person group, the chance are more likely to offer job interviews to candidates with decreased to 1%. When the two candidates were very similar, ‘white-sounding’ names. Recruiters make snap judgements individuals selected the best candidate approximately 50% of about individuals in interviews, and structure recruitment the time — basically, they had the same accuracy as tossing a processes (e.g. sending a cover letter and CV) in ways that coin. A crowd of seven, on the other hand, picked the superior give too much weight to factors (gender, race, social class) candidate more than 70% of the time. that should be irrelevant to an individual’s ability to do a role.” Of course, polling 400 reviewers for every job isn’t very prac- Research from APS Fellow Philip E. Tetlock (University tical. Ultimately, the evidence suggested that three reviewers was of Pennsylvania) has demonstrated that people are better at the optimal crowd size for recruitment, but more experiments forecasting outcomes when they work together in collab- are still in the works. orative teams. Tetlock and colleagues have spent years studying decision-making and expertise. One of their key findings has Turning the Science Inward been that pooling multiple perspectives can counter the cogni- “The Behavioural Insights Team likes to live by its own principles. tive biases that lead to bad decisions. The BIT drew on Tetlock’s When we examined the literature on how organizations can research to help inform their own approach to bias in hiring. improve their internal practices, we realized we had to apply “In fact, researchers have even shown that US defense them to BIT as well,” the team explains. intelligence analysts with access to classified information can To this end, the BIT has developed a platform called be beaten by some rudimentarily-educated amateurs: largely Applied. The goal of this project is to use findings from because they come to conclusions too quickly and struggle behavioral science to reduce the role of bias in the hiring to update their opinions in the face of new and conflicting process. information,” Glazebrook, Fellgett, and Ter Meer explain in Most job searches start with an applicant submitting their their BIT blog post. resume or CV along with a cover letter. Someone in human Research also suggests that people with varied backgrounds resources then sorts through the pool of applicants, narrowing it and experiences will tackle problems differently, and this diver- down to a set of individuals who will be invited for an interview. sity of perspectives can help organizations make better decisions. But the small experiment described above simply doesn’t sup- A team of psychological scientists led by APS Fellow Adam D. port the standard CV sift as a particularly useful hiring tool. For Galinsky (Columbia University) recently summarized empirical example, a candidate with a degree from a prestigious private arguments for more diverse teams in Perspectives in Psycho- university on his or her CV may be chosen over someone equally logical Science: “Homogeneous groups run the risk of narrow qualified who attended a state university, or a candidate with mindedness and groupthink (i.e., premature consensus) through a typically masculine name may be assumed to have greater misplaced comfort and overconfidence. Diverse groups, in leadership potential compared with a female job candidate. contrast, are often more innovative and make better decisions, “With respect to CVs in particular, research argues that in both cooperative and competitive contexts.” CVs typically contain information that is largely irrelevant So when it comes to reviewing resumes and interviewing appli- to a candidate’s performance on the job. Nevertheless, this cants, how big does the crowd need to be to maximize the benefits? information has the potential to prey on the unconscious biases The BIT designed a simple online experiment in which ap- of the assessor,” the BIT explains in their 2016 report. proximately 400 reviewers rated four hypothetical job candidates The Applied platform attempts to increase qual- based on responses to a generic recruiting prompt (i.e., “Tell me ity and diversity in hiring through implementing four key
Association for Psychological Science January 2017 — Vol. 30, No. 1 16 A
features: anonymization, chunking, collective intelligence, and There was a marked difference in the role of educational Nomination Deadline: predictive assessment. background between the two groups. While the CV sift favored First, the platform anonymizes applications by scrubbing applicants based on formal educational attainment, those who irrelevant information such as names (which can provide cues made it through the Applied sift had a much more diverse about an applicant’s gender, race, age, or ethnic background). educational background — that is, the people who had the most The applications then are organized by “chunking” — instead years of higher education didn’t necessarily have the best skills of reading through one full application at a time, reviewers for the job. This finding is in line with trends from companies compare a specific question from an application with the like Google and IBM, where formal college education or uni- same question from other applications. This helps reviewers versity grades increasingly are viewed as irrelevant predictors of to identify the overall best responses. someone’s performance on the job. Last, three or more people review the remaining pool of “We would never have hired (or even met!) a whopping 60 applicants. Agreement of multiple reviewers helps ensure that percent of the candidates we offered jobs to if we’d relied on their Recognizing Outstanding Ideas In the best possible candidate is ultimately chosen. Job assessments CVs alone,” Glazebrook and Ter Meer write. and situational work tests are chosen based on whether there’s Of course, more evidence is required to demonstrate that evidence showing that specific tests are “genuinely predictive this hiring approach will actually translate into on-the-job per- 100,000 Psychology of performance on the job.” formance. The cohort of hires at Applied is too small to use as a The Applied platform isn’t just for private organizations meaningful test of the platform’s capabilities in the real world, but and businesses: The BIT has used the platform to improve their the Applied team is looking for opportunities to run a larger test. T N I own hiring practices. On a national scale, the use of this kind of bias-limiting ap- A program is proud to announce the 2018 . A one-to-two-page letter of nomination, in proach could have an enormous impact on helping individuals award in Psychology. This award recognizes English, identifying the speci c idea being Can You Take the Bias Out of Hiring? get the jobs for which they’re qualified. As Glazebrook and Ter In an experiment to determine whether Applied was doing Meer explain, “even if 1 in 5 candidates were given jobs that they outstanding ideas in all areas of Psychological nominated and delineating the reasons why what it was supposed to, BIT researchers tested the platform otherwise wouldn’t have, across the economy, that’s hundreds of Science. The award is designed to recognize a the idea merits the award, based on the against a more traditional “CV sift” during their own 2015–2016 thousands of people getting jobs they otherwise wouldn’t have speci c idea, rather than a lifetime of criteria above. graduate recruitment period. based on merit.” accomplishment. Nominations are judged on First, the team designed a parallel A/B test of the 160 . A current mailing address, telephone candidates who had the best performance on an initial References the basis of originality, creativity, scienti c merit, number, fax number and e-mail address for multiple-choice test. The application materials for all candidates Behavioural Insights Team (2015). The Behavioural and breadth of impact on the discipline. were sent through both the automated Applied review and the Insights Team Update Report 2015–16. Retrieved from the nominee. http://38r8om2xjhhl25mw24492dir.wpengine.netdna- normal “sift” from a senior HR manager who reviewed CVs cdn.com/wp-content/uploads/2016/09/BIT_Update_ T N and resumes. The resulting pool of successful candidates was S N Report_2015-16-.pdf. The University invites nominations from then sent through a rigorous set of skill assessments and final Galinsky, A. D., Todd, A. R., Homan, A. C., Phillips, K. W., in-person interviews. throughout the world by individuals, Apfelbaum, E. P., Sasaki, S. J., ... Maddux, W. W. (2015). Director, Psychology Grawemeyer Award Ultimately, this process gave applicants two shots to get Maximizing the gains and minimizing the pains of professional associations, university hired: They could make it through the traditional review diversity: A policy perspective. Perspectives on Psychological Dept. of Psychological and Brain Sciences Science, 10, 742–748. doi:10.1177/1745691615598513 administrators, and publishers or editors of process based on an exceptional CV, or they could be chosen University of Louisville based on the scores from the evidence-based hiring tests used Glazebrook, K., Fellgett, T., & Ter Meer, J. (2016, February journals and books in Psychology. 17). Would you hire on the toss of a coin? Retrieved from Louisville, KY 40292, U.S.A. by Applied. Self-nominations are not permitted. Upon http://www.behaviouralinsights.co.uk/labour-market-and- T 502 852-0430 “When we pulled all of the data in, lots of things surprised economic-growth/would-you-hire-on-the-toss-of-a-coin/ receipt of their nomination, nominees will be u s ,” Glazebrook and Ter Meer write in a post on Medium. 502 852-8904 Glazebrook, K., & Ter Meer, J. (2016, September 21). Putting noti ed about the award conditions, the There was no correlation between the score for an applicant’s Applied to the test — Part 1. Medium. Retrieved from [email protected] CV and in-person performance in later rounds. Simply having https://medium.com/finding-needles-in-haystacks/putting- selection process and the supporting applied-to-the-test-part-1-9f1ad6379e9e#.6dut0omuw www.grawemeyer.org/psychology/ an impressive CV with recognizable schools and fancy titles materials needed. was a weak predictor for test scores during the other assess- Glazebrook, K., & Ter Meer, J. (2016, October 3). Can ments. There was, however, a significant, positive relationship technology improve diversity? Putting Applied to the test — Part 2. Medium. Retrieved from https://medium.com/ between the Applied scores and the two in-person interview PRIOR WINNERS finding-needles-in-haystacks/can-technology-improve- rounds — that is, people with high Applied scores on their diversity-putting-applied-to-the-test-part-2-a6fb98c26778#. 2001 Michael Posner, Marcus Raichle & 2008 Albert Bandura application materials also performed well in person. k367wpl3z Steven Petersen 2009 Anne Treisman But did Applied actually come through on delivering a more Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, 2002 James McClelland & David Rumelhart 2010 Ronald Melzack diverse set of hires? While there wasn’t a significant difference N., Bishop, M., … Tetlock, P. (2015). Identifying and 2003 Daniel Kahneman & Amos Tversky Walter Mischel between the two hiring groups on gender, there was evidence cultivating superforecasters as a method of improving 2011 that Applied was less biased against people with a disability and probabilistic predictions. Perspectives on Psychological 2004 Aaron Beck 2012 Leslie Ungerleider & Mortimer Mishkin Science, 10, 267–281. doi:10.1177/1745691615577794 people from non-White backgrounds, although the sample sizes 2005 Elizabeth Loftus 2013 Irving Gottesman were too small to provide a statistically significant conclusion Tetlock, P. E., & Gardner, D. (2015). Superforecasting: The Art 2006 Lynn Nadel & John O’Keefe 2014 Antonio Damasio and Science of Prediction. New York, NY: Crown. for these measures. 2007 Giacomo Rizzolatti, Vittorio Gallese & 2015 James McGaugh Leonardo Fogassi 2016 Steven Maier
January 2017 — Vol. 30, No. 1 Association for Psychological Science A Nomination Deadline:
Recognizing Outstanding Ideas In