Conscious robots may merit legal protections.

Developing machine capacities such as artificial intelligence (AI) and robots for human-robot interaction is of extreme technological value, but it raises moral and ethical questions. For example, one of the major sectors in robotics is the develop- ment of sexual robots (1). Although some researchers consider this phenomenon a gateway to the future acceptance of human- robot interactions, others see a danger for human society as robots modify the social LETTERS representation of human behaviors (2, 3). The robotics field must wrestle with these questions: Is it ethical to create and con- tinue to use robots with consciousness in Edited by Jennifer Sills global projection of winning representa- the way we use robots that were originally

tions and have metacognitive capacity, designed for our needs? Do robots deserve Downloaded from such as confidence estimates. If these types to be protected? This debate could challenge Conscious machines: of processes were strongly indicative of the limits of human morality and polarize consciousness, we would have to admit society’s views on whether conscious robots Defining questions that some machines are already conscious. are objects or living entities (4–6). In their Review “What is consciousness, We agree with Dehaene et al. that to As we approach an era when conscious

and could machines have it?” (27 October address the question of machine conscious- robots become part of daily life, it is http://science.sciencemag.org/ 2017, p. 486), S. Dehaene et al. argue that ness, we must start with a theory of human important to start thinking about the cur- the science of consciousness indicates that consciousness. Given current disagreement rent status of robots today. The purpose of we are not on the verge of creating con- on that topic, we are all left to speculate granting legal status to robots is not only scious machines. However, Dehaene et al. whether machines will ever be conscious. A to prevent inappropriate human-robot ask and answer the wrong questions. more pertinent question for the field might interactions but also to recognize and To determine whether machines are be: “What would constitute successful formalize the role of robots in society, thus conscious, we must ask whether they demonstration of artificial consciousness?” normalizing their existence (7). have subjective experiences: Do machines Olivia Carter,1 Jakob Hohwy,2 Jeroen Nicolas Spatola* and Karolina Urbanska consciously perceive and sense colors, van Boxtel,2 Victor Lamme,3 Ned Block,4 Université Clermont Auvergne, CNRS, LAPSCO, F-63000 Clermont-Ferrand, France. sounds, and smells? Do they feel emotions? Christof Koch,5 Naotsugu Tsuchiya2* *Corresponding author. Email: [email protected] Unfortunately, Dehaene et al. relegate 1University of Melbourne, Melbourne, VIC 3010, on January 25, 2018 2 this issue to the final paragraph of their Australia. Monash University, Mebourne, VIC REFERENCES 3800 Australia. 3University of Amsterdam, 1018 XA Review, dismissing it as a philosophical Amsterdam, Netherlands. 4New York University, 1. M. Scheutz, T. Arnold, “Intimacy, bonding, and sex robots: question “beyond the scope of the present New York, NY 10003, USA. 5Allen Institute for Brain Examining empirical results and exploring ethical rami- fications” (2017); https://hrilab.tufts.edu/publications/ paper.” Instead, they ask whether machines Science, Seattle, WA 98103, USA. *Corresponding author. scheutz2017intimacy.pdf. “mimic” consciousness by exhibiting the Email: [email protected] 2. J. Robertson, Body Soc. 16, 1 (2010). global availability of information (the 3. M. Coeckelbergh, Int. J. Soc Robot. 1, 217 (2009). REFERENCES 4. P. Lin et al., Eds., Robot Ethics (MIT Press, 2012). ability to select, access, and report infor- 5. A. L. Peláez, D. Kyriakou, Technol. Forecast. Soc. Change 75, mation) and metacognition (the capacity 1. C. Koch et al., Nat. Rev. Neurosci. 17, 307 (2016). 1176 (2008). for self-monitoring and confidence estima- 2. N. Tsuchiya et al., Trends Cogn. Sci. 19, 757 (2015). 6. K. Richardson, Comp. Soc. 45, 290 (2016). 3. M. Boly et al., J. Neurosci. 37, 9603 (2017). 7. I. Yeoman, M. Mars, Futures 44, 365 (2012). tion). Questions concerning to what extent 4. B. Odegaard, R. T. Knight, H. Lau, J. Neurosci. 37, 9593 (2017). machines have these capacities are inter- 10.1126/science.aar5059 esting, but neither capacity is necessary 10.1126/science.aar4163 or sufficient for subjective experience (1, Response 2). Furthermore, Dehaene et al.’s emphasis on metacognition and global broadcasting Conscious machines: Any discussion of machine consciousness presumes that the prefrontal cortex is the should start with empirical evidence, and home of consciousness, which remains a Robot rights our Review primarily consists of an empiri- matter of debate (3, 4). In their Review “What is consciousness, cal look at how nonconscious and conscious Finally, when arguing that machines are and could machines have it?” (27 October processing differ in humans [see also (1, 2)]. not yet conscious, Dehaene et al. highlight 2017, p. 486), S. Dehaene et al. suggest In contrast to Carter et al.’s interpretation of the “feedforward”—i.e., sequential—nature that machine consciousness, which would our conclusions, we suggest that conscious of information processing typical of these model human cognitive functions within subjective states are in fact on the verge of S I M computing systems. Although current a physical architecture other than the becoming implementable in machines and A M / I R O machines may not exhibit propagation of human brain, has not yet been achieved. that two computational ingredients (global T PH O information into the broadcasting network Creating consciousness is still a goal for information sharing and self-monitoring), if O: A P

hub or formal Bayesian metacognition, the future, but now is the time to consider jointly and correctly implemented, may pro- T

many artificial intelligence systems involve the implications of conscious machines. vide machines with conscious subjectivity. P H O

400 26 JANUARY 2018 • VOL 359 ISSUE 6374 sciencemag.org SCIENCE

Published by AAAS

DA_0126Letters.indd 400 1/24/18 10:49 AM INSIGHTS

Carter et al. claim that “neither capac- shortcuts in otherwise intractable compu- (11). In the future, denying machines any ity is necessary or sufficient for subjective tations (7). Thus, such subjective percepts form of subjectivity, when it is caused experience,” but that is begging the would arise in any efficient machine. by computations similar to those that question. It is possible that the subjec- To move forward, Carter et al. point out constitute core ingredients of conscious- tive experiences of humans are simply that it would be advantageous if we could ness in the human brain, may become as information-bearing representations with agree on the criteria that demonstrate contentious as denying it to other human the same specific properties we used for consciousness. But it is precisely because beings or to nonhuman animals with brain those of machines: being globally available we lack such a consensus that we think it is architectures similar to ours. and therefore reportable, and entering into best to start with empirical evidence based We therefore agree with Spatola and dedicated self-monitoring processes capa- on the known features of the human brain. Urbanska that the predictable emergence of ble of evaluation and criticism. Empirical Rapid progress in mapping the human conscious machines calls for an immediate evidence suggests that whenever human brain mechanisms of consciousness has consideration of its societal consequences. subjective experience is impaired, such as indeed revealed an important contribution The potential benefits should not be in psychosis (3) or blindsight (4), aspects of of the prefrontal cortex [e.g., (8–10)]. neglected: A powerful sentient artificial both of these processing functions are also When machines share enough features intelligence (AI) may collaborate with disrupted. Thus, our working hypothesis with conscious human brain process- humans in addressing major issues such is that subjective experience comes down ing, we should be prepared to accept the as energy management, ecology, or care in to nothing but a combination of specific possibility that they are conscious. Of an aging society. The risks, however, are forms of processing (including reality course, even if a machine shared those equally real and include job loss, concen-

monitoring as a crucial component). features and reported having subjective tration of power in a few hands, a military Downloaded from It has been suggested that failing to experiences, Carter et al. could still deny arms race, and social upheaval as humans recognize that machines may have subjec- that it experienced anything at all. But and AI increasingly compete for the same tive states reflects a lack of imagination such a solipsist position also applies to societal roles. Mitigating these disorders (5). In fact, many if not all visual illusions, humans: By such a standard, we likewise will require a major international effort, in which subjective perception diverges cannot prove that other human beings are and we can only heed here the conclu-

from objective reality, arise from efficient conscious. This position, and the associated sion of a recent academic statement on http://science.sciencemag.org/ computing, such as applying a Bayesian insistence on “qualia” and the “hard prob- the power and limits of AI (12): “Just like prior to noisy inputs (6) or taking efficient lem” of consciousness, are unproductive crash tests for transportation, the passing

on January 25, 2018

Published by AAAS

DA_0126Letters.indd 401 1/24/18 10:49 AM INSIGHTS | LETTERS

of ethical and safety tests, evaluating, for 6. Y. Weiss et al., Nat. Neurosci. 5, 598 (2002). cause concern: (i) lack of within-year climate instance, social impact or racial prejudice, 7. T. D. Ullman, E. Spelke, P. Battaglia, J. B. Tenenbaum, variables, (ii) low and variable estimates of Trends Cogn. Sci. 10.1016/j.tics.2017.05.012 (2017). could become a prerequisite to the release 8. M. Wang, D. Arteaga, B. J. He, Proc. Natl. Acad. Sci. U.S.A. covariance relationships across taxa, and (iii) of [artificial intelligence] software.” 110, E3350 (2013). a lack of mechanistic explanations for the pat- Stanislas Dehaene,1,2* Hakwan Lau,3,4 9. T. I. Panagiotaropoulos et al., Neuron 74, 924 (2012). terns observed; association is not causation. 10. B. Odegaard, R. T. Knight, H. Lau, J. Neurosci. Off. J. Soc. 5 Full text: dx.doi.org/10.1126/science.aan5028 Sid Kouider Neurosci. 37, 9593 (2017). 1Chair of Experimental Cognitive Psychology, 11. D. C. Dennett, in Consciousness in Modern Science, A. 2 Collège de France, 75005 Paris, France. Cognitive Marcel, E. Bisiach, Eds. (Oxford University Press, 1988), pp. Response to Comment on “Precipitation Neuroimaging Unit, Commissariat à l’Energie 42–77. drives global variation in natural selection” Atomique et aux Energies Alternatives (CEA), 12. Pontifical Academy of Sciences, “Final statement of Adam M. Siepielski, Michael B. Morrissey, INSERM, Université Paris-Sud, Université Paris- the workshop: Power and limits of artificial intel- Saclay, NeuroSpin Center, 91191 Gif/Yvette, France. ligence” (2016); www.pas.va/content/accademia/en/ Mathieu Buoro, Stephanie M. Carlson, 3 Department of Psychology and Brain Research events/2016/intelligence/statement.html. Christina M. Caruso, Sonya M. Clegg, Tim Institute, University of California, Los Angeles, Los Angeles, CA 90095, USA. 4Department of Coulson, Joseph DiBattista, Kiyoko M. 10.1126/science.aar8639 Psychology, University of Hong Kong, Hong Kong. Gotanda, Clinton D. Francis, Joe Hereford, 5 Brain and Consciousness Group (École Normale Joel G. Kingsolver, Kate E. Augustine, Supérieure, École des Hautes Études en Sciences Sociales, CNRS), Département d’Études Cognitives, TECHNICAL COMMENT ABSTRACTS Loeske E. B. Kruuk, Ryan A. Martin, Ben École Normale Supérieure–Paris Sciences et Lettres C. Sheldon, Nina Sletvold, Erik I. Svensson, Comment on “Precipitation drives global Research University, Paris, France. Michael J. Wade, Andrew D. C. MacColl *Corresponding author. variation in natural selection” Email: [email protected] The Comment by Myers-Smith and Myers

Isla H. Myers-Smith and Judith H. Myers Downloaded from focuses on three main points: (i) the lack REFERENCES Siepielski et al. (Reports, 3 March 2017, of a mechanistic explanation for climate- 1. S. Kouider, S. Dehaene, Philos. Trans. R. Soc. London Ser. B p. 959) claim that “precipitation drives selection relationships; (ii) the appropriate- Biol. Sci. 362, 857 (2007). global variation in natural selection.” This 2. S. Dehaene, Consciousness and the Brain: Deciphering ness of the climate data used in our analysis; How the Brain Codes Our Thoughts (Penguin Books, conclusion is based on a meta-analysis of and (iii) our focus on estimating climate- Reprint edition, 2014). the relationship between climate variables selection relationships across (rather than 3. P. C. Fletcher, C. D. Frith, Nat. Rev. Neurosci. 10, 48 (2009).

and natural selection measured in wild within) taxonomic groups. We address these http://science.sciencemag.org/ 4. Y. Ko, H. Lau, Philos. Trans. R. Soc. London Ser. B Biol. Sci. 367, 1401 (2012). populations of invertebrates, plants, and critiques in our response. 5. D. Dennett, Cognition 79, 221 (2001). vertebrates. Three aspects of this analysis Full text: dx.doi.org/10.1126/science.aan5760

on January 25, 2018

Published by AAAS

DA_0126Letters.indd 402 1/24/18 10:49 AM Conscious machines: Defining questions Olivia Carter, Jakob Hohwy, Jeroen van Boxtel, Victor Lamme, Ned Block, Christof Koch and Naotsugu Tsuchiya

Science 359 (6374), 400. DOI: 10.1126/science.aar4163 Downloaded from

ARTICLE TOOLS http://science.sciencemag.org/content/359/6374/400.1

RELATED http://science.sciencemag.org/content/sci/359/6374/400.3.full http://science.sciencemag.org/ CONTENT http://science.sciencemag.org/content/sci/358/6362/486.full

REFERENCES This article cites 4 articles, 2 of which you can access for free http://science.sciencemag.org/content/359/6374/400.1#BIBL

PERMISSIONS http://www.sciencemag.org/help/reprints-and-permissions

on January 25, 2018

Use of this article is subject to the Terms of Service

Science (print ISSN 0036-8075; online ISSN 1095-9203) is published by the American Association for the Advancement of Science, 1200 New York Avenue NW, Washington, DC 20005. 2017 © The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works. The title Science is a registered trademark of AAAS.