correspondence Mobilizing the private sector for responsible innovation in

To the Editor — Emerging political or economic manipulation, controversies about labor protection and raise important governance questions related direct-to-consumer (DTC) marketing employee surveillance14. Likewise, there is to, for example, dual use, brain data privacy, of devices that have variable, if any, a debate as to whether research into certain and manipulation of personal autonomy. effectiveness, dual use, do-it-yourself (DIY) types of BCI should be banned because of Although many public sector research neurotech and , and new forms dual-use applications (for example, covert initiatives have implemented measures to of inequality7–11. Although public-sector manipulation of personality), thus foregoing address these issues, similar systematic research has been quick to implement potential civil-use benefits (for example, measures in the private sector have yet to targeted programs to tackle these concerns— the restoration of sensorimotor functions emerge. This gap is critical, as neurotech for example, the ‘Ethics and Society’ strand after spinal cord injury)10,15. Challenges may innovation today is largely driven by a set of of the Human Brain Project12,13—the private even rise to the judicial and constitutional companies that are subject to growing public sector has thus far paid relatively scarce level: the landmark federal case US v. scrutiny1–5. Here we detail lessons, emerging systematic attention to them3. Semrau for the first time considered, though practices and open questions for responsible This gap between public and private ultimately dismissed, brain scans as a source innovation in the private sector that are the sector efforts to foster responsible innovation for lie detection16. The Chilean senate is result of three years of policy deliberations practices is critical insofar as, across OECD considering a constitutional amendment that began with a 2018 conference in countries, over 70% of all R&D is performed that, if approved, would be the first to legally Shanghai convened by the Organization for by the private sector. Moreover, the codify ‘neurorights’ to protect the mental Economic Co-operation and Development implications of many recent innovations have integrity and privacy of its citizens17. This (OECD) and led to the release of the become fully visible to society only when reflects wider debates about the need for “OECD Recommendation on Responsible scaled up by companies. This has placed new human rights in the age of rapidly Innovation in Neurotechnology” last year6. some of the most successful technology evolving and neurotech18. The principles therein cover opportunities firms increasingly into the crosshairs of Neurotech, like many other innovation and challenges for better innovation practices regulators and a public tech-lash. Facebook, domains, is also subject to a patchwork in company settings—including the use for instance, has been subject to a barrage of of national and regional regulations that of ethics advisory boards, company-level inquiries about the free speech and content create considerable uncertainty. National principles, and ethics-by-design moderation, data protection, or the effects of attempts to govern emerging technology approaches—with broad relevance beyond surveillance capitalism and echo chambers are frequently seen as ineffective or even neurotech to digital medicine and corporate on democracy. Clearview AI, a facial detrimental to innovative economies, R&D activities in today’s era of ‘tech-lash’. We recognition software company, promises prompting concerns that companies and argue that it is time for a radical shift in the greater public safety through scalable, technologies may simply move across conversation about governance of emerging app-based facial-recognition techniques, borders. Developing new international neurotech: effective governance must focus but has been criticized for enabling intrusive treaties, however, is notoriously difficult, on the private sector as a central actor early and potentially authoritarian uses. Digital and intergovernmental organizations on—before trajectories are locked in and platforms that have begun to transform often rely on soft law, such as the OECD scaling takes off—and requires a new set of entire service sectors, such as Uber or recommendation6,19. Within single policy perspectives and collaborative tools to Airbnb, have also raised concerns about jurisdictions, too, there is ample regulatory do so. These tools must complement existing new inequalities pertaining to undermining complexity, as neurotech straddles sectors, efforts in public-sector research ethics, labor laws or driving real estate speculation, applications and regulatory domains. post hoc product regulation and corporate respectively. From a policy perspective, BCIs, digital phenotyping apps and social responsibility. They must also these big tech examples beg the question psychopharmacology will be subject to reflect the growing recognition that we of whether ‘responsible innovation’ efforts different regulatory regimes straddling cannot rely on industry self-regulation focused on public-sector research simply health, safety, trade or drug regulation, alone to steer innovation activity in socially miss the mark. The same holds true for each of which are governed by different desirable directions. emerging neurotechnologies. governmental agencies and jurisdiction. Recognizing these challenges, Missing the mark with ethics and Traditional technology governance is policymakers have increasingly tried to responsible innovation initiatives? increasingly insufficient engage upstream governance approaches— Emerging neurotechnologies, such as For emerging neurotech, traditional means that is, early interventions during the brain–computer interfaces (BCIs) or digital of governance—including institutionalized research process—to complement traditional phenotyping for monitoring, research ethics, post hoc regulation and post hoc regulation. In public-sector hold considerable promise for health and market mechanisms—are ill-equipped to research, approaches such as anticipatory well-being, but also raise important ethical capture the ways in which these technologies governance20 and responsible research and social, and governance questions (Box 1). could reshape our societies, especially in innovation (RRI)21 have gained increasing These questions include concerns about brain terms of long-term consequences. The credibility. Instruments of Ethical, Legal data privacy, runaway , potential uses of non-invasive BCIs in the and Social Implications (ELSI) piloted individual autonomy, vulnerability to workplace, for example, are raising new by the Human Genome Project22, such as

Nature Biotechnology | VOL 39 | June 2021 | 661–670 | www.nature.com/naturebiotechnology 661 correspondence

Trade Commission in 2016 for deceptive Box 1 | Responsibility issues in neurotech companies claims about enhanced concentration and decreased cognitive impairment in patients • Brain data privacy. Companies attention in the context of digital work with Alzheimer disease using its products. In developing products, services and environments. the absence of good strategies, governments treatments relying on neurodata, • Direct-to-consumer marketing. are increasingly embracing experimental whether from behavioral apps or BCIs, Marketing as a DTC technology for strategies to tackle governance challenges face mounting questions regarding well-being and entertainment is the or test applications. The US Food and Drug data ownership, security, privacy and pathway of choice for many startups Administration is testing experimental consent. Tis debate echoes similar trying to avoid the high regulatory precertification programs partly to get a concerns raised in genomics or digital scrutiny associated with medical grip on, among other things, emerging platforms, but involves potentially devices. Tese boundaries can be mobile applications for mental health that more sensitive data around individual blurred easily, however, and many are increasingly marketed DTC. The city of motivations that could be used in court companies would prefer clear guide- Reno has embarked on a local experiment to or for unintended purposes. lines over amorphous expectations. offer app-based mental health services for its • Human enhancement. Many compa- • Dual use. Many products, especially residents through the company Talkspace to nies are developing products that walk those concerned with neuroenhance- help alleviate the devastating mental health a fne line between rehabilitation (for ment or suppression of cognition, strad- effects of the COVID-19 pandemic. This example, for war veterans’ injuries) and dle medical, consumer and military took place despite recent controversy around human enhancement (for example, applications. For companies, this situa- privacy issues for apps, reflecting a common conditioning). Echoing long-standing tion raises questions about export mar- ‘hands-off’ approach by local jurisdictions debates about psychopharmacology, kets and potential security concerns. towards responsible innovation. this tension raises questions about • Do-it-yourself neurotech and neu- diferent national standards for human rohacking. Tere is a growing DIY What companies should do enhancement, especially when com- neurotech scene similar to that in other The current lack of systematic responsibility bined with BCI technology and brain biotechnology domains, including gene frameworks does not mean that embedded stimulation. Similar concerns have editing. While sometimes welcomed upstream governance options for emerging been raised in the context of novel gene as a sign for a vivid entrepreneurial technologies cannot be implemented in editing techniques. ecosystem, neurohackers may fy the private sector. Our three-year dialogue • Autonomy and vulnerability to under the regulatory radar and develop processes revealed that a range of neurotech manipulation. Many emerging potential products or applications that companies are actively seeking guidance app-based neurotechnologies exploit might otherwise attract high scrutiny. and developing their own toolkits to bridge behavioral patterns to predict or • New forms of inequality. All of structural constraints and the apparent need respond to cognitive states. Te data the aforementioned challenges may for greater public oversight. What is more, collected by these apps may poten- introduce new forms of inequality. many leading neurotech companies have a tially be used to trigger mental health Unequal access to medical treatments, strong interest in publicly demonstrating interventions, but can make indi- populations vulnerable to exploitation responsibility and integrity, recognizing that viduals also vulnerable to surveillance, (for example, children or those unable the entire nascent sector can be harmed policing, and economic or political to understand terms of service), a ‘race by single irresponsible actors in the field. manipulation. Examples include to the bottom’ for enhancement, or Below, we list several emerging practices targeted advertisements or behavioral uneven regulation across countries are and principles that can help ensure better nudges. Some examples. Many companies are gauging governance of neurotechnology innovation have raised concerns about workplace the global landscapes for diferent busi- in corporate settings. surveillance—for example, measuring ness models. Enable responsibility review and diverse perspectives as part of the R&D focus group research, citizen juries, ethical new forms of inequality, vulnerability process. One example of a company that review through institutional review boards, or risk—are hard to capture as part of appointed an ELSI Advisory Board early or stage-gate processes, have been taken up company metrics, incentive structures and in its history is Mindstrong, a company by the Human Brain Project and BRAIN shareholder value logics. For another, the that develops apps to predict mental illness Initiative, among others23. field is largely driven by startup dynamics, relapses through patient smartphone Although ELSI and RRI frameworks which does not afford extended time for interactions. This board brought together have successfully penetrated wide parts deliberation or dedicated organizational engineers, ethicists, social scientists and of public research on neurotech, similar resources. The entrepreneurial mindset people living with mental health issues systematic frameworks in the private sector to move fast, break things, scale up, and to actively shape development of the are lacking. Companies tend to sit in a blind worry about consequences later24 is at odds technology. It was instrumental, for example, spot between early-stage research ethics and with traditional governance mechanisms in the decision to switch from collecting post hoc regulatory responses that primarily such as ethics board reviews and public text or global positioning system (GPS) focus on safety and efficacy, monopoly consultations during product development. data, which users considered intrusive of power or liability. Approaches found in RRI This need for speed and scale can lead their privacy, to content-free and less readily or ELSI programs are neither mandatory to unintended consequences as well as identifiable signals from the smartphone, nor easily applicable in corporate settings. overpromising. For example, Lumosity, a such as keyboard interactions patterns25. For one, the broader social consequences company providing a brain-training app, This diversity-oriented advisory board of technological change—including was fined $2 million by the US Federal strategy is broadly consistent with the

662 Nature Biotechnology | VOL 39 | June 2021 | 661–670 | www.nature.com/naturebiotechnology correspondence recent surge in corporate hires from the Mobilize tech transfer as a critical juncture generation of innovators, a growing number humanities and social sciences to inject for social impact. Many universities of universities are offering resources for critical and socially inclusive perspectives are adjusting their tech transfer rules to students, entrepreneurs and startups into innovation processes. Getting these better reflect social priorities. They are to consider responsibility and risks to structures right and sustaining them in emphasizing, for example, inclusiveness sustainability as part of business model a corporate environment is not trivial, in benefit-sharing and requirements development, including Arizona State however. Google famously had to dissolve to institutionalize certain values and University’s Risk Innovation Nexus and the its AI Ethics Council just one week after accountability structures. Historically, the Technical University of Munich’s Master of its much-anticipated launch, following incentive structures for technology transfer Arts program “Responsibility in Science, considerable internal and external backlash offices have tended to maximize revenue, the Engineering, and Technology,” in which one about its composition. number of startups, or scope of corporate of us (S.P.) is involved. The incorporation sponsorship, with little attention to ethical of responsible innovation into engineering Develop robust responsibility principles and social deliberation elements. With an education and nascent business models as part of a startup’s mission. One of us impressive list of signatories, the “Nine can create an added value—for example, by (D.B.) has developed a code of responsibility Points to Consider” code of good practice gauging long-term societal implications or for his neurotech startup Aifred, which in university technology transfer provides engaging early with potential future concerns applies deep-learning algorithms to enhance a model for how to leverage technology or regulations-in-the-making. In the long individualized psychiatric treatment. In this transfer for more responsible innovation run, the disconnect between CSR and ‘meticulous transparency’ framework, all practices, including in neurotech25. corporate R&D raises serious questions about machine-learning projects must be reviewed the adequacy of traditional CSR approaches by the clinical and machine-learning team Pressure investors to select for responsible for an era in which business models tend to with respect to their intended outcome, the technology development approaches. focus on innovation and disruption. target population, the representativeness of Shareholders are increasingly stepping up the available data, interpretability metrics, to inject responsibility considerations in Finding the right balance and monitoring for adverse effects of the company strategy. In 2018, two major Wall The past 10 years have brought into sharp model26. The framework helped resolve Street investors pressured Apple to take relief not only seemingly unregulated spaces concrete design dilemmas like the use of steps to fight addictions to iPhone use in in which innovative companies can rapidly binary predictive algorithm outputs, such children, which led to the development of an grow from small startups to powerful as ‘being’ or ‘not being’ at risk of suicide— app called Screen Time. In neurotech, some global forces, but also the difficulties in which the company decided should best be companies are actively seeking out investors exerting regulatory scrutiny in real-time designed as a warning system available only who match their values. Yet, as the Shanghai through traditional governance approaches. to clinicians and only with probabilistic, OECD conference revealed, the number The burgeoning field of neurotech is no rather than binary, outputs. This, in turn, of venture capital investors foregrounding exception. Yet, cognizant of the fallout affected the way the machine-learning responsible innovation concerns is limited, recently observed in the controversies analyses were conceptualized—an example despite considerable interest among startups surrounding ‘big tech’, many neurotech of responsibility-driven design. The focus on to work with specialized investors who companies are actively looking for guidance responsibility as part of concrete, embedded know and acknowledge the ethical and on how to increase the social robustness and code differs from the rather high-level, social challenges of their technologies. sustainability of their emerging products non-committal ethics guidelines for artificial This opens up an opportunity for a new and services in this field. intelligence and other technologies released subset of investment instruments or venture There is, of course, reason to be skeptical by the dozens by many corporate giants. capital niches dedicated to responsible that companies alone will ensure socially innovation practices, similar to the recent responsible technology trajectories. Industry Embrace collectively legitimated surge in sustainable investment and ‘green self-regulation has regularly failed to deliver ethics-by-design approaches. bond’ portfolios that target environmental the promised results and has instead stoked Standard-setting bodies like the Institute of or climate-related projects28. Such critiques of tokenization and greenwashing. Electrical and Electronics Engineers (IEEE) developments could be further supported A similar effect can arguably be observed are increasingly targeting the engineering by new standards or certifications on in the current wave of ‘ethics washing’ (the phase of product development to address responsible investment in tech startups. practice of implementing superficial ethics social values and standardize certain critical mechanisms or principles in response features from the beginning, including Rethink corporate social responsibility to public pressure while purposefully in the fields of neurotech and artificial approaches. Traditional corporate side-stepping more fundamental issues, as intelligence27. Upstream ethics-by-design social responsibility (CSR) typically most recently observed in the controversies approaches aim to hardwire values into addresses the protection of workers, local surrounding Facebook’s Oversight Board. downstream developments. Given their communities and the environment through Many large tech companies have accepted the consequences, however, these choices need self-governance tools. However, CSR has cost of asking for forgiveness—such as paying to be opened up to collective deliberation largely ignored innovation as a key arena out fines or legal settlements—as the cost of and be subject to some form of political for social impact and responsible business doing business. Thus, the promise of more legitimation. Through bodies like the IEEE, conduct—as evident in today’s controversies responsible innovation by way of industry public- and private-sector actors can work surrounding ‘big tech’. In most neurotech self-governance can only supplement, rather together to collectively define product companies, CSR approaches do not help than supplant, public oversight. standards and codify responsible design solve the aforementioned ethical, social and However, there is ample evidence that choices that embody shared commitments governance dilemmas. Likewise, engineering government regulation alone will not around values such as privacy and ethics frameworks tend to remain outside suffice, as traditional policy instruments transparency. the purview of CSR29. Targeting the next are increasingly at a disadvantage in today’s

Nature Biotechnology | VOL 39 | June 2021 | 661–670 | www.nature.com/naturebiotechnology 663 correspondence

innovation landscape. This is why dedicated University of British Columbia, Vancouver, British 23. Frahm, N. M. International governance of neuroscience and forums and spaces that can mobilize alliances Columbia, Canada. 6Sandra Day O’Connor College neurotechnology: whom to trust with the assessment of future pathways? Te Neuroethics Blog http://www.theneuroethicsblog. of companies, policymakers, academics of Law, Global Institute for Sustainability and com/2017/11/international-governance-of.html (2017). and citizens are needed to raise the bar for Innovation, Arizona State University, Phoenix, 24. Pfotenhauer, S.M., Laurent, B., Papageorgiou, K. & Stilgoe, J. responsible innovation and co-develop new AZ, USA. Soc. Stud. Sci. (in the press). ✉ 25. AUTM. In the Public Interest: Nine Points to Consider in Licensing mechanisms for self-governance on the e-mail: [email protected] University Technology https://www.autm.net/AUTMMain/media/ business side alongside new government Advocacy/Documents/Points_to_Consider.pdf (2007). regulation, including the ones mentioned Published online: 7 June 2021 26. Benrimoh, D., Israel, S., Perlman, K., Fratila, R. & Krause, M. in https://doi.org/10.1038/s41587-021-00947-y Recent Trends and Future Technology in Applied Intelligence above. International organizations such as (eds Mouhoub, M. et al.) 869–880 (Springer International, 2018); the OECD or IEEE are uniquely positioned https://doi.org/10.1007/978-3-319-92058-0_83 to speak to differences in national regulations References 27. IEEE. Ethically Aligned Design, Version 1, Translations and Reports 1. Eaton, M. L. & Illes, J. Nat. Biotechnol. 25, 393–397 (2007). and are already playing key roles in fostering https://standards.ieee.org/industry-connections/ec/ead-v1.html 2. Jarchum, I. Nat. Biotechnol. 37, 993–996 (2019). (2016). 3,6,27 the necessary dialogs . Universities, 3. Garden, H., Winickof, D., Frahm, N. & Pfotenhauer, S. M. 28. Kurtz, E. in Te Oxford Handbook of Corporate Social too, can mobilize their educational and Responsible innovation in neurotechnology enterprises. OECD Responsibility (eds Crane, A. et al.) 249–280 (Oxford Univ. entrepreneurial ecosystems to heighten Science, Technology and Industry Working Papers, No. 2019/05 Press, 2008); https://doi.org/10.1093/oxfordhb/9780199211593. (OECD, 2019); https://doi.org/10.1787/9685e4fd-en 003.0002 sensitivity to responsibility concerns and 4. Wexler, A. A skeptic’s take on and other consumer 29. Smith, N. M., Zhu, Q., Smith, J. M. & Mitcham, C. Sci. Eng. Ethics foster policy dialog when companies are in neurotech. Stat (7 April 2021). 27, 28 (2021). the startup stage. What is more, experimental 5. Wexler, A. & Reiner, P. B. Science 363, 234–235 (2019). 30. Engels, F., Wentland, A. & Pfotenhauer, S. M. Res. Policy 48, 6. OECD. OECD Recommendation on Responsible Innovation 103826 (2019). ‘living lab’ and ‘sandbox’ approaches could in Neurotechnology https://www.oecd.org/science/ be used to co-develop new regulations and recommendation-on-responsible-innovation-in- Acknowledgements neurotechnology.htm (2019). foster public debate about novel technologies, 7. Wexler, A. Front. Hum. Neurosci. 11, 224 (2017). S.P. acknowledges support by the European Union’s not just to create pro-business innovation 8. Schwartz, A. Science 350, 11 (2015). Horizon 2020 research and innovation program under environments through lower regulatory 9. Ienca, M., Haselager, P. & Emanuel, E. J. Nat. Biotechnol. 36, grant agreement 788359 (‘SCALINGS: Scaling up 805–810 (2018). co-creation? Avenues and limits for including society in standards, as is currently the case in many science and innovation’). J.I. is supported by the Canada 30 10. Ienca, M., Jotterand, F. & Elger, B. S. Neuron 97, 269–274 (2018). such settings . Neurotech companies, with 11. Nufeld Council on Bioethics. Novel Neurotechnologies: Research Chairs Program as the Canada Research Chair obvious social and ethical challenges on the Intervening in the Brain. https://www.nufeldbioethics.org/ in Neuroethics and co-leads the Canadian Brain Research horizon, have a chance to set an example for publications/neurotechnology (2013). Strategy (CIHR Grant #171583;03027 IC-127354). 12. Salles, A. et al. Neuron 101, 380–384 (2019). the entire tech industry. ❐ 13. Global Neuroethics Summit Delegates. et al. Neuron 100, Competing interests 19–36 (2018). Sebastian M. Pfotenhauer 1 ✉ , 14. De Stefano, V. Neuro-surveillance and the right to be human S.P. serves as academic program director for the master’s program ‘Responsibility in Science, Engineering, and 1 2 at work. On Labor https://onlabor.org/neuro-surveillance-and- Nina Frahm , David Winickof , the-right-to-be-humans-at-work/ (15 February 2020). Technology’ (REST) at Technical University of Munich. 3,4 5 David Benrimoh , Judy Illes and 15. Clausen, J. et al. Science 356, 1338–1339 (2017). D.W. is a senior policy analyst at the Organisation for Gary Marchant6 16. Jones, O. D., Marois, R., Farah, M. J. & Greely, H. T. J. Neurosci. Economic Co-operation and Development (OECD), 33, 17624–17630 (2013). 1Munich Center for Technology in Society (MCTS) where he heads the Working Party for Biotechnology, 17. Asher-Schapiro, A. Out of my mind: advances in brain tech spur Nanotechnology and Converging Technologies (BNCT) and TUM School of Management, Technical calls for ‘neuro-rights’. Reuters (29 March 2021). responsible for the Recommendation on Responsible 2 University of Munich, Munich, Germany. Working 18. Ienca, M. & Andorno, R. Life Sci. Soc. Policy 13, 5 (2017). Innovation in Neurotechnology. D.B. is a founder, 19. Marchant, G. & Allenby, B. Bull. At. Sci. 73, 108–114 (2017). Party for Bio-, Nano- and Converging Technologies, shareholder and employee of Aifred Health, a digital 20. Barben, D., Fisher, E., Selin, C. & Guston, D. in Te Handbook of mental health company. Organisation For Economic Co-operation and Science and Technology Studies (eds Hackett, E. J. et al.) 979–1000 Development (OECD), Paris, France. 3Department (MIT Press, 2008). Additional information of Psychiatry, McGill University, Québec, Montréal, 21. Stilgoe, J., Owen, R. & Macnaghten, P. Res. Policy 42, 1568–1580 (2013). Peer review information Nature Biotechnology thanks the 4 Canada. Aifred Health, Montreal, Quebec, Canada. 22. NIH. ELSI planning and evaluation history. https://www.genome. anonymous reviewers for their contribution to the peer 5Neuroethics Canada, Department of Medicine, gov/10001754/elsi-planning-and-evaluation-history (2012). review of this work.

The need for global access to biomedical innovations during pandemics

To the Editor — The COVID-19 pandemic early competition among high-income situation3,4. At best, most populations has fundamentally reshaped the way and upper-middle-income countries over in LMICs will face a protracted delay in which research and development is initially scarce supplies has prevented low- of months to years in receiving these conducted and, in the case of vaccines and and middle-income countries (LMICs) innovations at scale; at worst, some therapeutics, compressed a decade-long from accessing these innovations2. For populations may never receive them5,6. product development process to less than vaccines, this problem will likely continue To effectively respond to pathogens one year1. However, the new technologies if annual vaccinations are required and/or with pandemic potential, the product produced by this accelerated innovation— SARS-CoV-2 variants warrant modification development ecosystem needs to both such as mRNA vaccines, viral vector and subsequent boosters—both requiring develop biomedical products in an even vaccines, monoclonal antibodies and additional supplies. The lack of delivery shorter timeframe and distribute these state-of-the-art diagnostics—have not been systems to distribute these innovations products globally in an equally rapid equitably distributed worldwide. Intense in many LMICs further compounds this manner. Building on proposed strategies

664 Nature Biotechnology | VOL 39 | June 2021 | 661–670 | www.nature.com/naturebiotechnology