Ethical Barriers to Artificial Intelligence in the National Health Service

Perspectives

Ethical barriers to artificial intelligence in the national health service, United Kingdom of Great Britain and Northern Ireland Claire Louise Thompsona & Heather May Morgana

Artificial intelligence, the ability of sys- Patient data in training free-for-all for use in training algo- tems to replicate human behaviour in algorithms rithms, is necessary. an intelligent manner, shows promise in the United Kingdom of Great Britain Daily, the NHS collects and records vast Transparency and and Northern Ireland’s National Health quantities of data, providing a valuable accountability Service (NHS), which provides free- opportunity for methods requiring at-the-point-of-service health care via training through large data sets, such Nearly all artificial intelligence software a national insurance scheme (Fig. 1). as machine learning.3 Patient data, how- will fail to perform its intended pur- Recent advancements in artificial in- ever, could be potentially misused, and pose at some point during its lifetime.7 telligence have created sophisticated must be subject to relevant ethical and Although in some sectors, artificial software programmes that could revo- legal considerations by both developers intelligence failures can be trivial, fail- lutionize the NHS. Breakthroughs in and within the NHS. ures in the health-care sector can have machine learning, more notably deep A recent study found that 51% catastrophic consequences. Therefore, learning (Box 1), have led to algorithms (1020/2000) of people surveyed in the the ability to hold the responsible party capable of performing diagnostic skills United Kingdom were concerned about accountable is vital. However, the as- equivalent to those of doctors, automat- their data privacy as the use of artificial signment of accountability in artificial ing administrative tasks and assisting in intelligence increases; this finding was intelligence, and specifically machine complex treatment management. particularly relevant for those with less learning, can be challenging, primarily As a result of the vast linkable data that knowledge about artificial intelligence due to the lack of transparency. Some the NHS holds on all citizens through- capabilities.4 As part of the government’s studies suggest that humans may no out their lives, the service could have a code of conduct, complying with appro- longer be in control of what decision leading role in taking forward artificial priate legal and ethical frameworks is is taken and may not even know or un- intelligence development for health required to safeguard patients. However, derstand why a wrong decision has been care;1 however, its use remains limited, the growth of machine learning and its taken, because transparency is lost.7 with little overarching policy guiding its need for real-life data could present an Currently, artificial intelligence development and application. In 2018, ethical dilemma where patient data are within the NHS is a tool to support the government of the United Kingdom being used as an exploitable resource staff rather than a decision-maker, published a code of conduct outlining for purposes other than those for which meaning medical professionals are held expectations for artificial intelligence the data were originally collected. This accountable for decisions regardless of development in the NHS, covering as- situation has already occurred. The whether their decisions were influenced pects such as the appropriate handling of Royal Free NHS Trust failed to obtain by artificial intelligence technology or data, the need for algorithmic transpar- appropriate consent for the use of data not. The fact that NHS professionals can ency and accountability.2 The code states from 1.6 million patients in the develop- be held accountable for decisions influ- that, in combination with the confor- ment of one of its artificial intelligence enced by potentially inaccurate artificial mité européenne (CE) mark certification, applications.5 Repeated episodes such as intelligence, which cannot be proven in health research and relevant regulatory this one could decrease public trust in some situations due to lack of transpar- approvals, it should provide an overall the NHS handling of patient data and ency, may deter them from embracing policy and structure for the creation of might make patients refuse to share their the technology. Although the code of safe and effective artificial intelligence. information. Ensuring that patients have conduct highlights the importance of The code, however, is only in its initial explicitly consented to the use of their transparency, at present, ensuring this consultation stage. This paper discusses personal data in artificial intelligence transparency may not be entirely pos- the issues highlighted within the code development is therefore crucial. The sible and, if artificial intelligence is to of conduct and the ethical challenges introduction of a national data opt-out take on a greater role in supporting associated with addressing them to suc- programme for patient information in clinicians in decision-making, the issue cessfully integrate artificial intelligence 20186 has made it easier for patients to must be addressed. within the NHS. control the use of their data. Addition- The development of transparent ally, guaranteeing that patient informa- machine learning techniques would tion continues to be viewed as a valued benefit the accountability dilemma in asset subject to appropriate ethical and artificial intelligence and address the legal considerations, as opposed to a previously mentioned regulatory issues, a School of Medicine, Medical Sciences and Nutrition, University of Aberdeen, Polwarth Building, Foresterhill, Aberdeen, AB25 2ZD, Scotland. Correspondence to Heather Morgan (email: [email protected]). (Submitted: 14 May 2019 – Revised version received: 23 December 2019 – Accepted: 3 January 2020 – Published online: 28 February 2020 )

Bull World Health Organ 2020;98:293–295 | doi: http://dx.doi.org/10.2471/BLT.19.237230 293 Perspectives Ethical consideration and artificial intelligence in the national health service Claire Louise Thompson & Heather Morgan

Fig. 1. The structure of the National Health Service in the United Kingdom of Great and therefore is an important area for Britain and Northern Ireland study. Tackling transparency in artificial intelligence, specifically machine learning, can be challenging; however, England research suggests it could be possible. Scientists have recently identified meth- Department for Health England (55.98 million people, 130 279 km2): (UK government department NHS (independent body that oversees healthcare) ods for developing transparent deep 8 responsible for funding and creating learning neural networks. Furthermore, healthcare policies) developers tackled transparency issues, Clinical Commissioning Groups while improving artificial intelligence (responsible for commissioning local healthcare) run by general software for analysing ophthalmologic practitioners, nurses and consultants; allocate ~60% of budget images, by displaying selected information regarding how the software arrived at its recommendation.9 Despite these NHS Foundation Trust promising examples, technological (provide care as directed by Clinical Commissioning Groups) include hospitals, ambulances, mental health, social care and primary care solutions to making machine learning transparent are still in their infancy and will require developers with expertise in Scotland, Wales and Northern Ireland this fast-moving field.

Scotland (5.44 million people, 80 077 km2): 14 Boards (plan, commission and deliver all health Public trust and social care services) One of the key messages highlighted UK Parliament Wales (3.14 million people, 20 735 km2): throughout the code of conduct is that (allocates block funding to each 7 Local Health Boards (responsible for delivering all healthcare services gaining public trust is a high priority national government to decide how to within a geographical area) and 3 national Trusts (all-Wales services, and crucial to artificial intelligence’s spend it within their NHS) including ambulances, cancer care and Public Health Wales) successful deployment within the NHS. Current reports, however, suggest that Northern Ireland (1.88 million people, 14 130 km2): Department of Health, Social Services and Public Safety the NHS is in a less than desirable posi- (authority for health and social care services); Health and Social Care tion, with a recent poll finding that only Board (commission services); 5 Health and Social Care Trusts 20% (400/2000) of respondents support (provide services); plus an Ambulance Trust the use of artificial intelligence in health care,4 demonstrating the scale of the NHS: National Health Service; UK: United Kingdom Sources: National Health Service of England, Scotland, Wales and Northern Ireland websites: https://www. problem. Although there are numerous england.nhs.uk/; https://www.scot.nhs.uk/; https://www.wales.nhs.uk/; http://online.hscni.net/. Most artificial intelligence success stories recent national census data (2011) for each of the constituent of the United Kingdom: https://www.ons. from recent years, these have arguably gov.uk/. been outshined by catastrophic failures, which have inevitably dented the public’s already limited trust in the use of artifi- Box 1. Definition of machine learning and deep learning cial intelligence. High-profile incidents, in combination with unfamiliarity and a Machine learning lack of understanding present a signifi- A subset of artificial intelligence that allows programmes to learn without being explicitly cant problem. programmed. These programmes learn from data sets, identify patterns within them and use Public trust in the use of artificial the information to make predictions about data they have not been previously exposed to. intelligence is essential for its growth, Deep learning and relevant parties, including the A relatively new subset of machine learning, which uses algorithms that replicate the human brain government, must adopt measures to (neural networks) in structure and function. In machine learning, the more data the algorithm gain this trust. Efforts need to be made can use, the greater its performance. In older styles of machine learning, algorithms eventually reach a plateau in performance, regardless of how much additional data is used to train the not only to ensure that patients are suf- algorithm. Deep learning, however, can use greater quantities of data leading to higher levels ficiently empowered with education on of accuracy in algorithms that were not previously achievable, this is one of the main reasons current technology, but also reassured deep learning has revolutionized the artificial intelligence field today. Despite the advantages of that the technology is safe and developed increased performance, one of the major drawbacks of deep learning is the lack of transparency according to relevant technical and ethi- in the algorithm’s operation (known as a ‘black box’ system). In deep learning, viewing the input cal standards. and output of algorithms is possible; however, it is difficult, even for developers, to determine how the algorithm produced its output. This difficulty proves significantly problematic in disciplines such as health care, where justification for decisions reached is crucial to ensure Outlook maximum safety for patients. Ultimately, the use of artificial intelligence, especially machine learning, within the NHS has the potential to significantly improve patient care if measures are put in place to address

294 Bull World Health Organ 2020;98:293–295| doi: http://dx.doi.org/10.2471/BLT.19.237230 Perspectives Claire Louise Thompson & Heather Morgan Ethical consideration and artificial intelligence in the national health service current barriers. Due to the significant ing and capacity building in the techni- of artificial intelligence within the NHS volume of patient data required in cal aspects is needed to create expertise requires political attention to address artificial intelligence development, de- in how to investigate and make sense of current policy limitations and gaps velopers and providers must adhere to system failure. Such focus within the dis- within NHS trusts, and may also benefit data regulation and control to avoid in- cipline is improving and those working from nation-wide unity on how to ap- appropriate data exploitation. Data use on this area of computing sciences will proach artificial intelligence use across must be subject to relevant ethical and need to develop robust and transparent the NHS. Academic research, including legal considerations, which will require practices, and identify means of explain- embedded stakeholder engagement, will research attention from social scientists, ing technical processes to non-experts. be required to inform the development philosophers and bioethicists working in An updated code of conduct is needed of such policies, and advocacy will be applied health sciences and lawyers col- to guide practices and/or a benchmark needed to promote these policies, us- laborating with health-care innovation gold standard codebreaker process. An ing a range of science communication teams, and providers to secure appropri- example of such work is the Realizing approaches for multiple public audi- ate permissions and safeguards. A more Accountable Intelligent Systems project, ences.10 Although it may be some years joined-up, cross-disciplinary working an academic collaboration to develop before the NHS is fully able to use the model between academics and lawyers auditing systems for artificial intelli- potential of artificial intelligence, it is alongside NHS partners (technical and gence. Finally, with low levels of public time to prepare for its growth. ■ clinical) is needed at the earliest stage trust in artificial intelligence, educating of every project to inform implementa- the public, ensuring that artificial intel- Competing interests: HMM is the Associ- tion. Addressing data use could dimin- ligence policy is accessible and provid- ate Director of the Aberdeen Centre for ish transparency and accountability ing reassurance that software has been Health Data Science, Aberdeen, Scotland. concerns by ensuring the logic behind developed according to regulatory and algorithms is available and clear. Train- safety standards, are needed. The growth

References 1. AI sector deal. London: Government of the United Kingdom; 2018. Available 6. National data opt-out programme. London: NHS Digital; 2018. Available from: https://www.gov.uk/government/publications/artificial-intelligence- from: https://digital.nhs.uk/services/national-data-opt-out-programme sector-deal/ai-sector-deal [cited 2020 Feb 27]. [cited 2018 Nov 29]. 2. Initial code of conduct for data-driven health and care technology. London: 7. Algorithms in decision-making inquiry. London: House of Commons Government of the United Kingdom; 2018. Available from: https://www. Science and Technology Committee; 2018. Available from: https:// gov.uk/government/publications/code-of-conduct-for-data-driven-health- publications.parliament.uk/pa/cm201719/cmselect/cmsctech/351/351.pdf and-care-technology/initial-code-of-conduct-for-data-driven-health-and- [cited 2020 Feb 6]. care-technology [cited 2018 Nov 2018]. 8. Mascharka D, Tran P, Soklaski R, Majumdar A. Transparency by design: 3. UK can be 'world leader in AI' if NHS data can be tapped. London: Digital closing the gap between performance and interpretability in visual Health; 2018. Available from: https://www.digitalhealth.net/2018/04/uk- reasoning [preprint]. Ithaca: Cornell University; 2018. Available from: https:// world-leader-ai-nhs-data/ [cited 2018 Nov 29]. arxiv.org/abs/1803.05268 [cited 2018 Nov 29]. 4. How the UK can win the AI race. Amstelveen: KPMG; 2018. Available from: 9. Moorfields Eye Hospital NHS Foundation Trust - DeepMind. London: https://assets.kpmg.com/content/dam/kpmg/uk/pdf/2018/09/how-the- DeepMind; 2018. Available from: https://www.moorfields.nhs.uk/landing- uk-can-win-the-artificial-intelligence-ai-race.pdf [cited 2020 Feb 6]. page/deepmind-health-research-partnership [cited 2018 Nov 29]. 5. Hern A. Royal Free breached UK data law in 1.6m patient deal with 10. Balaram B, Greenham T, Leonard J. Artificial Intelligence: real public Google's DeepMind. The Guardian. 2017 Jul 3. Available from: https://www. engagement. London: Royal Society for the encouragement of Arts, theguardian.com/technology/2017/jul/03/google-deepmind-16m-patient- Manufactures and Commerce; 2018. Available from: https://www.thersa. royal-free-deal-data-protection-act [cited 2018 Nov 29]. org/discover/publications-and-articles/reports/artificial-intelligence-real- public-engagement [cited 2020 Feb 6].

Bull World Health Organ 2020;98:293–295| doi: http://dx.doi.org/10.2471/BLT.19.237230 295