Implementing Voice Communication Technology in Patient Applications

UPTEC IT 20030 Examensarbete 30 hp Augusti 2020

Implementing voice communication technology in patient applications

Victor Munthe

Institutionen för informationsteknologi Department of Information Technology

Abstract Implementing voice communication technology in patient applications Victor Munthe

Teknisk- naturvetenskaplig fakultet UTH-enheten AstraZeneca as a biopharmaceutical company performs clinical trials on human volunteers every day. It is crucial that these participants complete the trial to Besöksadress: maximise the medical learning from the trial but also to decrease the cost for Ångströmlaboratoriet Lägerhyddsvägen 1 AstraZeneca. Dropout’s from clinical trials can not be eliminated but as a part of the Hus 4, Plan 0 work that AstraZeneca is doing to reduce dropouts, this thesis investigates the possibility to implement voice control into patient applications. The goal is to increase Postadress: the usability of the systems used in clinical trials and in return reduce dropouts and Box 536 751 21 Uppsala increase medical knowledge.

Telefon: This thesis results in a discussion that can be used for future work on this topic. 018 – 471 30 03 Opportunities that a voice-controlled system brings, together with its limitations are

Telefax: presented and results in a discussion regarding “data ownership”. One of the main 018 – 471 30 00 conclusions is that a voice-controlled system would bring a lot of positive features but that either the technology or the authorities are ready for such a system today. A Hemsida: suggested way forward can be to start trying clinical trials, where free-text responses http://www.teknat.uu.se/student are used for some questions, instead of fixed responses. This can help show the authorities the benefits of such a system and motivate them to open up for more technology.

Handledare: Mats Pettersson Ämnesgranskare: Bengt Sandblad Examinator: Lars-Åke Norde’n UPTEC IT 20030 Tryckt av: Reprocentralen ITC

Popularvetenskaplig¨ sammanfattning

AstraZeneca ar¨ ett lakemedelsbolag¨ som utfor¨ kliniska studier pa˚ frivilliga deltagare varje dag. Det ar¨ avgorande¨ att deltagarna avslutar studien for¨ att maximera utfallet av medicinsk kunskap men aven¨ ur ett kostnadsperspektiv. Den har¨ rapporten undersoker¨ mojligheten¨ att implementera roststyrning¨ i kliniska studier for¨ patienter som en del i arbetet AstraZeneca gor¨ for¨ att minska avhoppen. Malet˚ ar¨ att oka¨ anvandbarheten¨ av systemen som anvands¨ i kliniska studier idag som i sin tur minskar antalet avhopp och bidrar mer till medicinsk kunskap. Rapporten resulterar i en diskussion som kan anvandas¨ i ett fortsatt arbete inom amnet.¨ Mojligheterna¨ som ett roststyrt¨ system medfor¨ tillsammans med dess begransningar¨ pre- senteras och slutar i en diskussion kring agandeskap¨ av data. En tydlig slutsats som arbetet resulterar i ar¨ att implementationen av ett roststyrt¨ system medfor¨ manga˚ positiva aspekter men att myndigheterna och teknologin inte har kommit lika langt.˚ Ett foreslaget¨ satt¨ att fortsatta¨ ar¨ att borja¨ testa kliniska studier med fragor˚ dar¨ svaret bestar˚ av fri text istallet¨ for¨ svarsalternativ. Resultatet fran˚ studien kan da˚ i sin tur anvandas¨ i diskussion med myndigheterna for¨ att visa pa˚ alla positiva foljder¨ som ett roststyrt¨ system medfor.¨

ii Acknowledgements

I would like to give a special tanks to some people that have helped and supported me through my master thesis work. Anders Lofgren,¨ AstraZeneca. Thank you for connecting me with my supervisor and for guiding me through this thesis work. It has been great to always be able to reach out to you! Mats Pettersson, AstraZeneca. Thank you for the opportunity to write my master thesis at AstraZeneca and thank you for all the interesting discussions. Bengt Sandblad, Uppsala University. Thank you for being my reviewer and thank you for all the support! A special thanks to you for providing important inputs to my work in a relaxed and inspiring way. Caroline Algvere, thank you for the daily stand up meetings and all the motivational pep-talks.

iii Contents

1 Introduction 1

2 Background 1 2.1 AstraZeneca ...... 2 2.2 Clinical trials ...... 2

3 Motivation 2 3.1 Challenge ...... 2 3.1.1 Free information ﬂow ...... 3 3.1.2 Compliance ...... 3 3.1.3 Miss communication ...... 3 3.2 Goal ...... 4

4 Delimitation’s 4

5 Related work 5 5.1 Virtual assistant in clinical trials ...... 5 5.2 Virtual assistants in the lab environment ...... 5 5.3 This thesis contribution ...... 5

6 Theory 6 6.1 Clinical trial ...... 6 6.1.1 Compliance ...... 8 6.1.2 Dropouts ...... 9 6.1.3 Compensating participants ...... 10 6.1.4 Adverse events ...... 10

iv 6.1.5 Placebo effect in clinical trial medication ...... 11 6.2 Chatbots and intelligent virtual assistants ...... 11 6.3 Patients withholding information from their clinician ...... 11 6.4 Usability ...... 12

7 Method 12 7.1 Usability tests on users ...... 13 7.1.1 Qualitative method ...... 13 7.2 Conversation between subject and conversational assistant ...... 14 7.2.1 Interview questions ...... 15 7.2.2 Analysis of the captured data ...... 15 7.3 Locating key persons at AstraZeneca ...... 16 7.4 Literature study ...... 16

8 AstraZeneca’s way of performing clinical trials 16 8.1 Digitize the clinical trial process ...... 17 8.2 Contract Research Organization (CRO) ...... 17 8.3 Smart medical devices used in clinical trials ...... 17 8.4 Clinical outcome assessment (COA) ...... 18 8.4.1 How AstraZeneca use eCOAs in clinical trials ...... 18 8.5 Example instruments used in ePROs by AstraZeneca ...... 19 8.6 Communication possibilities during clinical trials between the participant and investigator ...... 19 8.7 Using gamiﬁcation as a motivation for participants to complete clinical trials ...... 20 8.8 Clinical trial process ...... 20

v 9 Peoples attitude towards AI and voice control 21

10 Conversational assistants 22 10.1 Relying on a conversational assistant to give correct medical recommendation ...... 23 10.2 Personalized voice proﬁles ...... 24 10.2.1 Voice biometric authentication ...... 24 10.2.2 External device authentication ...... 24 10.2.3 Password based authentication ...... 25 10.2.4 Federal guidelines for authentication ...... 25 10.3 Comparison of available systems ...... 25 10.3.1 Language support ...... 25 10.3.2 Emotions in conversational assistants ...... 26 10.4 Beneﬁts of using conversational assistants in clinical trails ...... 26 10.5 Ways to implement conversational assistants into patient applications . . 27

11 Identiﬁed possibilities of a voice-controlled system and its target group 27 11.1 Target users ...... 28 11.2 Analyzing patients psychological health without any extra effort to increase medical knowledge ...... 29 11.3 Patient-device event handling by voice ...... 29

12 Identiﬁed challenges with implementing a voice-controlled system 30 12.1 Regulatory challenges with a voice-controlled system ...... 31

13 Patient recorded outcome (PRO) through voice control 32 13.1 Identiﬁed challenges ...... 33

vi 14 Usability study 33 14.1 Implementation ...... 33 14.2 Result from the captured data ...... 34

15 Discussion 35 15.1 Free speech handling ...... 35 15.2 Solving the language problem ...... 36 15.3 Removing the risk of miss-communication between patient and virtual assistant ...... 36 15.4 Does a voice-controlled system increase expectations ...... 37

16 Evaluation 37

17 Conclusion 38

18 Future work 38 18.1 Recommendations for AstraZeneca ...... 38

vii 2 Background

1 Introduction

Biopharmaceutical companies like AstraZeneca perform clinical trials with human volunteers all over the world every day, intending to add to medical knowledge. As- traZeneca is currently trying a digitized Clinical trial process in small scale, but usually, they are performed without any digital tools. When digitizing the process, it is important to design a usable system for the user since it is crucial to keep participants from exiting the trial. this might in return reduce unnecessary costs and labour for the pharmaceutical company. This thesis investigates the possibility to implement a voice- controlled system for patients in clinical trials to increase usability and in return keep patients in the trial. The possible benefits will be discussed together with both technical and regulatory limitations. The challenges and goal are described on a deeper level in Section 3. A voice-controlled system is an extension to AstraZeneca’s current project “UnifyTM” that is described in Section 8. In the UnifyTM project, AstraZeneca is digitizing the clinical trial process with an application for the participants. The application will ease the workload both for the clinician and the participant’s to reduce the number of dropouts from the trial. With a voice-controlled system, usability can be increased even further. The system will, e.g, make it possible for people without reading/write knowledge to participate in the trial. The target group that can benefit from a voice-controlled system and possible benefits of such a system is discussed in Section 11. A major challenge with a voice-controlled system is that the communication from the participant can not be limited. As it is, for now, AstraZeneca uses pre-defined answers to questions described in Section 3.1.1. With a voice-controlled system, the user can say anything he/she likes and AstraZeneca will in return be obligated to handle this information from the user. This together with several other challenges are investigated and possible solutions for them are discussed in this report.

2 Background

In this section, a brief explanation about clinical trials and the company AstraZeneca is provided to help the reader understand chapter 3. A more detailed theory that is relevant for this report is presented in section 6.

1 3 Motivation

2.1 AstraZeneca

AstraZeneca is a global, science-led biopharmaceutical company who are operating in over 100 countries with millions of patients using their medicine. AstraZeneca’s research and development are concentrated to the headquarters in Cambridge (United kingdom), Maryland (USA) with a focus on biopharmaceuticals and Molndal¨ (Gothen- burg, Sweden) with a focus on traditional drugs. AstraZeneca’s main research areas are medicine to treat cardiovascular, metabolic, res- piratory, inﬂammation, autoimmune, oncology, infection and neuroscience diseases [6].

2.2 Clinical trials

A clinical trial is research of medication together with human volunteers, with the goal to add to medical knowledge about the medication. This is done as a basis for all new medication before the medication can be approved for sale but also on medication on the market to determine the impact from long time use. The medication is tested on human volunteers to determine if the medication is safe to use and to determine possible side effects. During the clinical trial, data is gathered from the patients in different ways. The patient has direct contact with the clinician but is also asked to ﬁll out forms with predeﬁned answers. A more detailed explanation about clinical trials is given in Section 6.1.

3 Motivation

In this chapter, the challenges and goals of a voice-controlled system are presented. The challenges are divided into three subcategories where free information ﬂow is discussed together with compliance challenges and miss-communication in a voice-controlled system. The goal is also divided into sub-goals to make the evaluation of the project easier.

3.1 Challenge

Is it possible to implement voice control into patient applications for patients taking part in a clinical trial, to reduce the workload for the patient and increase the knowledge about the medications possible side effects?

2 3 Motivation

3.1.1 Free information ﬂow

Implementing voice control in clinical trials will add a new way that a participant can report information related to the clinical trial. The user will not be limited to the pre- deﬁned answers, e.g, a scale 1-5 described in Section 8.6 when answering questions related to the clinical trial. Instead, the user will use natural language and can say anything. The user might, e.g, give information about a critical medical condition. The natural language has to be interpreted and handled accordingly to regulations and ethi- cal aspects. This brings a set of questions:

If the voice-controlled device interprets information from the user or surrounding • people, not related to the question, do AstraZeneca need to handle that information?

How can natural language be interpreted to compare with questions used in text- • based questioners with predeﬁned answers?

3.1.2 Compliance

With the free information ﬂow discussed in 3.1.1 appears compliance questions. All information has to be handled in a secure way to protect the participant. Authentication of participants in a voice-controlled system is also a question that needs to be addressed. Authentication made on various websites today requires the user to type in its password, which is usually covered by stars for security reasons. The stars make sure that surrounding people are unable to see the password. This brings the question, “is it safe enough for a participant to tell the virtual assistant its password?” since surrounding people might hear it? Using an external device instead, to authenticate the user might be negative for the usability level of the system. “How is it possible to authenticate the user securely without compromising the usability?”

3.1.3 Miss communication

A problem that might occur from implementing voice control, is miss communication between the participant and the intelligent virtual assistant (IVA). There is always a risk that the voice from the participant does not get registered or get registered the right way. The IVA also have to sort the information collected from the participant, e.g, information that is not interesting should be thrown away, adverse events have to be

3 4 Delimitation’s reported accordingly to instruction and data regarding the trial have to be submitted to the trial. Sorting data like this brings a set of problems;

If the IVA does not perceive all information from the patient in a correct way, • important data can be lost. The IVA sorts data the wrong way. This can imply that important data is lost or • compromise patient-security and integrity.

3.2 Goal

The goal of this project is to investigate the possibilities and obstacles with a voice- controlled system for patients in clinical trials. This goal will be evaluated based on the investigation of these three questions;

What possibilities come with implementing voice control? • In this part of the project, the possible benefits of a voice-controlled system will be investigated. What impact can a voice-controlled system have on usability as opposed to the systems used today? Should we replace the current systems or should a voice-controlled system be implemented as a complement to the technology used today? Also, who will benefit from a voice-controlled system? What group of patients will benefit from it but also if a voice-controlled system can result in more extensive, medical research will be investigated. What can we do in a technical aspect? • The possibilities and limitations from a technical aspect will be investigated to find out what AstraZenecas possibilities are with the technology accessible today and what can be done when the technology improves. What are we allowed to do in a regulatory aspect? • All pharmaceutical companies have to follow regulations when performing clinical trials, to ensure the company and patients safety. This implies that there are regulations that have to be studied to investigate what limitations there are with the implementation of a voice-controlled system.

4 Delimitation’s

This project is limited to investigate the possibility to implement voice control in clinical trials. No technical implementation of such a system will be a part of this project. This is

4 5 Related work due to time constraints and AstraZeneca’s time plan, they are still developing UnifyTM and that system is not far enough in the development process to include this feature. This work is also limited to the European Union and the U.S., following their laws and regulations.

5 Related work

The related work section of this thesis highlights other similar projects that have been done and shows the different approach this project has.

5.1 Virtual assistant in clinical trials

A startup company in Italy have developed a product that uses an intelligent virtual assistant (IVA) for clinical trials, PatchAi [23]. The communication between the patient and the virtual assistant is by text messages in an application. The focus is to build a bond between the patient and the virtual assistant to reduce the risk of the patient dropping off the medical trial.

5.2 Virtual assistants in the lab environment

There is a company called LabTwin that are developing a voice-power digital assistant for the lab environment. The lab worker can use LabTwin as an assistant to make hands-free data documentation during experiments, locating lab equipment and access scientiﬁc data.

5.3 This thesis contribution

PatchAi is developing an IVA for clinical trials that uses free text. Information regarding how they handle their “free text data” can be interesting to this project as a part of implementing voice control. This project is more focused on the voice part of such system and how it could work for AstraZeneca to develop it. Knowledge from LabTwin’s research in the area would also be interesting for the project wherein particular, usability can be interesting. What functionality that brings value but also what they have done that did not work.

5 6 Theory

This project can be seen as a combination on LabTwin’s and PatchAi’s work were the “free text” issue is discussed as “free speech” and instead of the lab environment, the system is used by anyone. The main focus difference from PatchAi is that the possible system is voice-controlled instead of text-based.

6 Theory

In this section, the important theory for this thesis is presented. Clinical trials are discussed together with relevant concepts. Theory about chat-bots, patients withholding information from their clinician and theory about usability are presented as well.

6.1 Clinical trial

A clinical trial is a research done with human volunteers (participants) to add to medical knowledge. The organisation hosting the trial is called an investigator and they work together with the participants, following a protocol for the clinical trial. The participants in the trial will either change a behaviour such as change of diet or be handed a medical product that will be tested. A medical product can be either a drug or a medical device. When performing clinical trials, the outcome is often unknown. The investigators and the participants do not know if the trial will be helpful, harmful or have no effect. All results are documented by the participant together with the investigators. In a clinical trial, both new and old treatments can be subjects for a clinical trial. Every clinical trial has an “ethics committee”, who is independent and consists of healthcare professionals [3]. Their responsibility is to protect the participants that participate in the trial and express an opinion on the trial protocol. The committee has a responsibility to investigate the protocol, documents and methods that are used in the trial and obtain informed consent from the trial subjects. The US Food and Drug Administration (FDA) divides clinical trials into three different phases [11], to help design clinical trials, following a series from phase 1 to phase 4;

Phase 1 is a small-scale study with 20-100 participants where the safety and • dosage are established.

Phase 2 is a mid-range study with up to several hundred participants, where the • goal is to establish the efﬁcacy and side effects.

6 6 Theory

Phase 3 is a large-scale study with 300-3000 volunteers to monitor efﬁcacy and • adverse reactions, i.e, adverse events.

If phase three is a success, the investigators send an application to the FDA, asking to get the treatment approved. Only 10% of the drugs tested in clinical trials, that are performed on humans, are approved and for those trials still running in phase three, only 25-30% passes [10].

Phase 4 is carried out after the drug/medical device has been approved and re- • leased, collecting data that appears from long time use. In phase four the drugs/medical devices safety, rare side effects and efﬁcacy are established but also the opti- mal way of use.

There are different regulations and recommendations for clinical trials in different parts of the world. When performing a clinical trial, these have to be taken into account. To release a medication worldwide, all regulations have to be taken into account when performing the trial. Since there are different agencies approving medication in different parts of the world, some medication can get prohibited in, e.g, Europe while they get approved in the U.S. When the European union where created, all 28 countries had different regulations which now are synchronized and controlled by the “Clinical trial regulation” (CTR) [26] [12]. The U.S. relies on one agency, the FDA and their process on how to carry out a clinical trial. A comparison of the European unions and U.S. approval processes are shown in Figure 1 to show an example of how it can differ in different parts of the world.

7 6 Theory

Figure 1 Comparison of European and U.S. Approval Processes [12].

6.1.1 Compliance

As a company, AstraZeneca is bound by the “General Data Protection Regulation 2016/679” (GDPR) when handling patients personal information within the European Union (E.U.).

8 6 Theory

GDPR is an E.U. law and regulation to protect personal data and make sure that the members can feel free and secure when using internet [25]. There are no equivalents of GDPR in the U.S. which implies that companies in the U.S. are not obligated to inform their users how their data are used [16]. However, companies that operate within E.U., still have to follow this regulation with E.U. citizens. Some global companies like Face- book, Google and Apple have changed their privacy policies to satisfy GDPR [16] as a standard privacy policy. As a pharmaceutical company, AstraZeneca is bound by the “Clinical Trial Regulation” (CTR) when performing clinical trials [26] within the countries of the European Union (E.U.). For medical trials performed in the U.S. there is the “Food and Drug Adminis- trations” (FDA) [11] guidelines that regulate clinical trials. U.S. Department of health and human services has an act, Health Insurance Portability and Accountability Act of 1996 (HIPAA), to protect individuals medical records and other personal health information [17].

6.1.2 Dropouts

In clinical trials, there are always patients that drop out of the trial. Dropouts occur in every phase of the clinical trial, whereas in phase 3, it is not unusual that 30% of the participants dropout [27]. The reasons for this can be many, where some common reasons are [18];

Personal or family matters. • Lack of appreciation. • Misunderstood expectations. • Side effects. • Forgetting visits. •

Since participating in a medical trial is voluntary, participants have the right to drop out whenever they want, making it impossible to eliminate patient dropouts. Avoiding as many dropouts as possible is important since dropouts are expensive and a waste of resources for pharmaceutical companies.

9 6 Theory

6.1.3 Compensating participants

Compensating participants in clinical trials, for taking risks is a common thing to do and it is accepted as long as the participant does not become coercive [28]. Rules and boundaries for compensating participants are local and differ depending on what region the clinical trial takes place. AstraZenecas participants do not get payed but they might get compensated for travels, lost income from work when meting with clinicians etc. depending on the region the clinical trial takes place in but it also differs depending on the trial.

6.1.4 Adverse events

An adverse event (AE) is an unfavourable event that occurs from a clinical trial, e.g, unintended abnormal laboratory ﬁnding, a symptom or disease associated with the use of a medical treatment [8]. There are ﬁve grades of adverse events, where every grade is handled differently, which are graded in the following way [8];

Grade 1: Mild AE. • Grade 2: Moderate AE. • Grade 3: Severe AE. • Grade 4: Life-threatening or disabling AE. • Grade 5: Death related to AE. •

The European union’s deﬁnition of a serious adverse reaction is; death, life-threatening, requires hospitalisation or prolongation of existing hospitalisation, results in persistent or signiﬁcant disability or incapacity, or is a birth defect [9]. For a suspected unexpected serious adverse reaction (SUSAR), reporting requirements and processes differ from where the trial takes place. In the cases of a SAR where the reaction is life-threatening or death and that the clinician suspects that it is related to the medication, the timeline to report this is only 24 hours. In the European Union, a report must be submitted within 15 days (7 for death or life-threatening events) to the corresponding national authority. In the U.S. the adverse event has to be reported within 15 days to the FDA if the adverse event is serious and unexpected [24].

10 6 Theory

6.1.5 Placebo effect in clinical trial medication

To verify that treatment from a clinical trial is effective beyond the physiologic result, some participants receive an inert substance. This is done to rule out the possibility that the result from the clinical trial is based on the placebo effect. The result from the participants who receive the active substance is compared with the result from the group who receive the inert substance. The inert substance is usually a sugar pill and the participants are aware of the possibility that they might not receive an active substance.

6.2 Chatbots and intelligent virtual assistants

A chatbot is a software developed to answer questions through voice or text. The ﬁrst chatbots were developed in the late 1960s with simple functionality, scripted responses [1]. Since then, the development has only moved forward and today people talk about virtual assistants as a common name for chatbots and intelligent virtual assistants (IVA). IVA is kind of a chatbot but with machine learning functionality, which means that they learn from the user in every interaction [1]. This makes them far more intelligent than a chatbot and can serve the user in a better way. An IVA can assist a user in a large number of tasks since every response does not need to be scripted and are then more suitable for other tasks than just simple routine tasks.

6.3 Patients withholding information from their clinician

Most patients have withheld relevant information from a clinician at some point [20]. There are several different reasons why patients withhold information from their clinicians. From a study done at the University of Michigan, the most common reason for withholding information is that the patient does not want to be judged by their clinician [20]. Disagreeing with the clinician’s recommendations and failure to understand the clinician are also reasons why the patients withheld information. Security concerns when providing information is also an important reason why patients withhold information. An analysis of the fourth wave of the Health Information National Trends Survey shows that 12.3% of the respondents withhold information because of security concerns [4].

11 7 Method

6.4 Usability

Designing a usable system does not imply designing a system that someone can use, but a system where the user can do what he/she implies. The features that the system has together with what the user intends to do with the system determines its level of usability [2]. There is an official ISO 9241-11:2018 definition [19] of usability: “the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”. There was also an official standardization with guidance on how to design for usability, ISO 1340:1999, that was based on the definition of usability but is withdrawn. When designing a usable system, it is important to have the user in focus and that the process follows some key principles [15]. ‘User-centered` system design” is based on these principles where the user’s goal and activity are in focus in the design process. The user should be a participant through the entire design and development process.

Step 1, Analyze: An analysis of the user’s needs and requirements is done to- • gether with the user. Different scenarios are evaluated and the design goals established.

Step 2, Design: Design a prototype together with the user. • Step 3, Evaluate: An evaluation of the prototype starts where both the usability • and effect of the system is evaluated.

Step 4: Feedback: Back to step one with feedback from the current design. A • suggestion of change together with a plan for the change is developed.

When step 4 is done, the process starts over at step 1 again until the system is fully developed. Since the user is involved in every step, it is important to keep a simple design presentation where no prior technical knowledge is needed.

7 Method

The methods that are used in this project, how they are used and why those methods are used are presented in this section.

12 7 Method

7.1 Usability tests on users

Usability tests provide a quick method for identifying key usability problems in an interface or application, they cannot guarantee that all critical design problems can be identified but will give an indication of issues with the system. Interviews were held to determine if users felt comfortable with an automatic authentication model. For this usability study, a function for a voice-controlled assistant was made with the purpose to recognize the voice of the subject and give the subject functionality that the interviewer did not have. This was initially created for Amazons Alexa but Amazon had a problem with their personalized voice profiles, making it impossible to create a new profile. To overcome this problem, a function for Google Home was created instead and a Google Home mini was used in the study. The goal was to investigate if the subjects were comfortable revelling personal information in this environment and if the user trusted that no one else could access the subjects personal information.

7.1.1 Qualitative method

For this research, a qualitative research method was chosen to gather more information regarding the system. A quantity method is used when a measurable result is required, which is not the purpose of this study. A semi-structured interview was held with the subjects to investigate their impression of the system in the context of usability and regulatory aspects. An interview was chosen due to authentication through voice proﬁles is a new area of research and we want to know everything about the subjects experience. Both the subject concerns about the system and positive feedback from the subject were important for the study. The interviews were made semi-structured, meaning that the interview had some mandatory questions but the subject was also encouraged to express his/her opinions on other aspects of the system. Together with the interviews, an observational study took place where the subject’s reactions were observed in the interaction with the conversational assistant. The reason for this was to pick up on reactions like insecurity, fear, joy and excitement and use this to drive the subject in the interviews but also as a result of the study. This study gave a subjective result from the subjects with the reactions and thoughts only from a few people. Due to the low number of subjects, it was important to have a diverse test group with people of different age, gender and technical knowledge. This was also important since clinical trials include participant from all over the world with different experience and knowledge and a diverse group could then lead to a more lifelike result.

13 7 Method

7.2 Conversation between subject and conversational assistant

First of all, the subject was asked to register his/her voice in the Google assistant application to make the assistant recognize the user. Some users used their own Google ac- counts with their registered voice while some users used the interviewer’s account. The language of the conversation was Swedish since all subjects had Swedish as a native language. For the readability of this report, the conversation is translated into English. The subject was asked to start the interaction with the invocation phrase “Start Victor’s master thesis work” and were left to continue the conversation without any guidance. The conversation is shown in Figure 2.

Figure 2 Conversation between subject and conversational assistant in the usability study.

14 7 Method

7.2.1 Interview questions

The interviewer marked answers on a scale 1-10, depending on the subject’s answers. This was done by the interviewer to prevent that the subject gets stuck with numbers and a normal conversation was kept.

Observations;

Did the subject express discomfort in any part of the interaction? At what point • of the interaction did this occur?

Did the subject express some kind of joy, excitement or satisfaction from the • interaction? At what point did this occur?

Did the subject show any confusion regarding the interaction, e.g, if the subject • had a hard time understanding that he/she was authenticated.

Did the interaction work without any friction or was there some part of the inter- • action that did not work as planned?

Questions

How well did you understand that the system knew who you were? • How did you experience the interaction with the voice-controlled assistant? • What was your impression of the authentication model? Did it feel secure? Would • you rather authenticate yourself in another way?

Did you feel comfortable revealing private information regarding your medical • status to the application?

7.2.2 Analysis of the captured data

The interaction between the subject and conversational assistant was recorded with both video and audio to make sure that every reaction from the subject was captured. The main strategy was to keep going with the usability tests until no more new data were found. A minimum of six participants with a variety in both gender and age for the study were also established.

15 8 AstraZeneca’s way of performing clinical trials

A thematic analysis was chosen to find important data-patterns from the participant’s answers and reactions. Thematic analysis is a flexible method where the researcher wants to find patterns in the collected data. This study will have a latent approach meaning that underlying ideas, assumptions and conceptualization is part of the analysis. A latent approach was chosen to make sure that all data provided was collected since participants might be uncomfortable being recorded they might forget to reveal important information. The data captured will be coded into themes and every theme will be described in words together with quotations from the participants. This is done to create an easy way for the reader to get key insights into the result of the study.

7.3 Locating key persons at AstraZeneca

In a large company like AstraZeneca, the employees possess knowledge and information that have been acquired through experience from working at AstraZeneca. As part of this project, experts in areas related to clinical trials and voice control were located and discussions took place, with the purpose of information gathering. Locating key persons were done together with the external supervisor at AstraZeneca and the employees were then asked to point to other persons of interest. This was done as an initial process to get an understanding of how AstraZeneca works in clinical trials and what goals they are aiming for. Further, more extensive discussions were made to ﬁnd out what obstacles and possibilities there are when implementing voice control for patients in clinical trials.

7.4 Literature study

In the starting process of this project, background information was gathered and com- piled to build a foundation for future research and basic knowledge for interviews. In- formation regarding pharmaceutical companies, clinical trials, chatbots, adverse events and compliance questions are seen as basic knowledge in a pharmaceutical company and had to be investigated before moving forward. This research was done mainly through literature studies were regulations from the European Union and federal organisations in the U.S. were the main research area.

8 AstraZeneca’s way of performing clinical trials

An interview was made with Asa˚ Hagman, an employee at AstraZeneca, working on a project called UnifyTM. The purpose of the meeting was to get a deeper understanding

16 8 AstraZeneca’s way of performing clinical trials of AstraZeneca’s clinical trial process. She provided knowledge about how clinical trials are performed today and AstraZeneca’s vision on how trials will be performed once UnifyTM is implemented. UnifyTM is scheduled to be released at the end of 2020 if everything goes as planned.

8.1 Digitize the clinical trial process

UnifyTM is an AstraZeneca project in the development phase with the purpose to digitize clinical trials. The goal is to have a joined system where all concerned actors can operate. The participant can access relevant information regarding the clinical trial and to communicate relevant information to the investigator. The participant will receive notiﬁcations about meetings with the investigator when it is time to take the medication and will be able to complete tasks related to the clinical trial. The investigator will have access to a different part of the application were communicating important information to the participant will be possible. The investigator will also receive information logged by the participant and the participant’s smart medical devices. AstraZeneca will be able to follow the progress of the clinical trial through the application as well.

8.2 Contract Research Organization (CRO)

A CRO is an organization that provide clinical trials to pharmaceutical companies, i.e, competent staff, equipment and facilities. Contracting with a CRO implies that the pharmaceutical company do not need to hire permanent staff to do in-house trials, while still getting the required expertise. AstraZeneca exclusively uses CROs for clinical trials to reduce the need for infrastructure, ofﬁce space and staff that it takes to perform the clinical trial by themselves.

8.3 Smart medical devices used in clinical trials

AstraZeneca uses smart devices, connected to some of their medication in clinical trials, making logging information automatic, e.g, what time the patient used their asthma inhalator. This information will be connected to UnifyTM, giving the investigator access to this information. If the patient fails to follow the clinical trial process, the investigator will be made aware of this through the application and this information will be included in the clinical trial protocol. The purpose of these devices is to make the results from clinical trials as reliable as possible and to reduce the workload when logging information. Some participants still fail to follow through with the treatment and, e.g, open the

17 8 AstraZeneca’s way of performing clinical trials lid and discard the pill, making it impossible to know for sure if the patient used the medication.

8.4 Clinical outcome assessment (COA)

Patient recorded outcome (PRO) is a part of the clinical outcome assessment (COA), a health outcome reported directly by the patient. COA is a measurement used to evaluate the patient’s safety and besides PRO consists of; Performance Outcome (PerfO), Clinician-reported outcome (ClinRo) and Observer- or caregiver-reported outcomes (Ob- sRo). Electronic patient recorded outcome (ePRO) is an electronic tool used by As- traZeneca to collect information regarding the patient’s health in clinical trials. ePRO’s are carried out in various ways today, depending on which trial they are used in. For some trials the patient may bring their own smartphone or tablet, this is called Bring you own device (BYOD) while in some cases the patient will receive a phone from the clinician, that will be used to complete ePROs among other things. BYOD implies that the patient may use a personal device, e.g, a phone or a tablet to complete the ePRO survey. In some other trials, BYOD is not possible to use due to, e.g, sensitive information that needs to be deleted from the device after the trial is completed or that there is a risk that other applications might compromise the safety of the trial-related information. The phone supplied by the clinician is protected from external media and external devices, to protect the participant, e.g, the participant can not install applications or connect headphones to the phone. When the patient has completed the clinical trial, the phone is returned and AstraZeneca can delete all data that is stored from the trial, preventing sensitive information to be compromised. In clinical trials performed “on-site”, the participant uses a tablet provided to record ePROs. When the participant starts an ePRO survey the date and time have to be recorded and reported together with the data from the ePRO survey to AstraZeneca.

8.4.1 How AstraZeneca use eCOAs in clinical trials

A study team at AstraZeneca reach out to a patient centre science (PCS) team with information regarding a treatment that should be tested or something that needs to be measured through eCOA. A team of a PCS manager and director investigates the proposal and put together a plan on what instruments should be used to measure this treatment. The PCS team reach out to a department called “eCOA Enablement” in Poland who specialize in eCOA solutions and have direct contact with a company called ERT, who provides the technical solutions for eCOA instruments used in clinical trials. RWS is a company who provides licenses and language translations for all eCOA solutions that

18 8 AstraZeneca’s way of performing clinical trials

AstraZeneca use. All data collected from eCOAs are stored in databases and analyzed by statisticians at AstraZeneca. The result can be used as a ground when trying to get a permit for the treatment at the authorities or can be used for further research.

8.5 Example instruments used in ePROs by AstraZeneca

AstraZeneca uses a wide range of instruments as part of ePROs to get accurate information from the patient. The design of the instruments varies from questioners to more interactive instruments where the patient, e.g, shall paint on a virtual human body to show where he/she is in pain. An example question from a questioner where the symptom term is “dry mouth” is “In the last 7 days, what was the SEVERITY of your DRY MOUTH at its WORST??” with the answer options;

None. • Mild. • Moderate. • Severe. • Very severe. •

8.6 Communication possibilities during clinical trials between the participant and investigator

In the UnifyTM application, participants will be able to answer questions from the clinician, complete ePROs and report other trial-related information. All possible answer options in the questions and ePROs are predeﬁned to make sure that the study team only receives information that they requested from the patient. If the patient needs medical assistance or experience life-threatening symptoms, the user can not report this in the application and should instead call the emergency number or contact the health care provider immediately. The patient can always contact the study team to report other issues and concerns regarding the clinical trial, e.g, an adverse event that the ePROs and questions do not cover. There is no way for the participant to report an adverse event in the UnifyTM application, this has to be done directly to the study team.

19 8 AstraZeneca’s way of performing clinical trials

8.7 Using gamiﬁcation as a motivation for participants to complete clinical trials

In the UnifyTM application, gamiﬁcation techniques will be used to motivate the participants to complete their ePROs and other tasks related to the clinical trial in time. The participant will receive credits every time a task, related to the clinical trial, is completed. Receiving credits when completing a task shall motivate the participant to complete more tasks and will be a part of the work to reduce the number of dropouts that AstraZeneca has in clinical trials. The credits that the participant collected during the clinical trial can be traded for charity donations.

8.8 Clinical trial process

The clinical trial process for AstraZeneca, when UnifyTM is released, is divided into three actors; A human volunteer (participant), an investigator together with a team of healthcare professionals and a clinical research associate (CRA) together with other As- traZeneca actors. Every study starts with a local study leader (LSL) at AstraZeneca, which leads the trial and initiate contact with the investigator. The LSL initiate contact with the investigator, sending a clinical document architecture (CDA), for the investigator to review. The investigator decides if the clinical trial is a suitable project for them, in the sense of their potential patient populations, required equipment for the trial and facilities. The LSL reviews the investigator’s feasibility study, validates the data and schedule a site qualification visit with the investigator’s team. The LSL confirms site qualification and the agreement is made. A CRA is assigned to the new study, who views key information regarding the clinical trial and complete study-specific training before establishing contact with the investigator through UnifyTM. A training process is started where the investigator is introduced to the UnifyTM application and taught how to use it. The investigator also gets educated on how to perform a specific clinical trial. During this time, every involved countries ethics committee is involved in the process, who will ensure the patient’s safety during the trial. The corresponding countries medical product agency, are also informed, who are responsible for the regulation and surveillance of the clinical trial. Before intro- ducing any participants to the study, the corresponding authorities have to approve the clinical trial and the investigator should have the relevant education. The investigator initiates a candidate search, reviews their patient records to find possible prospects for the clinical trial. Some organizations that perform clinical trials have multiple trials running simultaneously, so participants might be interesting prospects for

20 9 Peoples attitude towards AI and voice control multiple trials. All possible prospects are informed about the trial and receive a publicly available URL with information regarding the trial. The prospect gets some time to reﬂect at home to decide if he/she is interested. If the prospect is interested to know more about the trial to participate, he/she can reach out to the investigator to receive information about the UnifyTM application. In the application the prospect has access to the informed consent form (ICF), supporting guidance, study material and the possibility to give the caregiver access. In the application, all content regarding the trial is presented, to make sure that the prospect can make an informed decision. A FAQ is also available in the application together with contact information to the investigator, if the prospect has any questions regarding the material. All prospects that decide to participate in the trial signs the ICF. When the ICF is signed, a screening process is initiated with the prospect which has to be passed before the prospect can become a participant of the clinical trial. During this process, the CRA can monitor the process through the UnifyTM application and answer the investigator’s questions. A teaching phase will begin where the participant will be introduced to UnifyTM and the corresponding medication that will be tested. The participant will receive information on how to self-administrate the medication. When the treatment begins, the participant receives ongoing and regular reminders through the UnifyTM application to take a dose, administer the drug, carry out tests at home, visit the Investigator, pre-visit checklists and carry out ePROs and surveys. Some treatments will not have a deadline, e.g, for some cancer treatments. If the treatment affects the participant, the treatment will proceed until the patient is healthy or until the patient ends up deceased.

9 Peoples attitude towards AI and voice control

A survey was done to establish peoples attitude toward intelligent personal assistants IPAs [21], sponsored by the National Science Foundation under grants No 1640640 and 1640697. The survey shows that people who expressed higher privacy concerns were less likely to use an IPA. The number of participants completing the survey was 1178 people, 18 cases were removed, since they did not meet the participant criteria or were missing signiﬁcant data, leaving a ﬁnal sample of 1160 persons. 59% of respondents were women, and 92% had obtained at least a bachelor’s degree. The age of the participants varies between 20 and 82 years old (M=38.17, SD=12.72, range: 20-82), a majority (61.5%) were iPhone users, and 37.8% were using an Android phone. 45% of the participants use an IPA on their phones, such as Siri or Google Assistant

21 10 Conversational assistants while 28% never had used IPAs on their phones. 11% had used IPAs on their phones in the past. For the home users of IPAs, like Google Home or Home Mini, Amazon Echo or Echo Dot, 29% owned at least one IPA device. 71% of the home IPA users owned an Amazon device, 29% a Google device and 12% both devices. The reason why people do not use IPAs were also investigated in the survey with a scale 1-5 (1 = Not at all important and 5 = Very important);

I do not see any beneﬁts from this feature (M=3.48, SD=1.23). • I do not like talking aloud to my phone (M=3.38, SD=1.36). • The user interface is frustrating (M=3.18, SD=1.03). • It is awkward to use (M=3.14, SD=1.03). • It does not understand my voice most of the time (M=2.96, SD=1.34). • I have privacy/security concerns about these features (M=2.88, SD=1.37). •

7% of the participants that were not using a phone IPA today, identiﬁed privacy concerns and trust issues as their primary reason. Problems like a frustrating interface and the system not understanding the users are things that is possible to improve and change to increase the number of IPA users. With a result of 45% frequent users of phone IPAs and 29% for a home IPA, it clearly shows that there is a need for people to use IPAs. It also shows that IPAs are not for everyone, at least not yet, which implies that it can be used in special circumstances by a part of the population but it needs a compliment. If the user is in a public space he/she might not feel comfortable speaking to the phones IPA and should be able to perform the task by standard phone use. People with a speech impairment is another reason why voice control can not exclude the technology used today, everyone can not make themselves understood by speech. On the other hand, implementing voice control in clinical trials might help visually impaired to participate in the trial.

10 Conversational assistants

A conversational assistant is an intelligent virtual assistant that communicates through voice control. There are some publicly known conversational assistants available on the

22 10 Conversational assistants market today where the most known ones are; Apple’s Siri, Amazon’s Alexa and Google Assistant. Some less common assistants are Microsoft’s Cortana, Hound, Bixby, An- droids Robin and Lyra Virtual Assistant. Alexa and Google Assistant are purely voice- based systems while Siri also has a digital interface that can aid the user in achieving the goal. Siri is only available on Apple platforms while Google Assistant and Alexa applications are available on iOS and Android together with their corresponding products, e.g, Google Home and Amazons Echo Dot. Conversational assistants goal is to support the user achieving his/her goal with minimum effort, i.e, assisting users who use speech rather than traditional technology. Some Examples where conversational assistants can be helpful are; Setting alarms, adding items to lists, ﬁnding the fastest travel route to the goal, ﬁnding information online and playing music.

10.1 Relying on a conversational assistant to give correct medical recommendation

A research was done where Siri, Alexa and Google Assistant were tested in the ability to provide medical advice [7]. The research only had 54 participants but it still shows a dramatic result which is used as an indication of how reliable medical recommendations from conversational assistants are. In more than half of the interactions, the conversational assistant failed and led the user to draw the wrong conclusion regarding his/her medical state. In 12.4% of the cases the conclusion made by the user, from the conversational assistant’s recommendations, could have led to harm for the user and in 6.9% the concluded action could have resulted in death. Siri had the lowest ratio of task failure (22.5%) but also the highest ratio of potential harm (29.9%) and death (20.9%). Alexa had the highest ratio of task failure (91.9%) but the most secure system with the lowest number of potential harm (1.5%) and death (1.5%). Google Assistant had a task failure of 51.8% and resulted in potential harm in 15.5% of the cases and 5.4% of the cases potential death. The participants of the research had the highest trust in Siri and the lowest in Alexa after the research. Both the conversational assistant and the users were causing the failures in the process of ﬁnding the correct diagnoses of the user, e.g, some users did not reveal important information regarding their medical condition to the conversational assistant since the assistant did not understand the ﬁrst time. This highlights the importance of a system with high usability and shows the serious impacts that a failing system might result in.

23 10 Conversational assistants

10.2 Personalized voice proﬁles

Several conversational assistants offer the functionality of personalized voice proﬁles, with the purpose to offer the user a personalized experience. This can be used when more than one person is using the same device, e.g, when a couple has different taste in music. The conversational assistant can then customize the music choice to the person speaking to the device. This is not a way to authenticate a user and personalization can not be trusted with sensitive information. For Siri, Google Assistant and Alexa, all three can recognize different voices to offer a personal experience for the user [22] [14] [5].

10.2.1 Voice biometric authentication

A personalized voice proﬁle can be an intriguing way of implementing authentication into a voice-controlled system. As explained in Section 10.2 though, this is not possible due to the security level of the system. With voice biometric authentication as an extra layer of authentication, this might be possible to implement. Voice biometric is a way to authenticate users by recognizing their voice when the user talks. Using voice biometrics as an extra layer of security can be interesting for a voice-controlled system in clinical trials, creating the possibility to authenticate the user with zero effort from the user. The user can use the conversational assistant as usual but with the ben- eﬁt of accessing personal information and settings without authentication. This feature can increase the usability of a voice-controlled system by minimizing the work for the user when completing tasks related to the clinical trial that normally requires the user to authenticate him/her self. Implementing voice biometric authentication in clinical trials requires a certain security level. This is due to health-related and personal information being processed which have to be kept safe as explained in Section 6.1.1. The level of usability the voice recognition system offers, is also important when discussing a possible implementation of the feature. Users are less likely to keep using a failing system and it is therefore important that the system has a high success rate.

10.2.2 External device authentication

External devices is another way to authenticate users, e.g, that the user can use face recognition or a secure password through his/her smartphone together with an application for authentication. This is widely used today where users can authenticate themselves in bank applications and other high-risk applications. This will create an additional step for the user before he/she can access the application and the user might have

24 10 Conversational assistants to get the smartphone. External device authentication is used today, which makes it comparably easy to implement but have an additional step that the user has to complete before using the application.

10.2.3 Password based authentication

Another identiﬁed way of authentication through voice control is a standard password- based authentication. This can be done through third-party applications described in Section 10.2.2 or via the voice-controlled assistant by voice. With an external device, we have the same issue that the user might have to search for his/her smartphone and if the password is spoken out loud, surrounding people might hear it which is a breach in security.

10.2.4 Federal guidelines for authentication

To get a medical application approved by the federal government’s, AstraZeneca has to prove for the authorities that the application is safe in a privacy context. Biometric authentication is classified as a high-risk authentication method since the impact can be severe if an imposter is able to fake, e.g, the user’s voice or fingerprint. To implement a biometric authentication method, AstraZeneca has to prove that the elevated risk is worth the gain from the benefits.

10.3 Comparison of available systems

The Amazon Alexa application is available for iPhone and Android users but speaking with Alexa is only possible on Alexa built-in devices, ruling out the possibility to have communication through voice on, e.g, iPhone. For Google Assistant, communication is possible through the Google Assistant application which is available for both Android and iPhone. Siri is only available on Apple products but has the positive feature that Siri can provide results on a physical interface.

10.3.1 Language support

Alexa is available in eight different languages, Google Assistant is available in 18 different languages and Siri in 20 different languages. From those languages, multiple

25 10 Conversational assistants accents are available, e.g, both British and American English. Using one of these existing conversational assistants would imply that the clinical trial only can include users that have a native language supported by the conversational assistant.

10.3.2 Emotions in conversational assistants

Amazons Alexa has the functionality of using emotions through tone when speaking, e.g, a happy tone when the user does something great and a sad tone if she is disap- pointed. Early customer feedback from Amazon indicates that using emotion-based conversation increases the voice experience by 30% [13]. If the experience is increased, the user is more likely to keep using the conversational assistant. This is a new feature who Amazon has limited to music and news at the moment but as a developer, it is possible to use this feature when creating Alexa-skills. Emotional conversations are not yet supported by Google Assistant and Siri and as a developer, it is not possible to include emotions when developing for those conversational assistants.

10.4 Beneﬁts of using conversational assistants in clinical trails

The main reason to use conversational assistants in applications for patients is to increase the usability of the system, including more ways for the patient to interact with the application. Some users prefer using traditional technology, using the computer key- board or the touch screen on the phone, while some prefer speech. If a voice-controlled feature is included to the patient application, more users will have their preferred way of communicating satisﬁed and the system will in return satisfy more users. Another huge beneﬁt of a conversational assistant is reminders. Sending a reminder to the user through the conversational assistant includes the possibility for the user to answer right away, e.g, “Hey USERS NAME, you have not completed this survey yet. Are you okay?” and the user can just answer “Oh sorry, no I have a headache.”. At this point, information regarding the user’s headache issue can be registered and the conversational assistant can continue the conversation to push the user to complete the survey.

26 11 Identiﬁed possibilities of a voice-controlled system and its target group

10.5 Ways to implement conversational assistants into patient applications

Some different possible strategies have to be evaluated when discussing implementing conversational assistants into patient applications. One way is to focus on bringing a conversational assistant into the application as a function where the user, e.g, might press a button or say “Hey, CONVERSATIONAL ASSISTANTS NAME” to start the voice-controlled assistant. This would limit the number of compliance issues since no external devices would be used. Another strategy is that the user can connect its own devices, e.g, the user can connect to the patient application through an Alexa echo dot or Siri from the users iPhone. This would imply that the user can use the conversational assistant and the technology that he/she is used to and that no additional device has to be provided to the user. Using both strategies is also a possible way to do it. This solution would include both users that regularly use conversational assistants and users that are interested to try it out without any extra effort. An additional way of addressing this implementation question is to use an external device connected to the patient application, e.g, a Google Home that the user get allotted. This take would limit the compliance problems to only one external device but it would also limit the target group since there is an additional device that the user will have to learn how to use and to connect. Participants that are regular users of Google home, in this case, might probably use the voice-controlled feature but for the users who own an Alexa device might not want to start using an additional device.

11 Identiﬁed possibilities of a voice-controlled system and its target group

There are a lot of possibilities that have been identified for a voice-controlled system during this project. A voice-controlled system would not only benefit the user in situa- tions where traditional phone-use is not possible, e.g, when the user is driving a car. A voice-controlled system will also increase medical knowledge since more data from the user can be processed. In this section, target users are identified together with interesting scenarios where a voice-controlled system might be useful.

27 11 Identiﬁed possibilities of a voice-controlled system and its target group

11.1 Target users

Clinical trials are performed worldwide including people of different cultures, age, gender and technical knowledge. This diversity implies that some of the users will have no prior experience in using technology, while some might use it every day. There is also a diversity in users, in the sense of physical ability, where users have different possibilities to use technology. To adapt to more users needs, some important target groups where a voice-controlled system might have an impact have been investigated. The identiﬁed user groups that can beneﬁt from a voice-controlled system are:

Users with no prior technical experience: • One of the target groups where a voice-controlled system would be beneﬁcial are users with no prior experience of technology. An intuitive voice-controlled solution might be easier for the low-experienced user to use, than learning to control both a phone/computer and the application within those technical devices.

Users who have a physical disability: • Some participants in clinical trials do not have the physical ability to use a computer or a phone to do clinical trial-related tasks. This can be due to a physical disability, e.g, that the user is not able to use his/her hands for some reason or that the user is visually impaired and can not interpret a physical interface. Some of these participants might be able to perform the tasks by voice instead.

Users who lack reading and writing knowledge: • An issue that AstraZeneca has encountered are dropouts related to participants that do not know how to read and write. The participants in question fail to complete trial-related tasks and can not complete the clinical trial. Implement- ing voice control in the patient’s application, as a way for the patient to complete trial-related tasks, would increase the number of prospects that can participate in the trial and reduce the number of dropouts.

Regularly users of conversational assistants: • For users that use conversational assistants regularly to complete everyday tasks, there might be easier to continue using the conversational assistant for clinical trial-related tasks than to learn how to use a new application.

Users who can use voice control as a complement to existing technology: • The conversational assistant can also work as a complement to traditional technology, aiding users to complete their tasks when they are not able to pick up their phone or computer.

28 11 Identiﬁed possibilities of a voice-controlled system and its target group

11.2 Analyzing patients psychological health without any extra effort to increase medical knowledge

There is technology connected to conversational assistants that can determine the user’s mood for the day. With daily conversations with a conversational assistant, the users change in behaviour and mood can be measured to determine if the users psychological health is increasing or decreasing. For a depressed user, it might be easier to answer questions that are related to the trial, e.g, that the user has taken the medication instead of answering questions about his/her health. A more reliable result can be acquired since the feeling of well-being is relative to former feelings and the user might have a good/bad day relative to the day before when answering questions. The user might not be able to compare his/her psychological health the day of the questioner, to the psychological health when starting the trial, since the user has an exceptional good/bad day relative to yesterday when answering the questions. With a system that analyze the user’s responses every day, the change in mood can be measured and compared over the whole trial without relying on the user’s perception of his/her psychological health and will in return give a more reliable result.

11.3 Patient-device event handling by voice

In some clinical trials, patients have smart devices connected to their medication, e.g, a device that monitors the activity of the user’s asthma inhalator. This is used to collect data regarding the user’s medication, especially deviations that might occur in the patient’s medication. Another example where a smart device might be used is to monitor the user’s heartbeat during a clinical trial through electrocardiography (ECG). When the user has extraordinary strong/weak heartbeat this will be registered as an event and investigated further. For now, these events have to be investigated by the study team together with the patient to ﬁgure out why the event occurred, e.g, if the patient were running during the time of the event or if the event might be related to a physical health issue. To help the study team analyze data from the smart devices connected to the user’s medication, a question could be sent to the user, e.g, “Is there anything that feels weird/different right now?”. The user, in turn, could either use the application or use the voice- controlled assistant to tell about a medical condition or that everything is ﬁne. Providing this possibility for the user to answer right away will give a more accurate result since the user does not have to remember exactly what happened at that time in a later stage when talking to the clinician. This will reduce the workload for the clinician by removing some false events from their investigation.

29 12 Identiﬁed challenges with implementing a voice-controlled system

Another feature related to smart devices can be to ask the voice-controlled assistant where the smart device is located. All smart devices are connected to a tracking system and can be located. The user could just ask “Where is my asthma inhalator” while searching and the assistant can, e.g, answer “It is in your bedroom”. This can be helpful since patients tend to forget where they put the device and an easy way to ﬁnd it can increase the usability of the system.

12 Identiﬁed challenges with implementing a voice- controlled system

Some of the general challenges regarding voice control identified in this project will be presented in this section. There are additional problems identified that are related to specific areas where voice-control can be interesting and they are presented in every corresponding section. The challenges stated here have to be solved before a voice- controlled system can be implemented in any of AstraZenecas patient applications;

How can conversational assistants be used for patients without WiFi • Conversational assistants require a WiFi connection to function and all participants of the clinical trial might not have a WiFi connection in their homes. In the current solution where the patient can receive a phone from the clinician, this will not be a problem since the phone can use the mobile network. If voice control is implemented into the phone application as a feature of the existing functionality, this will not be a problem either since access can be granted through the mobile network. With an external assistant, on the other hand, a WiFi connection has to be available for the conversational assistant to function.

Teaching patients how to use a conversational assistant where the patient • have no prior technical experience or are unable to leave their home Using a conversational assistant in clinical trials imply that the user has to be taught how to use the technology. For users that have no experience with technology or that are physically unable to install a technical device, there might be a problem to get started with a conversational assistant. Even if both these groups theoretically can beneﬁt from the device, they have to be able to set it up and taught how to use it. If the clinician can offer house calls to patients that need help with technology, this problem can be eliminated.

30 12 Identiﬁed challenges with implementing a voice-controlled system

12.1 Regulatory challenges with a voice-controlled system

When collecting data from a user, all companies have to follow the regulations that the corresponding country has, e.g, within the countries of the European Union (E.U.), GDPR has to be followed. When performing clinical trials, AstraZeneca also has to follow the Clinical Trial Regulation (CTR) within the countries of the E.U. and the Food and Drug Administrations (FDA) guidelines within the U.S. For medical devices, the E.U. has the regulation “Medical device regulation” (MDR), which have to be followed if the system is a medical device. When investigating the possibility to implement voice- control into the patient application, these regulations have to be followed. The identiﬁed challenges connected to these regulations are;

Data storage by external devices • In a conversation between a conversational assistant and a user, the data from the conversation is stored for machine learning purposes. To protect the user’s privacy according to regulations, the user has to give an additional informed consent stating that it is okay for the company hosting the conversational assistant to store the user’s data. If the user can do this on a voluntary basis, the users who want to use the voice-controlled system can sign a privacy agreement with the provider of the voice-controlled system. The users who are not interested in the feature do not have to do anything and no additional forms would be required by AstraZeneca. Another way this issue can be solved is if an agreement can be made between AstraZeneca and the provider of the voice-controlled device/system, stating that no data will be stored. This will prevent the provider from collecting sensitive data and the informed consent does not have to be changed. This can be done on a technical level but will be a question of an agreement. When designing an agreement with the provider of the voice-controlled system, temporary data storage have to be taken into account as well. Temporary data might be stored to prevent data loss in case of a technical failure. The agreement must state how this temporary data storage will be done to prevent any other actors from getting access to sensitive data.

Reporting adverse events to the authorities in time • When AstraZeneca receives information regarding a serious adverse event, e.g, that is life-threatening for the patient, they have 24 hours to report it to the authorities. To get a clinical trial approved, it requires that AstraZeneca has a clear plan on how the events shall be reported and how to access important data regarding the adverse event. Exceptions can be made when investigating a well-known treatment or medication but at new studies with unknown treatments, all information that might be related to the treatment is important. With the risk that important

31 13 Patient recorded outcome (PRO) through voice control

information related to an adverse event can not be accessed in time, the authorities might not approve the clinical trial. A clear strategy on how to handle these events and how to access additional information regarding the event in time is necessary to get a free text/speech system approved by the authorities.

Follow up questions • When a clinical trial is set up, all study material has to be presented for an ethics committee for review. In a conversation between a participant and conversational assistant, follow up questions to the participant will differ regarding the content presented. Is it possible to present every possible answer that the conversational assistant can provide to the ethics committee or do we have to limit the conversation due to the regulations?

Language issues • When performing global clinical trials, participants will have different native languages. All study-data used in clinical trials has to be translated to the participant’s native language to assure that the content is understood and that the patient can complete the content without extra effort. Conversational assistants available have limited language support and could only be used in clinical trials where all participants native language are available.

Authenticating users in a voice-controlled system • In patient-applications, the user has to authenticate themselves to make the data provided valid but also to protect the user’s conﬁdential information. A Data Pro- tection Impact Assessment has to be made to determine the level of risk that a voice-controlled system brings, i.e, are the user’s privacy protected? The authentication method has to be approved by the authorities as well, where AstraZeneca has to prove that it is a secure method. If we would, e.g, use voice biometrics as an authentication method, proof that this method is safe has to be presented for the authorities. A problem with this is that the provider of the biometric solution, might not want to share information regarding how their system works with AstraZeneca to protect their business idea.

13 Patient recorded outcome (PRO) through voice control

One of the main areas where it can be interesting to use voice control is in ePROs, where the patient can have an additional way of completing them and more information can be accessed regarding the treatment and patients safety. Some problems arise which have

32 14 Usability study to be handled before it is possible to implement it into ePROs, these will be discussed in this section together with possible solutions.

13.1 Identiﬁed challenges

ePROs that require other answers than text/voice • Some ePROs used in clinical trials that AstraZeneca performs requires the user to, e.g, paint on a virtual body where the user is in pain. How can this be explained in words? If the user, e.g, have an aching stomach, it is much easier for the user to point to the location of the pain instead of trying to explain it in words which will give a less accurate response.

Translation between predefined answers and free speech • Free text or speech is not used in ePRO surveys yet, all answers are predefined with, e.g, a scale from 1-10 described in 8.5. If users can use free speech instead of these predefined answers, a translation must be done to make the answers equivalent. Another way to do it is to use free text for those who do not use their voice and free speech for those who do, this will leave a more comparable result.

14 Usability study

The usability study had 6 participants, three were male participants and 3 were female. The age range was between 26 and 54 and the participants had a variety in occupation and technical knowledge. Two additional subjects were tried in the trial but they could not complete the trial due to technical failure with Googles voice proﬁles.

14.1 Implementation

Due to technical failure, the application had to be changed and could only simulate that the user’s voice was registered. The participant registered their voice into the Google Home application but without their knowledge, their voice was deleted and the response with their name was “hard-coded” by the interviewer. This had no impact on the result due to the conversation with the assistant, did not change. All interviews were carried out in different directions, letting the participant speak about his/her experience freely. Apart from the user freely expressing themselves, the set- questions were all answered by every participant to get some consistency in the result.

33 14 Usability study

All interviews were recorded with audio and video and the important data were mapped into themes together with important quotations. the result of the captured data is represented in Section 14.2 which shows the key points from the study.

14.2 Result from the captured data

Out of the six participants, 5 of them completed the interaction without any disturbance and one participant did not manage to complete the conversation. The participant that did not manage to complete the test wanted to use longer phrases when communicating with the assistant, e.g, when the assistant asked “What is your level of anxiety today on a scale 1-5?” the participant answered with “I feel pretty down today but let us say three”. The answer “three” were not registered due to the lack of training phrases provided to the program. The result form this interaction and the answers from the interview shows that a natural language communication without disturbance, where the assistant does not understand, can help to motivate the participant in using the conversational assistant. From the usability test, the most important themes were registered together with key insight and will be presented here;

The system felt trustworthy The main conception of the voice-controlled au- • thentication were that it was trustworthy and all the participants trusted that the system could make a difference on voices. Two people felt that it was an unusual way of authentication but expressed “I just need to get used to the authentication model, just like when face recognition ﬁrst came”. Three people also stated that it felt secure since AstraZeneca is a trustworthy company and if they are providing the service, they would feel safe.

“I would rather talk to the assistant than a medic” The general idea was that • the participants would much rather talk to the conversational assistant than to a medic. “It does not feel like the assistant is judging me” was expressed by three different participants, two participants answered “The assistant” when they were asked if they rather talk to the assistant or a person and the last one did not care if it was the assistant or a person.

The assistant answering with full name In the interaction, the assistant wel- • comed the user with his/her full name. Their full names were provided to the application through the interviewee and two people expressed discomfort about this. They also expressed that it would have felt better if they were the ones reg- istering their names. Three people said that it felt more secure when the assistant

34 15 Discussion

answered with their full name, “It was very clear that the assistant knew who I was when it answered with my full name”.

Automatic authentication or external device authentication? Most of the par- • ticipants expressed that they wanted to choose by themselves if they should use automatic authentication by voice or use their phone. Four participants expressed that they would use the automatic model and two participants wanted more information about it before their decision.

“The conversation felt natural” Five participants expressed that the conversa- • tion felt natural and easy while one participant expressed that the conversation has to be better before it is used.

An important thing to point out from this study is that the technology worked. If there was a problem with the technology, the result would probably not have been the same. It is therefore important to make sure that the technology is trustworthy before implementing such a system.

15 Discussion

A big part of this thesis has to be represented as a discussion since the knowledge level of a voice-controlled system is limited. Both AstraZeneca and the authorities have to adapt together with the new technology that is implemented right now and in the future. No solution chapter is provided since the main result of this thesis is a discussion that should be discussed further.

15.1 Free speech handling

Since voice control is a fairly new technology, the authorities do not know how it should be used in clinical trials. This is something that has to be ﬁgured out together with the pharmaceutical companies to keep the participants safe but also to give pharmaceutical companies all the possibilities when investigating medical devices and medication. As stated in this report, there are a lot of positive effects that a voice-controlled system may bring if we only can ﬁgure out a safe way to do it. As it is for now, “free text” or “free speech” is not used. This is due to that AstraZeneca lack the possibility to handle all information that goes into free text space. As stated

35 15 Discussion earlier in this thesis, the user can provide information that is not related to the trial and there are no clear regulations provided from the authorities on how to deal with that. A question that has been raised a lot regarding this project is; “When do AstraZeneca have the information from the user?”. For now, it seems that as soon the user speak- s/writes, AstraZeneca is the owner of that information. This obligates AstraZeneca to take action on it. On the other hand, e.g, Microsoft is not obligated to deal with all information that is passed via email through their servers. Microsoft can, without doubt, get access to the information provided in an email but still, it is okay for them to provide the service without taking action on the data. With this argument, one can say that it should be possible for AstraZeneca to provide free text/speech for the participants in clinical trials, without handling all the information that is not related to the trial. With an IT-service department, it can be possible to limit access to the user’s data, to support-functions. Then the user is the owner of his/her data and can declare who should have access to it. Like with a Google account, e.g, you as a user can give access to a part or all of your data to third-party applications. If this is implemented, maybe it is possible that the user is the owner of all spoken/written data and can grant AstraZeneca access to the part of data that is related to the clinical trial?

15.2 Solving the language problem

A way to overcome the language problems could be to help the provider of the voice- controlled system with their language training. A possible way to do this might be through audio-books, where both text and voice is available. The machine learning al- gorithm can practice the language from texted books together with their audio version and is also able to practice on different voices and accents since there are different read- ers for various books. This could be an interesting way for machine learning algorithms to learn new languages through natural language processing (NLP).

15.3 Removing the risk of miss-communication between patient and virtual assistant

When a patient is communicating with a voice-controlled system, there is always a risk of miss-communication. A possible way to deal with this might be, two-way veriﬁcation where the virtual assistant can repeat what the user said and ask if that was correct. This will also ensure the user that his/her information was processed correctly.

36 16 Evaluation

15.4 Does a voice-controlled system increase expectations

A voice-controlled system is a fairly new way of communicating with a machine, that uses natural language. A problem that might occur when people speak instead of writing their answers, is that the subject expects action from the machine, or in this case AstraZeneca. The way we normally use natural language is when we are speaking to another human. In that conversation, the other person usually responds to what we are saying and take action if necessary. I hypothesise that the expectations of action from answering questions in natural language might exceed the expectations that the patient has when he/she is writing.

16 Evaluation

The main goal of this thesis was to investigate the possibilities and obstacles with a voice-controlled system for patients in clinical trials. Overall, the thesis has presented both positive and negative outcomes from a voice-controlled system and also some solutions to the challenges. This thesis can not be seen as a guide on how to implement a voice-controlled system but as a pilot study for further investigation. For the sub-goals that builds up to the main goal, an evaluation is represented for each goal separately:

What possibilities come with implementing voice control? This part of the • goal is fulﬁlled in this thesis where a lot of positive features of a voice-controlled system is represented. Target groups are represented together with the impact that a voice-controlled system might have on usability. The question “should we replace the current system with a voice-controlled system?” is discussed and answered with a clear basis.

What can we do in a technical aspect? The goal of answering this question has • not been fully fulﬁlled. Some technical limitations are represented together with some technical possibilities for the future. The goal was to represent more data on this subject, with a clear plan on how to take this project forward.

What are we allowed to do in a regulatory aspect? Regulatory aspects and • limitations are focused on and stated in the discussion part of this thesis. Either AstraZeneca or the authorities know how to handle a free text/speech system since it has never been done before. The ultimate goal of this thesis was to state exactly what can be done and what can not be done. Even if that goal was not reached, I think that this thesis still provides an important discussion about the challenges and possibilities of the problem, that AstraZeneca can keep working on.

37 18 Future work

17 Conclusion

The beneﬁts of a voice-controlled system can be huge in terms of reaching a larger target group and satisfying more people. This can, in turn, result in more accurate results and a lower drop-out rate from clinical trials. As the study shows, people are open for an automatic authentication model, if they have more control and knows more about how the assistant knows who they are. These two results is a clear identiﬁcation of the need for a voice-controlled system. From the usability study made in this thesis, it is clear that the technology that is available today is not reliable enough for a voice-based authentication. When it works, it works great and it is a smooth way of authentication but other technologies have to be implemented for the conversational assistant to be reliable. For the regulatory aspects of the project, it is very clear that no one knows anything about this. Both voice-control and free text in clinical trial assessments is an uncharted area and have to be explored with ease together with the authorities. Implementing a voice-controlled or free-text system will not be approved as it is, for now, it will need to be approved small bits at a time to show the impact.

18 Future work

This thesis should be seen as a pilot study for future work and this section will represent my recommendations and the path I would follow to investigate this further.

18.1 Recommendations for AstraZeneca

Solving the free text/speech problem A free text/speech ﬁeld into clinical tri- • als can be a huge advantage for both usability and medical knowledge. Solving this issue with the authorities is, therefore, my priority recommendation as future work. Even if a voice-controlled system never is implemented, free text can ease the clinical trial process for the participants and will in return give more accurate results from clinical trials. A discussion regarding this subject is provided in Section 15.1.

Start testing with free text There are several obstacles to overcome before a • voice-controlled system can be implemented into clinical trials. To not overwhelm the authorities, it can be an idea to start a trial with just a few questions that have

38 18 Future work

free-text responses. These questions should not be crucial to the trial, just be used as an evaluation together with the corresponding authorities. Further testing on a voice-controlled system can be limited to, e.g, one region to overcome the language translation issue. Also, a voice-controlled system can be tested at the beginning with only ﬁxed responses accepted.

Find and start working with a provider of a voice-controlled assistant If a • voice-controlled system will ever be implemented into clinical trials, the language translation issue has to be solved. My recommendation is to start a collaboration together with a provider of a voice-controlled assistant. A collaboration might motivate the provider to start working with more languages and implement important features that a voice-controlled system need.

39 References

References

[1] “Intelligent virtual assistants: Going beyond chatbots,” https://haptik.ai/blog/ intelligent-virtual-assistants/, accessed: 12-02-2020. [2] “What is usability?” https://www.interaction-design.org/literature/topics/ usability/, accessed: 17-02-2020. [3] “The medical product agency’s provisions and guidelines on clinical trials of medicinal products for human use,” https://lakemedelsverket.se/upload/eng-mpa- se/lvfse/LVFS2003 6.pdf, 2003, accessed: 14-02-2020. [4] I. T. Agaku, A. O. Adisa, O. A. Ayo-Yusuf, and G. N. Connolly, “Concern about security and privacy, and perceived control over collection and use of health information are related to withholding of health information from healthcare providers,” Journal of the American Medical Informatics Association, vol. 21, no. 2, pp. 374–378, 08 2013. [Online]. Available: https://doi.org/10.1136/amiajnl-2013-002079 [5] The new sound of home. Apple. Accessed 2020-05-03. [Online]. Available: https://www.apple.com/homepod/ [6] AstraZeneca, “Astrazeneca to present comprehensive data from its diabetes portfolio and pre-clinical research at easd 2015,” astrazeneca.com, 2015. [On- line]. Available: https://www.astrazeneca.com/media-centre/press-releases/2015/ astrazeneca-diabetes-data-pre-clinical-research-easd-2015-11092015.html# [7] T. W. Bickmore, H. Trinhl, S. Olafsson, T. K. O’Leary, R. Asadi, N. M. Rickles, and R. Cruz, “Patient and consumer safety risks when using conversational assistants for medical information: An observational study of siri, alexa, and google assistant. j med internet res 2018;20(9):e11510 doi: 10.2196/11510 pmid: 30181110 pmcid: 6231817,” https://www.jmir.org/2018/9/e11510, accessed: 02-03-2020. [8] N. cancer institute. (2017) Common terminology criteria for adverse events (ctcae). National cancer institute. Accessed 10-02-2020. [Online]. Avail- able: https://ctep.cancer.gov/protocolDevelopment/electronic applications/docs/ CTCAE v5 Quick Reference 8.5x11.pdf [9] EMA. (2020) Glossary, serious adverse reaction. European medicines agency. Accessed 17-02-2020. [Online]. Available: https://www.ema.europa.eu/en/ glossary/serious-adverse-reaction [10] E. J. EZEKIEL, The Solution to Drug Prices, 2015. [Online]. Available: https://www.nytimes.com/2015/09/09/opinion/the-solution-to-drug-prices.html

40 References

[11] FDA. (2018) Step 3: Clinical research. Food and Drug Administration. Accessed 12-02-2020. [Online]. Available: https://www.fda.gov/patients/drug- development-process/step-3-clinical-research

[12] M. Gail A. Van Norman, “Drugs and devices,” 2016. [Online]. Available: https://reader.elsevier.com/reader/sd/pii/S2452302X16300638?token= FED41B12744E853046E53652BD5A49587D0459B274065654495E7B37AA75750FFA6B3D756B234A52CEB24698B0D9E528

[13] C. Gao, “Use new alexa emotions and speaking styles to create a more natural and intuitive voice experience,” https://developer.amazon.com/en-US/blogs/alexa/ alexa-skills-kit/2019/11/new-alexa-emotions-and-speaking-styles, Amazon, 09 2019.

[14] “Google developers,” https://developers.google.com/assistant, Google, accessed 2020-05-03.

[15] J. G. . B. Goransson,¨ “Key principles for user-centered systems design,” http: //www.it.uu.se/research/hci/acsd/KeyPrinciplesPoster-v.1.2en.pdf, accessed: 26- 05-2020.

[16] D. Hawkins, “The cybersecurity 202: Why a privacy law like gdpr would be a tough sell in the u.s.” 2018. [Online]. Available: https://www.washingtonpost.com/news/powerpost/paloma/the-cybersecurity- 202/2018/05/25/the-cybersecurity-202-why-a-privacy-law-like-gdpr-would-be- a-tough-sell-in-the-u-s/5b07038b1b326b492dd07e83/

[17] HHS. (1996) Health insurance portability and accountability act (hipaa). U.S. Department of Health Human Services. Accessed 18-02-2020. [Online]. Available: https://www.hhs.gov/hipaa/index.html

[18] M. Hosely, “Retention in clinical trials – keeping patients on protocols,” 2020. [Online]. Available: https://forteresearch.com/news/infographic/infographic- retention-in-clinical-trials-keeping-patients-on-protocols/

[19] “ISO/IEC 9241-11:2018 - usability - ergonomics of human-system interaction,” https://www.iso.org/standard/63500.html, International Organization for Stan- dardization, Geneva, Switzerland., accessed: 17-02-2020.

[20] A. G. Levy, A. M. Scherer, B. J. Zikmund-Fisher, K. Larkin, G. D. Barnes, and A. Fagerlin, “Prevalence of and Factors Associated With Patient Nondisclosure of Medically Relevant Information to Clinicians,” JAMA Network Open, vol. 1, no. 7, pp. e185 293–e185 293, 11 2018. [Online]. Available: https://doi.org/10.1001/jamanetworkopen.2018.5293

41 References

[21] Y. Liao, J. Vitak, P. Kumar, M. Zimmer, and K. Kritikos, “Understanding the role of privacy and trust in intelligent personal assistant adoption,” 2019. [Online]. Available: https://pearl.umd.edu/wp-content/uploads/2019/01/ Liao etal-2019-iconference.pdf

[22] M. Mittal, “Personalize your alexa experience with voice proﬁles (generally available),” https://developer.amazon.com/blogs/alexa/post/1ad16e9b-4f52-4e68- 9187-ec2e93faae55/recognize-voices-and-personalize-your-skills, Amazon, 10 2019.

[23] PatchAi, “The ﬁrst empathic virtual assistent for patients enrolled in clinical trials,” website: patchai.io, January 2020, retrived January 26, 2020 from https://patchai. io/en/.

[24] T. Phillips, “What is a susar in clinical trials?” 2020. [Online]. Available: https://www.thebalance.com/what-is-a-susar-375698

[25] E. union, “Regulation (eu) 2016/679 of the european parliament and of the council on protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (general data protection regulation),” 2014.

[26] ——, “Regulation (eu) no 536/2014 of the european parliament and of the council on clinical trials on medicinal products for human use, and repealing directive 2001/20/ec,” 2014.

[27] A. Walter, The Uphill Path to Successful Clinical Trials, 2013. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3684189/#b1-ptj3804225

[28] ——, “The uphill path to successful clinical trials,” PT, 2013. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3684189/#b1-ptj3804225