User Recommendations for Intelligent Personal Assistants

Prepublication version:

Lopatovska, I., Griffin, A., Gallagher, K., Ballingall, C., Rock, C., and Velazquez, M. (2019). User recommendations for intelligent personal assistants. Journal of Librarianship and Information Science. USER RECOMMENDATIONS FOR INTELLIGENT PERSONAL ASSISTANTS

Abstract Adoption of intelligent personal assistants (IPA) is on the rise. Published studies on IPAs often focus on the analysis and critique of existing IPA features without understanding specific user needs that the technology aims to address. We present an exploratory study that gathered user recommendations for the design of their ideal IPA. The study relied on focus group and content analysis methods for data collection and analysis. Major themes in participants’ recommendations for IPA design were identified and included feature improvements (e.g. speech recognition, input/output modalities, device feedback); customizability and increased control over IPA features and functions; transparency and understanding of IPA hardware, software, and data management processes; personification; compatibility with third-party platforms; accessibility; and aesthetics. Many of these recommendations are rooted in basic user experience design principles and have been previously discussed in the context of PDAs (personal digital assistants) and other technology. Addressing these recommendations would advance IPA technology and improve user experiences with it. Keywords Intelligent personal assistants, conversational agents, smart speakers, digital personal assistants, technology adoption, user recommendations, design principles

1. Introduction

Many companies attempt to use conversational technology to extend and enhance their brands and support “two-way relationships” with consumers (Wilson et al., 2017). Intelligent personal assistants (IPA) (also known as conversational agents, smart speakers, digital/intelligent personal assistants, or voice-controlled agents) are designed to accept spoken or typed input, answer queries in a natural language, or perform tasks that might include presenting search results, playing music, placing online shopping orders, managing a calendar, or controlling Internet of Things (IoT) devices (Canbek and Mutlu, 2016). The main conversational agents on today’s market include Apple’s Siri, Amazon Alexa, Google Assistant, and Microsoft Cortana. Based on a survey conducted in May 2018, 54.4 million people, or 21.6 percent of U.S. adults own a smart speaker (Kinsella and Mutchler, 2018). Amazon holds most of the smart speaker market share at

61.9 percent, with Google at 26.9 percent, Apple at 4.1 percent, Sonos at 3.8 percent, and others holding the remaining 3.4 percent (Kinsella, 2018).

With the growing rates of IPA adoption, the number of published IPA studies is increasing as well. However, the studies often focus on analysis and critique of existing IPA features without understanding specific user needs that the technology aims to address. In an environment where companies’ market research data are not available, and the user needs for IPAs can only be inferred from advertisements, the benefits of IPAs for companies are often clearer than the benefits of IPAs to consumers. Our own prior research shows that users often question the benefits of IPAs and struggle to justify their use (citation removed for review)While rising IPA sales and integration of conversational technology and IPA features into IoTs, cars, and other devices point to the irreversible trend of IPA adoption, published studies suggest that users’ reactions to this technology are lukewarm. In order to make this emerging technology more palatable to users, we need to understand user needs that are currently addressed, not addressed, and could be addressed by future IPAs.

We conducted a study to investigate user needs and recommendations for the design of their ideal IPA based on their existing and potential experiences with this technology.

2. Literature review

2.1 IPA features IPAs vary by input and output modalities (voice and/or touch), hardware requirements, and functionality (summarized in Appendix A). IPAs can be accessed via platforms or applications on multipurpose devices (phones, computers, iPads, etc.) or on standalone devices like Amazon Echo, Apple HomePod, or Google Home, which are purchased separately. A number of authors discuss the strengths and weaknesses of IPAs on specific tasks. For example, not surprisingly, one of Amazon Alexa’s strengths includes support for voice-activated purchasing from Amazon’s website (Crist, 2016). Amazon Alexa links to major audio services (e.g. Spotify Premium, Pandora, or Amazon Music), which contributes to its success in satisfying users’ music requests. Amazon’s partnership with IoT companies allows Alexa to perform a wide variety of smart home tasks, including climate control and lighting (Thompson, 2017; Villas-Boas, 2017). While Google Home is more expensive than Amazon Echo, it comes with the large Google platform that appeals to the many users of Google’s search engine and other products (Ranj, 2018). Google Assistant, powered by almost two decades of web searching experience, is stronger than other IPAs in answering informational questions (Betters and Grabham, 2018). While Apple’s Siri is limited in music-playing and smart home capabilities, it uses an algorithm that allows Siri to make suggestions based on analysis of user interactions; it also appeals to Apple loyalists (Betters and Grabham, 2018). Due to its integration with the large suite of Microsoft projects, Microsoft Cortana has been found to excel in task reminders, calendars, and communication support (Graus et al., 2016). Recently, Chen (2018) compared the performance of Google Home, Amazon Echo, and Apple’s

HomePod on tasks related to music, navigation, productivity, cooking, home automation, and others. The author found the overall performance of the three IPAs satisfactory and comparable, with Siri underperforming on certain specific tasks, such as ordering a car through the Uber app and setting up a meeting.

2.2 User interactions with IPAs While some publications review IPA features and performance, others focus on users’ interactions with IPAs, including user expectations, concerns, and factors attributing to their satisfaction with this technology.

Several studies emphasized the lack of user understanding of IPA technology and false expectations about its features and capabilities. Luger and Sellen (2016) and Cowan et al. (2017) reported that users often do not know how to interact with IPAs, resulting in interactions that test the system’s capabilities or cause the user to feel overwhelmed. Bopp (2018) observed that first-time users of an Amazon Echo who had been given no instructions on how to work with the device were more likely to be dissatisfied. Additionally, a third of the participants who had received instructions wanted more guidance (Bopp, 2018).

False expectations of IPA intelligence and capabilities are often fueled by the IPA’s conversational interface and “personality” characteristics (e.g., name, gender, response style). In a study by Luger and Sellen (2016), one participant referred to Siri as “sassy” and another thought Siri was being sarcastic when responding to prompts. Other studies found that users thank the device (Kiseleva et al., 2016), or do not want to hurt the IPA’s feelings (Cowan et al. 2017). Such anthropomorphic features afford users to interact with the IPA as if it were a human (Purington et al., 2017) and these human-like features are tied to the continuing success of IPA adoption (Han and Yang, 2018). However, current technology is not perfect: long, sustained, coherent, on-topic conversations with IPAs, resembling human conversational patterns, are not supported (Guo et al., 2017), leading to frustration, disappointment, and discontinued use (Bopp, 2018; Cowan et al., 2017; Kiseleva et al., 2016; Luger and Sellen 2016).

The conversational nature of IPAs also leads to challenges of integrating them into public settings. Lopatovska and Oropeza (2018) examined Alexa usage in a public space of an academic department. The authors found that only a small percent of people who used the space chose to engage with the IPA. The most cited reasons for disengagement included lack of need or awareness, reluctance to distract others and privacy concerns. Easwara Moorthy and Vu (2015), who explored social concerns around the use of voice-controlled assistants, found that people tended to avoid using voice input in public settings due to privacy concerns. Similar findings were obtained in a study of smartwatch users, where watch owners avoided using voice commands because they were concerned this was not socially acceptable behavior (Efthymiou and Halvey, 2016). Porcheron et al. (2017) examined a case of Amazon Echo use in a social setting and noted that even when a group of users was willing to engage with IPAs, they still chose to use the IPA primarily for information seeking.

Despite their limitations, voice interfaces are valuable in supporting hands-free interactions. Pradhan et al. (2018) examined Amazon Echo adoption by disabled and, specifically, visually impaired users and found that this hands-free technology helps users to mitigate some of the accessibility challenges as well as provides some unexpected support for speech therapy and other daily needs. Participants in the Cowan et al. (2017) study reported the need for IPAs during “hand/eye busy” situations, such as driving, playing with children, and cooking. However, a mismatch between input and output modes (e.g., a spoken command that produces screen text) leads to frustration, disappointment, and, often, abandonment of the IPA (Luger and Sellen, 2016). Users’ preference for hands-free interfaces does not apply to all situations, however. Users distrust voice commands for tasks that require precision, such as sending text messages and e-mails or making phone calls to personal contacts, and expect visual feedback before completing the task (Cowan et al., 2017; Luger and Sellen 2016).

While users distrust IPAs to perform certain tasks in the situations mentioned above, users will trust IPAs in other contexts without questioning the quality of the information or services they are receiving. Lei et al. (2017) and Kreuser (2018) reviewed companies’ practices of marketing their products through IPAs. For example, Tide’s Stain Remover skill on Amazon Alexa provides step-by-step instructions on how to clean tough stains; Campbell’s Kitchen provides recipes and shopping lists; Zyrtec, has an application that factors in the weather in order to give users allergy-related advice. Without clear reference to the information source, many users do not question the quality and intentionality of the information they receive from IPAs. One of the more “vulnerable” groups of users include children. Botsman (2017) described a study conducted with three- to ten-year-old children by M.I.T. Media Lab. The study found that almost all the children had complete trust in Alexa. “Some children believed they could teach the devices something useful, like how to make a paper plane, suggesting they felt a genuine, give-and-take relationship with the machines” (Botsman, 2017: 4).

Distrust in IPAs is directly linked to privacy and security concerns, which is discussed by several researchers. Brenner (2017) clarified privacy risks and outlined defensive measures to combat some of the security issues with IPAs, specifically Amazon Alexa. The author identified the recording of users’ private conversations as a recurrent concern among IPA users. Another privacy risk discussed by researchers is the possibility of an attack on IoT devices linked to IPAs. Lei et al. (2017) detailed security threats specific to Amazon Alexa and explained ways in which Alexa can aid in home burglaries by disabling alarms or placing unintentional online shopping orders. Alhadlaq et al. (2017) suggested that a possible solution to users’ privacy concerns would be for IPAs and any connecting third-party applications to offer users clear privacy policies and practices.

Several studies focused on the usability issues of current IPAs. Lopatovska and Oropeza (2018) reported a high percent of user utterances that is not addressed by the IPAs. In some of these instances, the IPAs admitted their limitations, provided wrong response or remained unresponsive. Lopatovska et al. (2018a) found that users frequently reported satisfaction with IPAs, even on the failed tasks. Kiseleva et al. (2016) found a negative

4 correlation between user satisfaction with IPAs and task complexity and the effort needed to complete the task. User satisfaction was also negatively correlated with the user having to use multiple modes (voice, text, gesture) to complete the task (Kiseleva et al., 2016). Additional causes of dissatisfaction have been linked to lack of feedback from the device on the status of task completion (Luger and Sellen, 2016; Sörenson, 2017) and poor system accuracy in the area of speech recognition, which was noted in a survey by Moore et al. (2016). A possible solution to improving IPA usability is presented in Santos et al., 2018. The authors propose improving IPA performance by autonomously collecting contextual user data from the IoT devices. While the authors discuss the technical challenges of such an undertaking, the ethical implications of the solution are not discussed.

Orehovački et al. (2018) attempted to classify IPA attributes by giving undergraduate student participants a number of IPA tasks. Upon task completion, participants evaluated “perceived qualities” of the IPAs that researchers then grouped into essential, sufficient, desired, optional, or not relevant categories. Essential IPA attributes included qualities such as ease of use, satisfaction, controllability, accuracy, and efficiency. And desired IPA attributes included qualities such as manageability, aesthetics, reliability, and error prevention The IPA qualities categorized into these groups were informed by a previous study by Orehovački et al. (2013), which proposed these attributes for evaluating Web 2.0 applications such as social networking sites, blogs, or wikis.

Overall, while industry and promotional publications focus on the myriad of IPA features and functions (Enge, 2018; Mills, 2018), researchers point to the limitations in IPA transparency, speech features, and performance. Most of the reviewed studies focused on existing IPA features and users’ reactive behaviors with IPAs. Our study took a more proactive stance in trying to gauge user needs in IPAs, how they imagine their ideal IPAs and interactions with them.

3. Methods

In order to gain an understanding of user needs and design preferences in regards to IPAs, we developed an exploratory study to answer the question: What are the user recommendations for the design of IPAs based on their existing and potential IPA uses? The data to address the research question were collected using the focus group method. The focus group method relies on a guided group conversation about a given topic or topics. It is widely used in market research, branding, and new product development contexts (Greenbaum, 2000) and has a long history in the social sciences (Wilkinson, 2010). Within information disciplines, it is frequently used for studying user behavior, expectations, and preferences related to information technology (Hoseth and McClure, 2012; Seeholzer and Salem, 2011), as well as for new product development (Paterson and Low, 2011). For example, one of the frequently cited recent studies of technology adoption used focus groups to investigate students’ use of cellphones at the University of

Edinburgh (Paterson and Low, 2011). Researchers used focus groups in conjunction with a survey through which participants were recruited. Participants’ feedback served as data to inform future cellphone library services (Paterson and Low, 2011). Similar studies that applied focus groups to examine technology uses have been reported by Seeholzer and Salem (2011) and Hoseth and McClure (2012). One of the studies on IPAs cited earlier (Cowan et al., 2017) applied the focus group method to investigate infrequent IPA users’ views, practices, and barriers to use of IPA technology. We employed the focus group method in our study due to its potential to produce rich quantitative data on user experiences, opinions, and interactions with technology (Wilkinson, 2010).

The study recruited 18 participants with prior experiences with IPAs. These 13 female and 5 male participants were recruited primarily from the community of the Pratt Institute School of Information students and alumni. The study was promoted on the institution’s mailing lists, social media channels, campus posters, and visual displays. Potential participants were asked to fill out a brief online questionnaire about their IPA usage in order to determine their eligibility for the study. Thirteen participants identified themselves as students. The participants’ ages ranged from 18 to 55 years of age, with five participants ages 18-25, seven participants ages 26-35, four participants 36 to 45 and two participants 46-55 years of age. Based on participant responses we also inferred that at least one participant was a faculty member, three participants had advanced understanding of IPA technology and/or used its advanced features (we refer to these participants as tech-savvy), and several participants had international accents.

The data were collected during five mini focus groups with three to four participants per group. A smaller group format was chosen in order to establish participants’ trust, stimulate open conversations and ensure greater control of the conversation by the moderator (Flores Letelier et al., 2000). The mini groups were run over the course of three weeks in March 2018 and were offered on different days of the week and different times of the day to accommodate participants’ schedules. Each session lasted between 45 to 60 minutes. Light snacks were offered during each session. At the end of the session, participants received a $10.50 transportation card for their participation.

The focus groups followed a discussion guide, a required instrument for guiding the moderator’s actions, content of the questions, and timing of the main discussion points (Greenbaum, 2000). The first part of the discussion guide included an introductory script and administrative details for the moderator. As part of the introduction, the moderator played an excerpt from an online Amazon Alexa commercial on a projector screen in order to refresh participants’ recollections of the main functions of IPAs. The subsequent discussion was organized around the research question and included conversations about participants’ current experiences with IPAs and features that they would like to improve, change, or add to existing IPAs. Each conversation started with a question to stimulate a discussion among participants. The guide then included further probing questions and examples that the moderator could use to encourage discussion. At the end of the discussion, participants were asked to provide a summary of their main recommendations

6 to product designers or the companies’ CEOs. The focus group guide was pilot tested to ensure instrument quality and provide training for the moderators.

The focus groups included a researcher-moderator who facilitated conversations among participants, and a note-taker. In addition to the notes taken during focus groups, the discussions were audio recorded. Audio recordings were partially transcribed to enhance the written notes that served as the primary source of data for answering the research question. The text of the notes was open coded to identify the main themes in participants’ conversations. Early in the data analysis process, a sub-set of data were coded by four researchers and discussed, the coding book with code definitions and relationships was developed and used for coding the remainder of the data. The consistency of the final coding was validated by two additional researchers on a randomly generated sub-set of qualitative data, resulting in the > .8 Fleiss Kappa coefficient.

The study followed general guidelines for focus group research (Greenbaum, 2000; Wilkinson, 2010) and was approved by the IRB.

4. Results and Discussion

In order to understand participants’ background and usage of IPAs, participants were asked which IPAs they use and what they use them for. Participants reported using Amazon Alexa, Apple’s Siri, Google Assistant (Home), and Microsoft Cortana, either exclusively or interchangeably. Alexa and Siri were the most frequently used in conjunction (reported by 3 participants), followed by Google/Siri (2) and Google/Alexa and Cortana/Alexa (1). Four participants reported that they own a smart speaker or regularly interact with a smart speaker (4). Thirteen participants discussed using an IPA on their personal devices (iPad, laptop, smartwatch, and mobile phone) (13). Six participants indicated using IPAs primarily in their homes (6), a finding that is well- supported by prior studies which reported that IPAs are used infrequently in social and public settings (Efthymiou and Halvey, 2016; Lopatovska and Oropeza, 2018; Porcheron et al., 2017). Three participants reported using their IPAs primarily on-the-go (3), while two participants indicated primarily using IPAs for texting and directions while driving (2).

The most frequent uses for IPAs across all platforms were for weather inquiries (9 mentions) and simple reference questions (8) ranging from locations of cultural institutions to vocabulary definitions. The high frequency of such queries supports previous findings that weather reports are one of the most commonly requested pieces of information from IPAs (Lopatovska et al., 2018). Possible reasons for the high frequency of this type of query is its simplicity and the ease of accomplishing it without much effort, which is in line with Kiseleva et al.’s (2016) findings related to participant satisfaction on simple IPA tasks. Weather and simple reference inquiries are also well- suited for the hands-free searches supported by IPAs and can be performed in parallel

7 with other activities (e.g. choosing the right clothes for the day). Participants did not mention having any issue with weather inquiries, aside from one user reporting that Alexa was not able to provide the weather information in Celsius instead of the default Fahrenheit.

Four participants reported using their IPA frequently for music requests, which has been reported in prior studies as a high frequency request. This is attributed to the accessibility of IPAs on multiple platforms (phones, computers, tablets, smart speakers, etc.), the minimal effort required to perform the task, and the hands-free nature of IPAs which can support tasks, such as putting on music, that are often performed in parallel with other activities (Cowan et al., 2017; Kiseleva et al., 2016; Lopatovska et al., 2018).

Additional tasks mentioned by participants included programming the device to control household items like lights in a room (3) or the television (3); set timers (3); send messages (4); and look up files on their computers (2). Participants reported using both voice (15) and type (2) input for controlling their IPAs.

Overall, IPA tasks reported by our participants are not unique and are in line with prior observations about IPA usage.

4.1 User recommendations for the design of IPAs based on their existing and potential IPA uses During the focus group conversations with participants, their recommendations for IPA design were closely intertwined with accounts of their current experiences, recollections of successful and not successful interactions with IPAs. Analysis of the participants’ accounts and recommendations resulted in the following six broad categories of design recommendations (Appendix B): 1. Feature improvements: proposed improvements to existing IPA features, such as speech recognition, input/output modalities, device feedback, as well as addition of new features 2. Customizability: user requirements for increased control over IPA features and functions, ranging from speech speed to search parameters 3. Transparency: a group of recommendations pertaining to users’ desire to understand the IPA hardware, software, and data management processes 4. Personification: users’ preferences for the increase or decrease of human-like IPA characteristics 5. Compatibility / accessibility: preference for linking IPAs to additional platforms and applications and expanding it to new user groups 6. Aesthetics: requirements related to the aesthetic design features of stand-alone IPAs.

4.1.1 Feature recommendations The most frequently mentioned recommendations were related to the improvements and additions to existing IPA features (78 mentions, see Appendix B for the summary of themes). One of the most prominent sub-themes within the cluster was improvement to

8 speech recognition and language control features. Several international participants expressed disappointment at the inability of IPAs to understand accents or “speak” other languages. This concern has been discussed in prior studies (Cowan et al., 2017; Dizon, 2017) and humorously illustrated by a Wired magazine video displaying people with a variety of accents attempting to connect with three major IPAs (Calore, 2017). Improving languages and speech recognition is considered a key step in improving the technology (Santos et al., 2018). However, at the time of this study, speech recognition was still a large concern and one of the top user requirements for IPA design.

Additional speech-related suggestions included the ability to maintain conversations with IPAs. Evidence suggests that Amazon has considered users’ desires for conversations by recently integrating Follow-Up Mode (Dickey, 2018), a feature that keeps Alexa active for a few seconds after completing a command in case a user wants to ask a follow up question. While this is a step in the right direction, it does not seem to have all the conversational features users want, as the device does not vocalize a prompt to further conversation.

The recommendation for the device to produce audio, visual, or tactile feedback during interaction sessions is another example of an IPA requirement that has been previously identified (Luger and Sellen, 2016; Sörenson, 2017) but not addressed. As stated by one participant:

“I asked Alexa a question, realized she was unresponsive [...] It should have a resting light or I am alive” [P17]

The importance of device feedback is one of the fundamental design requirements expressed in such classic texts as Jakob Nielsen’s 10 Heuristics for User Interface Design: “The system should always keep users informed about what is going on, through appropriate feedback within reasonable time” (1995: n.p). A highly visible sign of what is happening with their device is exactly what the participants in this study were looking for.

Participants’ preferences for multi-modal interactions, availability of multiple input and output modes, and capacity to switch between them depending on the context (e.g. give audio responses when the user is driving) have also been previously documented. For example, two decades ago in the context of PDAs (Personal Digital Assistants like Apple Newton and IBM Simon), John de Vet and Vincent Buil (1999) discussed how it would be useful to access the music application by the stylus and then receive a voice response. Some IPAs on multi-functional devices (e.g. smartphone, tablet) are currently giving users options to read or listen to the response. However, they largely still lack contextual awareness that would offer the appropriate input and output mode to the user. Other smart speakers are lacking the multimodal interface by design. Here are a few quotes

9 from our participants to illustrate current IPA limitations for multi-modal contextual support:

“I think the way it’s being used is not the right way[…] for instance [if] the person cannot reach out to a screen like how can he or she do that? So, I think there are many other ways which personal assistants… can be used.” [P04]

“Making it more hands-free because if I ask the weather Siri does not read it to me it just goes ‘okay, here’s the weather,’ and I still have to walk over to my phone and check.” [P03]

“I have a big desire to see and touch and get information that way and an audio-based one just seems so nebulous.” [P09]

In their study of IPA usage by people with visual impairments, Pradhan et al. (2018) found that users wanted different modes of communicating with the device such as remote, smartphone, smartwatch, gestures, and touching the device. An interviewee in the Pradhan et al. (2018) study mentioned not wanting to yell at the device when it was far away, a problem echoed in a comment by one of the participants in our study:

“Perhaps, some type of remote mic that you can move around. Alexa is in the living room. I don't want to scream” [P17]

The Amazon Echo Show and Amazon Echo Dot do give users the ability to use a touchscreen, make video calls, and check security camera feeds, but technology reviewers have mixed feelings about these devices (Crist, 2017; Pierce, 2017) and none of the participants in our study mentioned that they were aware of these touch-enabled features.

IPA designers should consider innovative multimodal interfaces for supporting user interactions with IPAs, but there is still little research in the area of human-computer interaction on how to create more natural, multimodal user interfaces (Munteanu et al., 2017). For example, in the context of autonomous vehicles Hastie et al. (2017) revealed a challenge to developing a successful multimodal interface. The study found that when the vehicle’s interface had several modes of interaction, and the users understood how to use these modes, they trusted the device more. However, the authors also found that too many input and output possibilities could be overwhelming to the user (Hastie et al.,

2017). The findings suggest that developing the right balance of multi-modal options in IPAs would present a challenge.

Participants expressed desire for additional features related to the ability to link IPAs to additional IoT home devices, enabling GPS, improving information sources/quality of retrieved information, and offering a recommendation feature. Some of these suggestions have been noted in prior studies (Santos et al., 2018) and should be considered in future versions of IPAs.

4.1.2 Customizability The second most prominent theme in participants’ recommendations for IPA design was the ability to customize their IPAs. Several of our participants expressed the desire to have more control over the search function and sources of information:

“As far as searching: I would like to have something that would be like ‘What are your parameters of the search? What are your keywords?’ Being able to use that in a way to narrow down searches[…] Like having a research assistant in my phone.” [P03]

“I’d be more interested using it not for how it could connect to other devices, but more information-based things[…] I still would rather see with my eyes the date the article was written, who it was written by, what the design of the website looks like -- how valid is this website?” [P09]

Prevalence of such search-related comments in our study could be attributed to the characteristics of our participants, many of whom were iSchool students and faculty or required to conduct research as part of their work. While more work is needed to understand if preference for customizable information retrieval options is shared by the average IPA user, it is worth noting that such preference was strongly expressed within an academic context and should be considered by IPA designers at least for the academic market segment.

Desire to change voice settings or speech speed was also mentioned by our participants and reported in prior studies. This is a significant recommendation because users may interpret a slow IPA response time as a failure (Porcheron et al., 2018). Some work has been done on changing speed/speech settings (Bragg et al., 2018), which would make IPAs more attuned to users’ speech patterns, but more work is still needed.

Closely related to speech customization is the desire to change IPA personality, the way the IPA presents itself via vocal characteristics (gender, accent, speed, language) and characteristics of speech content (types of responses, personification, jokes).

“If it’s possible to change mood[…] if I am a sarcastic person I want her answers in the same sarcastic way or if I am a serious person I want her answers to be in a very academic and serious way.” [P08]

Customizability is an important aspect of IPAs for users (Orehovački et al., 2018) and can affect how a user feels about an interaction with an IPA (Cowan et al., 2017). Luger and Sellen (2016) discussed how the personality-like aspects of IPAs, such as humor, set up unrealistic expectations for current IPA capabilities, since users expect more from a human-like interaction. Perhaps personality customization could lessen disappointment with IPAs because it would make them more of a tool than a preconfigured human-like entity.

The prevalence and diversity of customization requirements in our participants’ comments point to the need for the adoptive customization design approach in which “standard goods or services… can easily be tailored, modified, or reconfigured to suit each customer’s needs” (Gilmore and Pine, 1997: n.p.).

4.1.3 Device transparency Within the broader theme of device transparency, the most commonly discussed requirement was the ability to have a better understanding of how IPAs work:

“[IPAs] seem ‘nebulous’” [P09] “[I want to have a] better understanding of how it's doing what it's doing" [P09]

Discussion of the uncertainty of how IPAs work was followed closely by a discussion about how to use IPAs:

“I don’t know all of Alexa’s features and am unaware of how Alexa would respond.” [P01]

“Unless you’re a tech geek or looked into it, you’re not going to know those things, you’re going to be sticking with the basics of ‘play music,’ ‘check weather,’ ‘what’s the time’…” [P03]

Our participants have linked the lack of IPA understanding to dissatisfactory and disempowering feelings.

The need for IPA transparency has been previously expressed by Luger and Sellen (2016) and Cowan et al. (2017). Additionally, Orehovački et al. (2018) found that learnability and understandability, two aspects of a transparent system, are important attributes for users when determining the quality of an IPA. Providing “visible or easily retrievable” instructions constitutes one of the fundamental design principles (Nielsen, 1995: n.p) that has to be addressed in the designs of IPAs.

Privacy concerns were often voiced in the same context as the desire for more transparency:

“I’m actually a fan of some of the things that are going on and some of the systems[…] It’s more just the idea that there’s so much happening under the cover that we’re not agreeing to on the face of it[…] If it really did function as something where it didn’t record information until I gave the hail word[…] if it was going into an actual off-state -- as opposed to linking to Amazon repeatedly, several times per minute, and possibly recording every conversation -- and going on and deleting it, whatever its functionality is, I’d feel more safe using it and investigating the other applications of it.” [P10]

One participant mentioned reading an article about how often smart home devices were “communicating with the hub” and Amazon Alexa was the most “chattery.” This was a “cause for pause” because she didn’t know what was being sent to or from the hub:

“I would love to see more transparency in the data collection… what’s actually being collected?... And the times when it’s on and off? All those security issues that I think people in the know are aware of, but it’s a big question mark. It would be extremely ethical if they were to step forward and offer that information.” [P10]

Individuals expressed distress that they did not know when IPAs were “listening”; what data were being collected from them; and where that data was going or being kept. Most of the participants acknowledged that they were aware of data mining and “always listening” devices, and this knowledge sometimes even negated their interest in IPAs. The theme of privacy is not new and has been discussed in earlier studies (Cowan et al., 2017; Kreuser, 2018; Lei et al., 2017), though it is still not addressed by IPA manufacturers. One of the proposed possible solutions to users’ privacy concerns would be clear privacy policies and practices for the IPAs and their third-party applications (Alhadlaq et al., 2017).

Two tech-savvy participants discussed transparency in the context of “unlocking” technology and making it available for customization and experimentation. The less tech- savvy IPA users mainly cared about transparency in the context of increasing their trust in the technology and its data management processes, as well as improving their understanding of available functionalities.

4.1.4 Personification The theme of IPA device personification was mentioned 21 times. Participants were split in their recommendations for IPAs to be more or less human-like. One participant mentioned that the current IPAs made her think of a lower scale artificial intelligence (AI) on Westworld (a TV show about human-like robots).

More participants’ comments indicated a preference for human-like behavior and drew parallels with human personal assistants:

“The ideal for a real-life personal assistant is that they’re anticipating your needs and what’s next, right? So, if Siri or Alexa are giving you reminders then it could function in that way.” [P02]

“An actual PA is not just someone who aggregates data, it’s someone who aggregates data and then makes an action of their own volition with your stamp on it[…] There’s definitely a lot more creative energy going on in interacting with an actual person that we’re still struggling to get with a digital assistant.” [P10]

However, some participants who initially expressed a preference for IPAs to “behave” more like a human assistant backed off of their initial preference. They realized that the IPA would need to collect, store, and process even more information about them in order to anticipate their needs, further compromising their privacy.

Fewer participants indicated a clear preference for non-human-like IPAs:

“I would be uncomfortable if it was more human-like” [P02]

Within the context of personification, participants discussed the positive effects of Apple’s Siri being “polite” and the negative effects of Amazon Alexa not having a “personality” that incorporates politeness or manners.

“Ideal IPA would say please and thank you” [P03]

We discuss IPA personification in some of our earlier work and note that users’ responses to personification range from love and mindless politeness to uneasy feelings explained by the “uncanny valley” hypothesis (Lopatovska and Williams, 2018; Mori, 1970). One way to reconcile such polarized preferences for personification would be through customizable features, by giving a user an ability to choose an IPA “persona” that suits them. This persona could fulfill the role of anticipating user’s needs and resemble a human assistant, or be machine-like and only address user queries directed at it, or be anything in between.

4.1.5 Compatibility/accessibility Participants were interested in IPA compatibility with other platforms and IPA systems (11 mentions). For example, several participants expressed interest in being able to conduct Google searches with an Amazon device in order “to gather the best search results.” Participants who use Google products (e.g. mail and calendar), complained that they are unable to use Apple’s Siri to its fullest extent. Participants’ frustration with the lack of support for IPA use across multiple platforms is captured in this comment:

“If the point [of IPAs] is to integrate all these systems, do that better... there’s so many company-specific limitations of what will interact with what” [P10]

User preferences for interoperable IPA platforms have been reported previously (Cowan, et al., 2017), but it is not unusual for emerging technology to be proprietary and not interchangeable. Even as common an object as the light bulb did not initially work with all hardware (Freeberg, 2013). However, we hope that IPAs will soon pass this phase and become compatible with each other, IPA-enabled IoT devices, and other systems. Some

15 steps toward greater compatibility have already been made by enabling Google searches on Amazon Alexa (Alexa Mods).

Several participants indicated a strong preference for the ability to “deconstruct” and have more control over IPAs, enabling original and innovative uses. This preference is different than the customizability preference mentioned above because it refers to the preference for being able to build on existing IPA platforms, as opposed to just controlling available features. Wanting to make IPAs “hack-able” for innovation was frequently expressed in the context of opening IPA technology to new communities, such as users with disabilities or “non-internet users--people without access to internet/living in areas that do not have information highways (dead zones)” [P05]. Participants recommended enabling IPAs to operate in an off-line mode and incorporating assistive technologies into IPA designs.

Some examples of innovative uses of IPAs can be found in the work of Pradhan et al. (2018). The authors investigated IPA usage by people with disabilities and showed that users with mobility or visual impairments could independently play music, check the time or weather, set their schedule, and control smart home devices. Some users with disabilities reported using IPAs in emergencies by sending messages to alert someone in another room (Pradhan et al., 2018). The researchers even found some unexpected uses of IPAs, which included speech therapy (users had to practice speaking loudly and clearly to interact with the IPA) and memory assistance (setting reminder alarms and asking what day it is). The researchers found that IPAs had good potential for users with disabilities and could be further developed to address the challenges of accurate speech recognition, the availability and diversity of apps/skills, troubleshooting, and other issues (Pradhan et al., 2018). While none of the participants in our study reported using an IPA in conjunction with having a disability, there are similarities between the recommendations and needs that they expressed and those expressed by the participants in Pradhan et al.’s study (2018). IPA developers should consider all of these suggestions when enhancing the usability of these devices.

4.1.6 Aesthetics Within participants’ recommendations, the aesthetic design of the stand-alone IPA devices was mentioned nine times. One participant wanted to see smaller devices:

“Perhaps, a smaller one doesn't feel so intrusive, I think.” [P13]

Two others expressed the desire for a customizable look.

“Color customization, look and feel to whether people want to see more robot-like devices or ambient technology” [P12]

“I guess it’s Google Home Mini that comes in different colors and it just seems nicer as a piece in your apartment or looks friendlier[…] Maybe if it was cuter I’d be interested in actually trying it out whereas now I’m really just not interested at all.” [P01]

Yet other participants expressed general recommendations to make the devices “more interesting,” “friendlier looking,” and “approachable”.

The role of aesthetics in regard to IPAs is mentioned in Orehovački et al.’s (2018) as a “desired attribute” of IPAs, but most of the previous studies examined for this research do not discuss it. Despite the lack of research around user requirements for IPA aesthetics, studies about the role of aesthetics of technology in general show that the aesthetics of a device do affect consumers’ purchasing decisions or how pleased they are with the device. Thüring and Mahlke’s (2007) study on the connection between aesthetics and usability supported the idea that pleasing aesthetics positively affect how users perceive usability of a device. Additionally, Toufani et al. (2017) conducted a study on whether the aesthetics of smartphones affected consumers’ purchase decisions. The authors found that design, color, tactile feel, and shape were factors people considered in determining aesthetics. Further, the study found that, although aesthetics did not directly influence buying decisions, the social and emotional values tied to aesthetics did influence them.

While aesthetics was mentioned only a few times in this study, it is clearly something that is already being considered in the development of these devices. This trend can be witnessed through the ever-changing design of the containing devices for IPAs (i.e. Amazon Alexa Echo 1st Generation versus 2nd Generation versus Echo Dot) (See Figures 1 and 2).

[insert Figure1.]

Figure 1. Formats and the designs of Amazon Alexa-enabled 1st Generation and 2nd Generation devices. Image from Amazon. “Echo Buttons work with these Echo devices,” at https://www.amazon.com/Echo-Buttons-2-Pack/dp/B072C4KCQH, accessed 8 August 2018.

[insert Figure 2.]

Figure 2. Different styles of the Amazon Echo smart speaker (released in 2017). Image from Megan Wollerton, 2017. “Amazon’s new $100 Echo promises to do everything better,” at https://www.cnet.com/news/amazons-new-100-echo-speaker- promises-to-do-everything-better/, accessed 27 July 2018.

5. Conclusion

We conducted a study to investigate user recommendations for the design of IPAs based on their current and desired experiences with this technology. The data were collected from 18 IPA users during five mini focus groups. During the conversations about participants’ current experiences with IPAs and their recommendations, the following IPA requirements emerged: • Improvements of existing features, ranging from speech recognition and input/output modalities to device feedback; and introduction of new features, such as GPS for context awareness. • Customizability of IPA features and functions. • Transparency of hardware and software designs and capabilities, as well transparency of privacy settings and data management processes. • Changes in the personification characteristics of IPAs. • Compatibility of IPAs with a range of platforms/applications and development of accessible IPAs for new user groups and new contexts. • Improved aesthetics of stand-alone IPA devices. A surprising number of recommendations that emerged from the study are rooted in basic user experience design principles for interactive technology, including requirements for customizability, transparency, aesthetically pleasing designs, and better support for multimodal interactions. These principles have been previously discussed in the context of PDAs (personal digital assistants) and other technology, and yet are not satisfactorily addressed by current IPAs. While we understand that no new product is ever perfect, certain features based on these classic principles and basic user expectations need to be prioritized.

Some of these features should take inspiration from human-to-human interactions. From the get-go, IPAs were marketed as “assistants,” therefore some parallels with human assistants would be reasonable. For example, when hiring an assistant, one would expect to see a resume that lists a potential assistant’s skills and capabilities. A similar “introduction” would be expected of a digital assistant during the initial interaction with a user. The initial device set-up could be a good time to list the device’s key functions, ways to acquire additional apps/skills, explain how the IPA works so that the user doesn’t expect miracles (from focus group feedback, current IPAs seem programmed to fool users into thinking that they are not information systems, but rather personalities who learn from their mistakes and have senses of humor and personal opinions). Another

18 example of a basic requirement is that user utterance and device feedback should be in the same modality. For example, when calling another person, we expect acknowledgement that we’re heard (in a form of “yes”, “I’m here”, “working on your request”). Similar feedback in the same modality as user utterance would be expected from the IPA (right now Echo lights up when it is listening, but doesn’t generate vocal acknowledgment of being active and ready to receive the user request). Basic customization of voices and languages would also be expected based on availability of these features in other voice-controlled technologies, such as navigation devices and apps. However, most IPAs have limited voice choices (only female voice for Amazon Alexa) and have the most advanced support for English, but not other languages.

Overall, we noted that none of our participants were unequivocally enthusiastic about IPAs. Frequent users had issues with their IPA’s technical limitations in the context of programming it and integrating it with IoT and other systems, while infrequent users struggled to find a useful purpose for IPAs and complained about the quality and delivery of information. All participants reported enjoying the hands-free convenience offered by IPAs on simple tasks (e.g., music, weather), however the general attitude could be captured by this quote from one of the participants who was asked about the likelihood of buying a smart speaker: “It’s not so much the look as the amount that I use it and how much it costs to get it, it doesn’t equal itself out.” [P03]

Due to the opaqueness of the IPA development process, we can only guess at the factors that determine companies’ prioritization in developing certain IPA features, which may include company strategy, technical challenges, and costs. However, since many of the recommendations that emerged in our study have been discussed in the earlier literature, we suggest that the developers start paying closer attention to user responses to IPAs. Identifying and addressing causes of users’ satisfaction and frustration could increase user acceptance of this emerging technology. Evidence does already suggest gradual improvements in IPAs (e.g. availability of Amazon Alexa smart speakers in multiple colors/textures), but other changes might be more difficult to implement and/or be less visible to the users. Speech recognition is an example of an IPA-specific requirement that is not only technically challenging but also illustrates how an improvement would be hard to observe. A user that has already given up on an IPA due to a lack of appropriate responses will not experience the improvements in new IPA devices.

User recommendations collected by the study could not only be used to improve IPA designs, but also to inform future research. Some of the research directions could include examining: - comparable human-to-human interactions with librarians, customer service representatives and other professionals in assisting roles in order to model IPA functionality based on familiar user experiences; - the right balance of contextually appropriate input and output modalities for IPAs, potentially including tactile feedback;

- GPS-driven contextualization, , for example, offer location, traffic and points of interest information; - users’ preferences for and willingness to engage in management of data collection and privacy settings on IPA and connecting devices (where more user data could produce more accurate system performance but compromise user privacy); - developing and testing multiple IPA personas (e.g. funny/personable, professional/cold, child-friendly, etc.) - innovative uses of IPAs by various communities of users - user expectations for IPA device aesthetics (e.g., big/small, water-proof for the shower or outdoor use, horizonal/vertical attachment points, color scheme and finishes, highly visible or ambient designs).

Our study had a number of limitations. The study relied on a relatively small sample drawn mainly from an academic community. The fact that most of our findings are in line with prior IPA research is reassuring in terms of the study’s reliability and generalizability, however a larger-scale study is warranted. In collecting participants’ recommendations, we did not distinguish between various IPAs, so some of the participants’ comments pertaining to Amazon Alexa, for example, might not hold in the context of Apple’s Siri. Future research might focus on specific types of IPAs to inform more precise device improvements or enhance a certain IPA app. Another challenge of IPA research includes rapid technology changes that necessitate equally rapid research. During the time it takes for data collection and publication, IPAs may undergo changes which render the study findings no longer relevant. We tried to address this challenge by conducting the study and writing up the results in a timely manner.

We hope that the recommendations of our participants will be of use to the communities of IPA designers, its current or future users, and IPA researchers. We also hope that designers of new interactive technology will comply with the classic design principles and ensure appropriate device feedback, support for basic customization, and transparency of device capabilities and other features. Such compliance would ensure more positive user experiences with technology and, possibly, its more enthusiastic adoption.

Acknowledgements We would like to thank our study participants for volunteering their time and ideas. We are grateful to the anonymous reviewers for their thoughtful feedback on the earlier drafts.

Appendix A. Features of the main IPAs

Most Common IPA Features* Apple’s Amazon Google Microsoft Siri Alexa Assistant Cortana

Calls and messaging x x x x

E-mail x x x

Translation x x x

Multiple profiles/User accounts x

Announcements via devices x x

Two-command capabilities x

Conversational command x x capabilities

Smart home capabilities x x x x

TV/Entertainment/Sports/Music x x x x

News x x x x

Math x x x x

Date and time x x x x

Alarms and timers x x x x

Scheduling/To-do x x x x lists/Reminders

Weather x x x x

Search capabilities x x x x

Navigation x x

Purchasing x x

Connections to third-party x x x x applications

*Based on the following sources: Jacobsson Purewal, 2016; Jacobsson Purewal and Cipriani, 2017; Martin and Priest, 2018; Martin and Priest, 2017.

Appendix B. Themes and their frequencies in participant recommendations for IPA design

Theme family Sub-theme Frequency Participant Quote

Feature 78 improvements

Speech- 30 “understand accents” recognition/langua “speak clearly” ge control “I think they should create these in multiple languages as soon as possible”

Feedback 16 “I want a visual feedback” “ambient feedback” “Clear feedback” “There is no signal I can recognize on Google Home”

Support for multi- 16 “if I were to have a personal assistant modal interactions it would be something that was more tactile or visual…”

Additional features 16 “help to open the door, or turn on air conditioner” “I cannot get most [non-US centered] answers from personal assistants… it doesn’t have much knowledge on other stuff” “I don't think it has a recommendation feature -- like what you do --- hobbies- related.” “Include GPS”

Customizability 53 “I would like it to be more customizable” “[When searching yourself] You get to choose what sources you’re looking at and if you’re using Google or if you’re using Bing and which one you trust more.” “[customize voice] speed setting ‘cause it talks slowly” “Allow users to pick a [IPA] personality”

Device 38 transparency

Understandability 22 “[it] feels "disempowering" because I can't understand how it works” “Want the process to be more transparent” “I can't see what's under the surface.”

Privacy 16 "My discomfort stems from it knowing me” “Nothing is private” “Just more control… just a better understanding of how it’s doing what it’s doing so maybe you can protect yourself a little bit more.”

Personification 21

More human-like 15 “more creative answers, perception that you are talking to a human”

Less human-like 6 “would like the IPA to remain as much in its non-human form as possible”

Compatibility/ 17 accessibility

Compatibility 11 “Alexa doesn't have Google application” “should [...] be better at connecting to other devices. Company-specific limitations are frustrating of what will interact with what.”

Accessibility 6 “allow innovation and make IPA open to the community [of] disabled people, non-internet environments.” “expand horizons, cater to more complex people/schedules”

Aesthetics 9 “I would suggest making IPAs more interesting, it's boring and not design- driven” “Aesthetics would be great, have customizability, more adaptability so that it blends more into the home” “A friendlier looking piece would be appropriate. IPAs don’t seem approachable”

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Alhadlaq A, Tang J, Almaymoni M and Korolova, A (2017) Privacy in the Amazon Alexa skills ecosystem. In: 17th Privacy Enhancing Technologies Symposium, Minneapolis, MN, 18-21 2017. Available at: https://petsymposium.org/2017/papers/hotpets/amazon-alexa-skills-ecosystem- privacy.pdf (accessed 14 November 2018).

Alexa Mods. Available at: https://alexamods.com/ (accessed 10 August 2018).

Betters E and Grabham D (2018) Apple HomePod vs Google Home vs Amazon Echo: What’s the difference? Pocket-Lint, 6 February. Available at: https://www.pocket- lint.com/speakers/buyers-guides/139063-apple-homepod-vs-google-home-vs- amazon-echo-what-s-the-difference (accessed 20 July 2018).

Bopp C (2018) User experience with voice user interfaces – a diary study using Amazon Echo as an example. In: Service Plan Blog. Available at https://serviceplan.blog/en/2018/01/study-voice-user-interfaces-amazon-echo/ (accessed 11 July 2018).

Botsman R (2017) Co-parenting with Alexa. The New York Times, 7 October), SR4.

Bragg D, Bennett C, Reinecke K and Ladner R (2018) A large inclusive study of human listening rates. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18), Montréal, Canada, 21-26 April 2018. New York, NY: ACM Press. DOI: 10.1145/3173574.3174018.

Brenner B (2017) Know the risks of Amazon Alexa and Google Home. Naked Security, 27 January. Available at: https://nakedsecurity.sophos.com/2017/01/27/data-privacy- day-know-the-risks-of-amazon-alexa-and-google-home/ (accessed 20 July 2018).

Calore M (2017) Watch people with accents confuse the hell out of AI assistants. Wired, 16 May. Available at: https://www.wired.com/2017/05/ai-assistants-accented- english/ (accessed 11 July 2018).

Canbek NG and Mutlu ME (2016) On the track of artificial intelligence: Learning with intelligent personal assistants. Journal of Human Sciences 13(1): 592-601. DOI: 10.14687/ijhs.v13i1.3549.

Chen BX (2018) Apple's HomePod has arrived. Don't rush to buy it. The New York Times, 6 February. Available at: https://www.nytimes.com/2018/02/06/technology/personaltech/apple-homepod- review.html (accessed 20 July 2018).

Cowan BR, Pantidi N, Coyle D, Morrissey K, Clarke P, Al-Shehri S, Earley D and Bandeira N (2017) “What can I help you with?”: Infrequent users' experiences of intelligent personal assistants. In: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (Mobile HCI ‘17), Vienna, Austria, 4-7 September 2017. New York, NY: ACM Press. DOI: 10.1145/3098279.3098539.

Crist R (2016) Amazon Alexa: Device compatibility, how-tos and much more. CNET, 8 April. Available at https://www.cnet.com/how-to/amazon-alexa-device- compatibility-how-tos-and-much-more/ (accessed 6 August 2018).

Crist R (2017) Amazon Echo Spot review: Alexa’s touchscreen misses the sweet spot. CNET, 16 January. Available at https://www.cnet.com/reviews/amazon-echo-spot- review/ (accessed 25 July 2018).

De Vet J and Buil V (1999) A personal digital assistant as an advanced remote control for audio/video equipment. In: Proceedings of the Second Workshop on Human Computer Interaction with Mobile Devices (MobileHCI’99), Edinburgh, Scotland, 31 August, pp.87-91. Available at: http://www.dcs.gla.ac.uk/mobile99/posters/de_vet.pdf (accessed 1 August 2018).

Dickey MR (2018) Now you can have a conversation with Alexa without screaming “Hey, Alexa” for every request. TechCrunch. Available at:

https://techcrunch.com/2018/03/10/now-you-can-have-a-conversation-with-alexa- without-screaming-hey-alexa-for-every-request/ (accessed 21 July 2018).

Dizon G (2017) Using intelligent personal assistants for second language learning: A case study of Alexa. TESOL Journal 8(4): 811-830. DOI: 10.1002/tesj.353.

Easwara Moorthy A and Vu K-PL (2015) Privacy concerns for use of voice activated personal assistant in the public space. International Journal of Human-Computer Interaction 31: 307-335. DOI: 10.1080/10447318.2014.986642.

Efthymiou C and Halvey M (2016) Evaluating the social acceptability of voice based smartwatch search. In: 12th Asia Information Retrieval Societies Conference, (AIRS 2016) (eds S Ma, J-R Wen, Y Liu, Z Dou, M Zhang, Y Chang, and X Zhao), Beijing, China, 30 November - 2 December 2016, pp.267-278. New York, NY: Springer. DOI: 10.1007/978-3-319-48051-0_20.

Enge E (2018) Rating the smarts of the digital personal assistants in 2018. Report, Stone Temple Consulting, 23 April. Available at: https://www.stonetemple.com/digital- personal-assistants-study/ (accessed 11 July 2018).

Flores Letelier MF, Spinosa C and Calder BJ (2000) Taking an expanded view of customers' needs: qualitative research for aiding innovation. Marketing Research 12(4): 4-11.

Freeberg E (2013) The age of Edison: Electric light and the invention of modern America. New York, NY: The Penguin Press.

Gilmore JH and Pine BJ II (1997) The four faces of mass customization. Harvard Business Review 75(1): 91-101.

Graus D, Bennett PN, White RW and Horvitz E (2016) Analyzing and predicting task reminders. In: Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization (UMAP '16), Halifax, Nova Scotia, Canada, 13-17 July 2016, pp.7-15. New York, NY: ACM. DOI: https://doi.org/10.1145/2930238.2930239.

Greenbaum T (2000) Moderating focus groups. Thousand Oaks, CA: Sage Publications, Inc.

Guo F, Metallinou A, Khatri C, Raju A, Venkatesh A and Ram A (2017) Topic-based evaluation for conversational bots. In: Thirty-first Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, 4-9 December. Available at: https://arxiv.org/abs/1801.03622 (accessed 13 February 2018).

Han S and Yang H (2018) Understanding adoption of intelligent personal assistants: A parasocial relationship perspective. Industrial Management & Data Systems 118(3): 618-636. DOI: 10.1108/IMDS-05-2017-0214.

Hastie H, Liu X and Patron P (2017) Trust triggers for multimodal command and control interfaces. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI 2017), Glasgow, UK, 13-17 November, pp.261-268. New York, NY: ACM. DOI: 10.1145/3136755.3136764.

Hoseth A and McClure M (2012) Perspectives on E-books from instructors and students in the social sciences. Reference & User Services Quarterly 51(3): 278-288.

Jacobsson Purewal S (2016) The complete list of Cortana commands. CNET, 6 May. Available at: https://www.cnet.com/how-to/the-complete-list-of-cortana-commands/ (accessed 1 August 2018).

Jacobsson Purewal S and Cipriani J (2017) The complete list of Siri commands. CNET, 27 November. Available at: https://www.cnet.com/how-to/the-complete-list-of-siri- commands/ (accessed 1 August 2018).

Kinsella B (2018) Smart speaker users pass 50 million in U.S. for the first time. Available at: https://voicebot.ai/2018/06/28/smart-speaker-users-pass-50-million-in-u-s-for- the-first-time/ (accessed 3 July 2018).

Kinsella B and Mutchler A (2018) Smart speaker consumer adoption report. Report, Voicebot.ai. Available at: https://voicebot.ai/wp- content/uploads/2018/03/smart_speaker_consumer_adoption_report_2018.pdf (accessed 3 July 2018).

Kiseleva J, Williams K, Jiang J, Awadallah AH, Crook AC, Zitouni I and Anastasakos T (2016) Understanding user satisfaction with intelligent assistants. In: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (CHIIR ‘16), Chapel Hill, NC, 13-17 March 2016, pp.121-130. New York, NY: ACM Press. DOI: 10.1145/2854946.2854961.

Kreuser AP (2018) Here's how 5 huge brands are marketing to you through Amazon Alexa. Inc., 12 February. Available at: https://www.inc.com/amanda-pressner- kreuser/heres-how-5-major-brands-are-marketing-to-you-through-amazon-alexa.html (accessed 13 February 2018).

Lei X, Tu G-H, Liu AX, Li C-Y and Xie T (2017) The insecurity of home digital voice assistants - Amazon Alexa as a case study. Available at https://arxiv.org/abs/1712.03327 (accessed 13 February 2018).

Lopatovska I and Oropeza H (2018) User Interactions with “Alexa” in Public Academic Space. In: Annual Meeting of the Association for Information Science and Technology, Vancouver, Canada, 9 - 14 November 2018. Available at: https://irenelopatovska.files.wordpress.com/2018/08/asist18-alexa-in-public-02.pdf (accessed 14 November 2018).

Lopatovska I, Rink K, Knight I, Raines K, Cosenza K, Williams H, Sorsche P, Hirsch D, Li Q and Martinez A (2018) Talk to me: exploring interactions with the Amazon Alexa. Journal of Librarianship & Information Science 7 March 2018. DOI: 10.1177/0961000618759414.

Lopatovska, I. and Williams, H. (2018). Personification of the Amazon Alexa: BFF or a mindless companion?. In Proceedings of ACM CHIIR conference, New Brunswick, NJ USA, March 11-15, 2018 (CHIIR’18), 4 pages. DOI: 10.1145/3176349.3176868

Luger E and Sellen A (2016) “Like having a really bad PA”: The gulf between user expectation and experience of conversational agents. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ‘16), San Jose, CA, 7-12 May 2016, pp.5286-5297. New York, NY: ACM Press. DOI: 10.1145/2858036.2858288.

MartinT and Priest D (2017) The complete List of Alexa commands so far. CNET, 24 September. Available at: https://www.cnet.com/how-to/amazon-echo-the-complete- list-of-alexa-commands/ (accessed 1 August 2018).

Martin T and Priest D (2018) The complete list of Google Home commands so far. CNET, 14 November. Available at: https://www.cnet.com/how-to/google-home- complete-list-of-commands/ (accessed 1 August 2018).

Mills C (2018) Here are all the things Google Assistant now does that Amazon Alexa can’t. BGR, 9 May. Available at: https://bgr.com/2018/05/09/google-assistant-vs- amazon-alexa-app-siri-2018-features/ (accessed 11 July 2018).

Moore RK, Li H and Liao S-H (2016) Progress and prospects for spoken language technology: What ordinary people think. In: INTERSPEECH 2016, San Francisco, CA, 8-12 September 2016, pp.3007-3011. DOI: 10.21437/Interspeech.2016-874.

Mori M (1970) The uncanny valley. Energy 7(4): 33-35.

Munteanu C, Irani P, Oviatt S, Aylett M, Penn G, Pan S, Sharma N, Rudzicz F, Gomez R, Cowan B and Nakamura K (2017 Designing speech, acoustic and multimodal interactions. In: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17), Denver, CO, 6-11 May 2017, pp.601-608. New York, NY: ACM Press. DOI: 10.1145/3027063.3027086.

Nielsen J (1995) Nielsen Norman Group: 10 heuristics for user interface design. Available at: https://www.nngroup.com/articles/ten-usability-heuristics/ (accessed 1 August 2018).

Orehovački T, Granić A and Kermek D (2013) Evaluating the perceived and estimated quality in use of Web 2.0 applications. Journal of Systems and Software 86(12): 3039-3059. DOI: 10.1016/j.jss.2013.05.071.

Orehovački T, Babić S and Etinger D (2018) Modelling the perceived pragmatic and hedonic quality of intelligent personal assistants. In: IHSI 2018: Advances in Intelligent Systems and Computing (eds W Karwowski and T Ahram), Dubai, United Arab Emirates, 7-9 January 2018, pp.589-594. Basel, Switzerland: Springer International Publishing AG. DOI: 10.1007/978-3-319-73888-8_91.

Paterson L and Low B (2011) Student attitudes towards mobile library services for smartphones. Library Hi Tech 29(3): 412-423. DOI: 10.1108/07378831111174387.

Pierce D (2017) Review: Amazon Echo Show. Wired, 26 June. Available at: https://www.wired.com/2017/06/review-amazon-echo-show/ (accessed 25 July 2018).

Porcheron M, Fischer JE and Sharples S (2017) “Do Animals Have Accents?”: Talking with Agents in Multi-Party Conversation. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17), Portland, OR, 25 February-1 March 2017, pp.207-219. New York, NY: ACM Press. DOI: 10.1145/2998181.2998298.

Porcheron M, Fischer JE, Reeves S and Sharples S (2018) Voice interfaces in everyday life. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). Montréal, Canada, 21-26 April 2018. New York, NY: ACM Press. DOI: 10.1145/3173574.3174214.

Pradhan A, Mehta K and Findlater L (2018) “Accessibility came by accident”: Use of voice-controlled intelligent personal assistants by people with disabilities. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). Montréal, Canada, 21-26 April 2018. New York, NY: ACM Press. DOI: 10.1145/3173574.3174033.

Purington A, Taft JG, Sannon S, Bazarova NN and Taylor SH (2017) “Alexa is my new BFF”: Social roles, user satisfaction, and personification of the Amazon Echo. In: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ‘17), Denver, CO, 6-11 May, pp.2853-2859. New York, NY: ACM Press. DOI: 10.1145/3027063.3053246.

Ranj B (2018) Amazon vs Apple vs Google vs Sonos: How 4 of the most popular smart speakers stack up. Business Insider, 3 February. Available at: http://www.businessinsider.com/best-smart-speaker-comparison-amazon-echo- apple-homepod-google-home-sonos-one-2018-1 (accessed 20 July 2018).

Santos J, Rodrigues JJPC, Casal J, Saleem K and Denisov V (2018) Intelligent personal assistants based on Internet of Things approaches. IEEE Systems Journal 12(2): 1793-1802. DOI: 10.1109/JSYST.2016.2555292.

Seeholzer J and Salem JA (2011) Library on the go: A focus group study of the mobile web and the academic library. College & Research Libraries 72(1): 9-20. DOI: 10.5860/crl-65r1.

Sörenson I (2017) Expectations on chatbots among novice users during the onboarding process. KTH Royal Institute of Technology, School of Computer Science and Communication. Available at: http://kth.diva- portal.org/smash/record.jsf?pid=diva2%3A1078272&dswid=6048 (accessed 21 July 2018).

Thompson C (2017) Amazon's Alexa is officially coming to Ford cars. Business Insider, 4 January. Available at: http://www.businessinsider.com/amazon-alexa-coming-to- ford-cars-2017-1 (accessed 20 July 2018).

Thüring M and Mahlke S (2007) Usability, aesthetics and emotions in human–technology interaction. International Journal of Psychology 42(4): 253-264. DOI: 10.1080/00207590701396674.

Toufani S, Stanton JP and Chikweche T (2017) The importance of aesthetics on customers’ intentions to purchase smartphones. Marketing Intelligence & Planning 35(3): 316-338. DOI: 10.1108/MIP-12-2015-0230.

Villas-Boas A (2017) 14 ways you can control your home with your voice using Amazon's Echo and Alexa. Business Insider, 10 January. Available at: http://www.businessinsider.com/amazon-echo-alexa-control-smart-home-with-voice- 2017-1/#smart-home-hubs-compatible-with-alexa-1 (accessed 20 July 2018).

Wilkinson S (2010) Focus group research. In: Silverman D (ed) Qualitative research: theory, method and practice. London: SAGE Publishing, pp.177-199.

Wilson HJ, Daugherty PR and Bianzino NM (2017) When AI becomes the new face of your brand. Harvard Business Review, 27 June. Available at: https://hbr.org/2017/06/when-ai-becomes-the-new-face-of-your-brand (accessed 20 July 2018).