Affective Interfaces of Embodied Conversational Agents
Studies of Hardware and Character Interfaces
Yi-Chen Hsu
DOCTORAL THESIS School of Design Studies College of Fine Arts The University of New South Wales
October 2011
ORIGINALITY STATEMENT ‘I hereby declare that this submission is my own work and to the best of my knowledge it contains no materials previously published or written by another person, or substantial proportions of material which have been accepted for the award of any other degree or diploma at UNSW or any other educational institution, except where due acknowledgement is made in the thesis. Any contribution made to the research by others, with whom I have worked at UNSW or elsewhere, is explicitly acknowledged in the thesis. I also declare that the intellectual content of this thesis is the product of my own work, except to the extent that assistance from others in the project's design and conception or in style, presentation and linguistic expression is acknowledged.’
Signed Date 01/05/2012 COPYRIGHT STATEMENT ‘I hereby grant the University of New South Wales or its agents the right to archive and to make available my thesis or dissertation in whole or part in the University libraries in all forms of media, now or here after known, subject to the provisions of the Copyright Act 1968. I retain all proprietary rights, such as patent rights. I also retain the right to use in future works (such as articles or books) all or part of this thesis or dissertation. I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstract International (this is applicable to doctoral theses only). I have either used no substantial portions of copyright material in my thesis or I have obtained permission to use copyright material; where permission has not been granted I have applied/will apply for a partial restriction of the digital copy of my thesis or dissertation.'
Signed Date 01/05/2012
AUTHENTICITY STATEMENT ‘I certify that the Library deposit digital copy is a direct equivalent of the final officially approved version of my thesis. No emendation of content has occurred and if there are any minor variations in formatting, they are the result of the conversion to digital format.’
Signed Date 01/05/2012
Acknowledgements
I would like to thank several people and institutions for providing me with the necessary guidance and support to complete this research. First of all, I truly express my wholehearted gratitude to my supervisor, Professor Ross Harley, for his solid knowledge and rich experience in the field of multimedia research. This insight has benefited me throughout my work, from the starting point of my dissertation to the end, involving the interpretation of the findings of my study. I also sincerely express my thanks to my co- supervisor, Associate Professor Fang Xu. His wise advice has greatly contributed to improving my research structure. Finally, in terms of academic counselling, I thank Doctor Steve Clark, whose shrewd suggestions significantly helped me with the formation of concepts and in the design of research.
I express my heartfelt thanks to the Institute of Information Science (IASL), Academia Sinica, which gave me detailed technical suggestions and support on the artificial intelligence aspects of my research. I offer considerable appreciation to the Apple University Consortium for the loan of equipment for performing the hardware interface experiment. Of course, I also want to give my regards to the friends around me for always supporting me with my research. In particular, I want to thank Ivy and Dora, who assisted me with two experiments, Sindy and Shmmindy, who gave me invaluable advice on statistics, Kevin and Haor, who provided me with some technological help on system developments, and Philip and Meeno, who helped me improve my practical skills in professional writing. There are so many more individuals who have assisted me throughout the duration of my research, although too many to mention in this piece. For those of you not referred to by name, I also want to thank you profusely.
Finally, my deepest gratitude goes to my family, especially my parents, my brother and my sister, who provided every type of support that I needed to succeed in this endeavour.
iv Table of Contents
Originality Statement ...... iii
Acknowledgements ...... iv
Table of Contents ...... v
List of Figures ...... ix
List of Tables ...... xi
Abstract ...... xiii
Glossary ...... xiv
Prologue. The Interface is a Social Actor ...... 1
Chapter 1. Introduction ...... 9
1.1. When Computers are Embodied Characters ...... 11
1.2. The Importance of Affective Factors ...... 14
1.3. What are Anthropomorphic Agents? ...... 16
1.4. Domain Analysis ...... 21
1.5. Summary ...... 25
Part I. Study of Affective Interfaces of ECAs ...... 29
Chapter 2. From AI to ECAs ...... 31
2.1. Brief Introduction to Artificial Intelligence (AI) ...... 31
2.2. Intelligent Agents ...... 38
2.3. Affective Computing ...... 42
2.4. Embodied Conversational Agents (ECAs) ...... 44
Chapter 3. Interface Design of Agents ...... 51
3.1. User Interfaces ...... 51
3.2. Interfaces of Embodied Agents ...... 62
3.3. Visual Impact of Agent Interfaces...... 66
3.4. Methods of Agent Interface Study ...... 72
Chapter 4. Affective Factors of Intelligent Agents ...... 77
v 4.1. What is Affect? ...... 77
4.2. Affective Displays – Emotions ...... 80
4.3. Affective Science ...... 83
4.4. Affective Factors and Influences ...... 87
4.5. Affective Agents ...... 90
Part II. Surveys on Hardware and Character Interfaces of ECAs ...... 93
Chapter 5. Method of Hardware Interfaces of Chatbots ...... 95
5.1. Method ...... 95
5.2. Measures ...... 100
5.3. Materials ...... 103
5.4. Hardware Interface Design of Chatbots ...... 110
Chapter 6. Evaluation of Hardware Interfaces of Chatbots ...... 115
6.1. Procedure...... 115
6.2. Subjects ...... 118
6.3. Analysis of Questionnaire ...... 120
6.4. Qualitative Results ...... 124
6.5. Summary ...... 132
Chapter 7. Method of Character Interfaces of CompanionBots ...... 135
7.1. Method ...... 136
7.2. Measures ...... 139
7.3. Materials ...... 142
7.4. Design and Classification of CompanionBots ...... 144
Chapter 8. Evaluation of Character Interfaces of CompanionBots ...... 149
8.1. Procedure...... 149
8.2. Subjects ...... 152
8.3. Questionnaire Analysis ...... 155
8.4. Qualitative Results ...... 162
8.5. Summary ...... 167 vi Part III. Adopting Results to Build a Multi-agent Platform ...... 171
Chapter 9. Development of a Multi-agent Website with Affective Interfaces ...... 173
9.1. Introduction ...... 174
9.2. Requirements and Specifications ...... 178
9.3. System Architecture and Interface Design ...... 178
9.4. Building the Website ...... 184
Chapter 10. Conclusions and Future Work ...... 187
10.1. Summary of Significant Contributions ...... 187
10.2. Implications ...... 189
10.3. Discussion ...... 191
10.4. Future Work ...... 193
10.5. Concluding Remarks ...... 194
References ...... 197
Appendices ...... 217
Appendix A. Materials of the Survey of Hardware Interfaces of Chatbots ...... 217
Participant Information Statement and Consent Form ...... 217
Questionnaire–Subject Personal Information Component ...... 219
Questionnaire – Chatbot on a Computer Component...... 221
Questionnaire – Chatbot on a Portable Device Component ...... 223
Questionnaire – Impression of Subject on both Chatbots Component ...... 225
Advertisement ...... 226
Participants’ Intake Script ...... 227
Static Images of Chatbots’ Animations ...... 231
Emotional Keywords of Chatbots ...... 232
Sample Scripts of Gary – PI-Bot ...... 233
Appendix B. Materials of the Survey of Character Interfaces of CompanionBots ..... 243
Participant Information Statement and Consent Form ...... 243
Questionnaire – Subject Personal Information Component ...... 245
vii Questionnaire – Preference before Interactions Component ...... 247
Questionnaire – CompanionBot Andrew Component ...... 248
Questionnaire – CompanionBot Ben Component ...... 250
Questionnaire – CompanionBot Chris Component ...... 252
Questionnaire – User Engagement Component ...... 254
Participants’ Intake Script ...... 256
Static Images of CompanionBots’ Animations ...... 259
Emotional Keywords of CompanionBots ...... 261
Sample Scripts of IELTS – IELTS-Bot ...... 263
Appendix C. Materials of the Multi-agent Website ...... 269
The Annotated ALICE AIML Files (AAA Files) ...... 269
Static Images of CompanionBots’ Animations ...... 271
Disc Content ...... Back Cover
viii List of Figures
Figure 1. Left: Droz’s Writing Boy (Fryer & Marshall, 1979); Right: Kempelen’s Speaking Machine (Gerard, 2008)...... 2 Figure 2. Left: HAL 9000 in 2001: A Space Odyssey (Kubrick & Clarke, 1968); Right: R2-D3 and C-3PO in Star Wars Episode IV: A New Hope (Lucas, 1977)...... 3 Figure 3. Left: Space Invaders (Nishikado, 1978); Right: Pac-Man (Iwatani et al., 1980)...... 4 Figure 4. Left: Ananova © Orange SA; Right: STEVE (Johnson & Rickel, 1997)...... 4 Figure 5. Social actors as interfaces in film/video. Top left: The Knowledge Navigator in Apple’s concept videos (Sculley, 1989); Top right: Dr. Know in Artificial Intelligence: AI (Spielberg, 2001); Bottom left: a virtual librarian in The Time Machine (Wells, 2002); Bottom right: a virtual shop assistant in Minority Report (Spielberg, 2002)...... 5 Figure 6. Top left: REA (Cassell, 2000); Top middle: Laura (Bickmore, 2003); Right: Anna (Inter-IKEA Systems B.V., 2011); Bottom left: ALICE (ALICE AI Foundation, 2008); Bottom middle: Sam (Ryokai et al., 2003)...... 6 Figure 7. Social robot (Breazeal, 2003)...... 12 Figure 8. Left: Microsoft Office Assistant, Clippit (Dickinson College, 2007); Right: Knowledge Navigator, Phil (Sculley, 1989)...... 17 Figure 9. REA (Real Estate Agent) (Cassell, 2000)...... 19 Figure 10. Demonstration of ALICE (ALICE AI Foundation, 2008)...... 19 Figure 11. Web version of Xiaoi (Incesoft, 2008)...... 20 Figure 12. Artificial toys; Left: Pleo (UGOBE Inc, 2008); Middle: Furby (Tiger Electronics, 1998); Right: AIBO (Sony, 1999)...... 38 Figure 13. The oN-Line System display, keyboard and mouse (Reimer, 2005)...... 54 Figure 14. The Xerox Star (Raymond & Landley, 2004)...... 54 Figure 15. Our Room - The Study (Andy, 2005)...... 56 Figure 16. Upper unit action units and some combinations (Tian et al., 2001). 85 Figure 17. The conversational user interface of Gary...... 98 Figure 18. The chat flow of chatbots...... 99 Figure 19. Coding scheme of the emotional influences of chatbots...... 101
ix Figure 20. Coding scheme of the user-agent relationships...... 102 Figure 21. Faces of ‘imploring’, ‘ordering’ and ‘suggesting’ (Pelachaud & Poggi, 1998)...... 107 Figure 22. Faces of ‘joy’, ‘surprise’ and ‘mild’ (Pelachaud & Poggi, 2002). . 107 Figure 23. Ten facial expressions of a Caricature Female (Koda & Maes, 1996)...... 108 Figure 24. A range of sample expressions of Kismet (Breazeal, 2003)...... 108 Figure 25. Japanese comic style facial expressions (Koda et al., 2009)...... 109 Figure 26. American comic style facial expressions (Koda et al., 2009)...... 109 Figure 27. Total data consumed by Opera Mini users per month (Opera Software, 2010)...... 111 Figure 28. The interaction between one subject and Gary on a computer...... 117 Figure 29. The interactions between subjects and Gavin on portable devices.117 Figure 30. Conversational user interfaces of learning companion agents...... 137 Figure 31. 18 characters in the character interface survey...... 138 Figure 32. The chat flow of the CompanionBots...... 139 Figure 33. The six characters belonging to the human classification...... 145 Figure 34. The six characters belonging to the animal classification...... 146 Figure 35. The six characters belonging to the creative creature classification...... 146 Figure 36. Exaggerated expressions of Andrew...... 148 Figure 37. Static images of fear animation of Andrew...... 148 Figure 38. The ranks of the three character classifications...... 156 Figure 39. The correlation between preference, relationship and engagement161 Figure 40. US Internet users who believe a mobile phone is a necessity (Phillips, 2010)...... 175 Figure 41. Structure of the front-end of website...... 180 Figure 42. Interface structure of the multi-agent website...... 182 Figure 43. The conversational user interface page...... 183
x List of Tables
Table 1. ECAs developed for these two surveys...... 24 Table 2. A summary of research questions, hypotheses and findings...... 24 Table 3. The visual impact of agent interfaces on user experiences, behaviours and performances...... 69 Table 4. ECAs as chatbots in the hardware interface survey...... 97 Table 5. Facial expressions of the chatbots in the hardware interface survey. 110 Table 6. Gender of subjects in the hardware interface survey...... 118 Table 7. Age groups of subjects in the hardware interface survey...... 118 Table 8. Subjects’ usage experience of computers and portable devices in the hardware interface survey...... 119 Table 9. The number of hours of computer usage and carrying portable devices per day in the hardware interface survey...... 119 Table 10. Wilcoxon signed-rank of the emotional influences of chatbots on subjects...... 121 Table 11. Wilcoxon signed-rank of the relationships between subjects and chatbots...... 122 Table 12. ECAs as learning companion agents (CompanionBots) in the character interface survey...... 136 Table 13. Summary definitions of the psychological constructs of the character survey ...... 141 Table 13. Emotional expressions of the CompanionBots in the character interface survey...... 144 Table 14. Representations of the three selected characters from three character classifications...... 146 Table 15. Gender of subjects in the character interface survey...... 152 Table 16. Age groups of subjects in the character interface survey...... 152 Table 17. English learning experience of subjects in the character interface survey...... 153 Table 18. Computer usage experience of subjects in the character interface survey...... 153 Table 19. Computer usage per day of subjects in the character interface survey...... 153 Table 20. The average time subjects spent chatting with others per day...... 154 Table 21. The average time subjects spent chatting with others in English per
xi day...... 154 Table 22. The scores of each character...... 156 Table 23. Item analysis about the questionnaire of the three CompanionBots...... 158 Table 24įġ One-way ANOVA of seven affective responses of subjects toward CompanionBots...... 159 Table 25. Pearson’s product-moment correlation coefficient of preference, relationship and engagement...... 161 Table 26. Ten characters applied to the multi-agent website...... 176 Table 27. The layout and content of web pages...... 180 Table 28. Facial expressions of the CompanionBots on the multi-agent website...... 183
xii Abstract
This research aims to investigate existing approaches to development and design guidelines of affective interfaces in relation to Embodied Conversational Agents (ECAs) from the perspective of technology, psychology and design. It also asks whether the affective influences of ECAs differ across the various devices used to display these characters. This research conducted two surveys to investigate the affective interactions between users and agents. The first hardware interface experiment explores the emotional influences of ECAs and user-agent relationships when ECAs are applied to computers and portable devices. The second character interface experiment studies the affective influences of ECAs on users in the learning tasks. The thesis asks whether there are significant experiential differences between ECAs when they are represented by different hardware interfaces and character classifications, such as emotional influences, character preference, user engagement and user-agent relationships. This research followed design guidelines that enabled the exploration of existing research and my own original experiments into the practical use of multi-agent in the context of a language learning website. I argue that the user is always at the centre of any ECA design, and therefore needs to be at the centre of any design process. The thesis discusses some concepts for developing agent interfaces with more positive affective influences for the completion of learning tasks.
The main contributions of this research are summarised as follows: (1) the hardware interfaces of ECAs have a distinct relationship to the affective responses of users, including emotional influences and user-agent relationships, (2) character classifications can be related to human affective factors, such as character preference, user engagement and user-agent relationships, and (3) there are significant relations that need to be accounted for among character preference, engagement and user-agent relationships.
xiii Glossary
Agent interfaces: Agent interfaces are defined as the external representations of intelligent agents, including GUIs and hardware interfaces.
Embodied Conversational Agents (ECAs): ECAs encompass one type of intelligent agent represented by virtual characters, and possess the ability to interact with humans by means of language and gestures.
Affective interfaces of ECAs: The interfaces of ECAs that are able to express emotions or capture human emotions lead to the alterations of human affective states during human- computer interactions, such as emotions, feelings, moods and affects.
Affective factors: Affective factors are those responses of human affected by human emotions and feelings, such as empathy, attitude, mood and motivation.
Anthropomorphic agents: Anthropomorphic agents generally are the computer systems displayed in human forms, and are also types of affective computing.
Natural Language Processing (NLP): NLP is a system that computer scientists have developed to simulate human language processing.
Affective Computing: Affective computing is a subdivision of AI, and is a multi- discipline research spanning several areas, such as computer science, psychology and cognitive science. It differs from other research regarding AI because affective computing not only focuses on usability, but also takes social factors into account, such as emotions.
Affective science: Affective science is the scientific study of affects or emotions, and involves the causes of emotions, emotional expressions, emotional experience and recognition of emotions in its scope (Kring & Moran, 2008).
Social actor: A social actor is one type of virtual character that is equipped with artificial intelligence and interacts with humans just like another real individual.
xiv Prologue. The Interface is a Social Actor
Visual artificial agents are the key focus of this research. This particular type of man- made creature originates in the human imagination as it has evolved historically. This can be seen in such imaginative works as the myth of Pandora in Greek mythology and the character of Frankenstein in Mary Shelley’s leading nineteenth century novel. In fact, these imaginative figures lie at the heart of the social imagination that has enabled individuals to envisage artificial intelligence (AI). As theories and technologies have been expanded, these figures of the imagination have led to the creation of embodied virtual creatures.
The modern manifestation of man-made creatures, which typically exist in the sphere of computing, are an augmented form of these earlier depictions, developed into virtual or social actors to simulate the actions of real humans. This has been predominately described in the research as functioning within the paradigm of computers behaving as a social actor, as exemplified by Nass, Steuer and Tauber (1994) in “Computers are Social Actors”. This paradigm suggests that a social actor is one type of virtual character that is equipped with artificial intelligence and interacts with humans just like any other real individual. It is likely that users may be unconsciously interacting with these social actors and reading personality, beliefs, and attitudes into them as well as acting on these beliefs (Mishira, 2006).
Another form of the human imagination that has been generally depicted in the cultural imagination is the figure of the automaton, which is a man-made machine that possesses the ability to talk, play, and interact with real humans in a natural manner. Automatons evolved from their beginnings in the eighteenth century where designers attempted to develop machines and processes that had the potential to closely match the thoughts or replicate certain behaviours of humans. This was evidenced when the automaton builder, Jacque Droz, created The Writing Boy in 1774, which had the ability to blink its eyes, move its head and write (Fryer & Marshall, 1979). Another example of an automaton includes Wolfgang Von Kempelen's Speaking Machine, which attempted to produce sounds replicating human speech, and this device could produce some speech sounds including entire words and short sentences (Gerard, 2008).
1
Figure 1. Left: Droz’s Writing Boy (Fryer & Marshall, 1979); Right: Kempelen’s Speaking Machine (Gerard, 2008).
In contrast to the functional and scientific aims of the developers who formulated automata in an earlier age, many of those built in the nineteenth century were primarily conceived for entertainment purposes. In general, the main forms of this period were mainly created to simulate human actions (Cassell, 2000). Furthermore, the limitations of the prevailing technological and scientific sphere of this era meant that development of man-made creatures mostly addressed exterior features.
Nevertheless, the onset of the progression of AI, which began in the 1940s to 1960s, led to new thinking and capabilities. Bush (1945) postulated that people in the future would not personally handle each and every piece of information – as this is likely to have been excessively time-consuming – to determine the usability of every piece of information. Therefore, Bush proposed a future facility, which would have the potential to handle vast libraries of information, namely the Memex. The Memex was conceived as one type of mechanised private filing and library system for individual use. This device also facilitated users to store all available forms of information, records, and communication. Furthermore, the Memex was mechanised so that it could be consulted quickly and easily. This idea also directly inspired the development of intelligent agents and also influenced the conception of the graphical user interface (GUI) in the 1960s and 1970s (Reimer, 2005). AI made significant advances during this period. These advancements have been attributed to technological developments of this period, which enabled individuals to conceptualise their dreams and imagination with the aid of technical assistance (Laurel, 1990). Memex entailed various practical descriptions, first posited by Bush as well as other researchers that followed. The dream of realising man-made creatures enabled assorted scientists to develop AI and GUI. However, Bush had not depicted the appearance of Memex in his research, as he only addressed its functionality.
Further developments of AI occurred from early stages in the 1960s to the 1980s where it began to find its way into the popular imagination, especially in science fiction literature
2 and cinema. One of the most popularised images of the potential of AI was the HAL9000 computer in Stanley Kubrick’s 2001: A Space Odyssey in 1968. This illustrated how AI could be utilised to enable a computer to naturally interact and converse with astronauts on a spacecraft. Curiously, HAL does not have any anthropomorphic form, albeit represented purely as a red camera eye that speaks in a conversational manner through the course of the film (Kubrick & Clarke, 1968). The computer’s decision-making processes led it to kill the astronauts in order to save itself meaning that the device had been conceived with a murderous element. Other representations of conversational robots as human servants can be found in the 1977 image of R2-D2 in Star Wars Episode IV: A New Hope (Lucas, 1977). In fact, a friendly AI astro-mech droid R2-D2 and its companion C-3PO are major characters throughout all six Star Wars films. These fictional AI characters in films provide complex imaginings of how man-made creatures might exist in reality. This derives from their ability to think and act like humans as well as partially resemble them.
Figure 2. Left: HAL 9000 in 2001: A Space Odyssey (Kubrick & Clarke, 1968); Right: R2-D3 and C-3PO in Star Wars Episode IV: A New Hope (Lucas, 1977).
Besides these well-known examples of anthropomorphised film characters, embodied virtual creatures were also being developed in the computer game industry around the same time. One of the most notable and influential of these videogame inventions was Space Invaders, created by Toshihiro Nishikado who programmed the game for Taito of Japan in 1978 (Nishikado, 1978). In truth, this videogame was one of the first instances where users could interact with virtual characters, which were represented by alien forms. Another famous example of this situation was Pac-Man, which was released in 1980, in which several hungry virtual creatures hunted the elusive Pac-Man (Iwatani, Funaki, & Kai, 1980). This game was a milestone in the development of embodied computer characters in the advancement towards more interactive and social forms (Haake, 2009).
3
Figure 3. Left: Space Invaders (Nishikado, 1978); Right: Pac-Man (Iwatani et al., 1980).
In the last three decades of the 20th century, the breadth of AI applications expanded beyond movies and videogames to encompass a variety of commercial uses. In reality, many AI products were commercialised during this period in an attempt by researchers to develop more human-like agents and robots. In 1997, IBM’s Deep Blue beat world champion Garry Kasparov in a chess competition while in the same year Microsoft embedded its Office Assistant in Microsoft Office for Windows. There are other human- like agents, such as Ananova, the newscaster and STEVE, the pedagogical agent (Johnson & Rickel, 1997).
Figure 4. Left: Ananova © Orange SA; Right: STEVE (Johnson & Rickel, 1997).
In the current era, Memex’s influence remains evident in the virtual characters developed for both the movie and gaming industries. Other contemporary applications include the Knowledge Navigator, which is a virtual assistant in Apple’s concept videos (Figure 5, top left) (Sculley, 1989). This influence was also clear in recent movie characters, such as Dr. Know, a virtual consultant in the movie Artificial Intelligence: AI (Figure 5, top right) (Spielberg, 2001), the virtual librarian in the movie The Time Machine (Figure 5, bottom left) (Wells, 2002), and a shop assistant in the movie Minority Report (Figure 5, bottom right) (Spielberg, 2002). These fully functional virtual characters represent the uppermost ability of what is currently imagined as artificial conversational agents.
4
Figure 5. Social actors as interfaces in film/video. Top left: The Knowledge Navigator in Apple’s concept videos (Sculley, 1989); Top right: Dr. Know in Artificial Intelligence: AI (Spielberg, 2001); Bottom left: a virtual librarian in The Time Machine (Wells, 2002); Bottom right: a virtual shop assistant in Minority Report (Spielberg, 2002).
Nevertheless, constant improvements in computing technology for operating high-speed arithmetic systems and the advancement of graphics technology for visualising animated virtual characters have facilitated AI researchers to implement their ideas for fantastical characters. In truth, many embodied conversational agents (ECAs) have been developed for various purposes in diverse social and technological contexts in the current period. ECAs encompass one type of intelligent agent represented by virtual characters, and possess the ability to interact with humans by means of language and gestures. One of the foremost researchers credited with the development of ECAs has been Justine Cassell. Her work has based upon utilising her knowledge of human conversations in ECAs, and the inception of several foundations of ECAs’ components, including speech, gestures, and facial expressions (Cassell, 2011).
These improvements have empowered individuals worldwide to interact with ECAs in different forms, such as the real estate agent REA (Figure 6, top left) (Cassell, 2000), relational agent Laura (Figure 6, top middle) (Bickmore, 2003), chatbot Anna (Figure 6, right) (Inter-IKEA Systems B.V., 2011) and ALICE (Figure 6, bottom left) (ALICE AI Foundation, 2008), and pedagogical agent Sam (Figure 6, bottom middle) (Ryokai, Vaucelle, & Cassell, 2003). In a similar vein, the aforementioned Microsoft Office Assistant represents the leading example of a software agent. This assistant aids users by way of interactive animated characters, which give access to the Microsoft Office help content.
5
Figure 6. Top left: REA (Cassell, 2000); Top middle: Laura (Bickmore, 2003); Right: Anna (Inter- IKEA Systems B.V., 2011); Bottom left: ALICE (ALICE AI Foundation, 2008); Bottom middle: Sam (Ryokai et al., 2003).
In another related development, embodied agents have further developed from their origins as low resolution creatures to become more complex high-resolution characters that can interact with people in real time digital games. Researchers have advanced these developments, most noticeably in the past two decades, in terms of their functional and operational speed. One of the most common areas of contemporary research has been the visual representation of agents. This field will form the starting point for this thesis. This interest may derive from humans’ first impressions of intelligent agents typically being by way of their visual representations. However, the question remains as to what extent the appearance of these man-made creatures is likely to determine the quality of the computer user’s interactions. This thesis, therefore, focuses on the appearance of embodied agents in order to investigate how affective interaction works between human computer users and ECAs.
The brief history offered here provides an overview of the development of ECAs from the imaginary Pandora to the practical Memex, and this build-up of ECAs could be described in both functional and visual ways. Laurel (1990) also has stated that computational agents are predisposed to behave on both functional and stylistic levels. In this way, the functional aspect of ECAs generally consists of AI while visual component typically comprises interface design. AI also provides ECAs with substantial aspects, such as high intelligence and several functional areas. Conversely, interface design personifies ECAs gestures and facial expressions and this research will mainly address this key area of ECA interface design.
6 Moreover, ECA interface design is primarily based on anthropomorphism, where anthropomorphising interface agents assist users to communicate naturally on a psychological level, and provide users with additional responsiveness and the capacity to perform actions in a functional way (Laurel, 1990). ECAs represent the main medium for users to naturally communicate with computational devices. This approach is analogous to conversing with real humans through gestures and language notwithstanding some remaining disputes about the precise appearance of ideal ECAs. While human beauty is an appealing characteristic, this may not be the optimal approach for all forms of ECAs. In fact, when ECAs are developed for varied tasks, such as information, work, learning and entertainment, the interface design of these ECAs differ according to their exact needs.
For the purposes of this thesis, ECA interface design will represent the core component of my research. This research intends to investigate the influence of ECA interface design on the affective interaction between humans and ECAs. It also intends to show how the interface design should differ according to the precise mission that each particular ECA performs in their different social contexts.
7
Chapter 1. Introduction
This research aims to investigate existing developments and design guidelines of affective interfaces of ECAs in terms of technological, psychological and design aspects. This will also facilitate the completion of missing components, such as affective influences among different hardware interfaces and character classifications of ECAs. This chapter addresses the key motivations that underpin the exploration of affective interactions between users and computers.
This chapter commences with introduction, which offers an overview of the entire research study. This is followed by five other sections as follows: 1) When Computers are Embodied Characters, 2) The Importance of Affective Factors, 3) What are Anthropomorphic Agents?, 4) Domain Analysis, and 5) Summary. The first three sections present the three core components of this research, namely agent interfaces, affective factors and ECAs. This chapter introduces these general relationships and possible research interests among these three core components. Subsequently, more related theories and discussions will be addressed in the following chapters that are from Chapters 2 to 4.
The issue of agent interfaces is the most critical focus of this research. Therefore, the first section of this chapter similar to the prologue presents a more provisional definition of agent interfaces and their importance. This research concerns human feelings when they interact with computational agents, and these are regarded as affective factors. The second section contains a discussion of certain affective factors that influence human minds and behaviours. Computational agents that interact with humans through natural language with embodied characters are referred to as ECAs, and these are one type of anthropomorphic agents. The third section presents the underlying reasons why agents simulate human appearances as well as offering additional examples of ECAs. The fourth section discusses hardware interfaces and character classifications of ECAs, and these are the primary domains that this research intends to explore.
This research also tackles alterations of human affective states when users interact with different ECA interfaces. For the purposes of this research, interfaces will be regarded as the medium that people use to interact with technological products. ECAs consist of a combination of embodied agents and natural language processing (NLP). As a consequence, humans interact with computational devices that are represented by ECAs by means of a natural and intuitive manner, and this enables easy communication via
9 natural language and gestures. With developments in computer technology and advancements in computer graphics, ECAs are expected to be increasingly employed in human-computer interactions (HCIs), in which affective interfaces of ECAs play a critical role. When ECAs are developed using different interfaces (such as those found in different devices with various characters), what components of the ECA interfaces cause the alterations of human affective states during HCIs? This work conducted two surveys to investigate the influence of these specific components of agent interfaces on human affective states. The results of two surveys were practically applied on a multi-agent website through better-designed agent interfaces.
Several ECAs with different interfaces were developed for experimental purposes to study ECA interface-related factors that contribute to user affective states. In the first survey (the hardware interface experiment), where ECAs act as talkative agents (named chatbots in this research) on different hardware interfaces (one on computers and the other one on portable devices), human affective factors were observed when subjects respond to agent interfaces. In the second survey (the character interface experiment), where ECAs act as learning companions agents (named CompanionBots in this research) in different classifications of characters (a human, an animal, and a creative creature), the human affective factors involved in a learning task were explored at length.
This research intends to answer the questions, which occur when humans interact with different ECA interfaces, for example (1) when ECAs appear on computers and portable devices, do human affective factors relate to the different hardware interfaces of ECAs? Or (2) when ECAs are represented by distinct character classifications, do humans respond to various levels of affective factors toward these ECAs in a learning task? Or (3) among preference, engagement and user-agent relationships, are there any relations between these affective factors during HCIs? The main contributions of this research can be summarised as follows: (1) the hardware interfaces of ECAs can be related to the affective responses of humans, such as emotional influences and user-agent relationships, (2) character classifications can be related to human affective factors in a learning task in terms of user engagement, and (3) there may be a significant association among character preference, user engagement and user-agent relationships. Additionally, (4) applying the experimental results and existing design guidelines into practical use on a multi-agent website.
The goal of this research is bettering user interfaces of ECAs and further improves the interactions between users and ECAs. These findings of this research did provide some
10 design guidelines and references for designers and developers for further ECAs developments.
1.1. When Computers are Embodied Characters
Agent interfaces are a core component of this research. The major issue that this study addresses is when ECAs are represented by various interfaces, which are computers depicted by different embodied characters. How do users respond to these ECAs affectively? Interfaces, specifically agent interfaces, represent the primary focus of this research. As mentioned in the prologue, various forms of interfaces have been represented by social actors. Computing interfaces are devices or programmes that allow users to communicate with computers. When computing interfaces are denoted by graphical icons, windows, and visual graphics, they are referred to as GUIs (Turk & Robertson, 2000). Moreover, an agent interface is an intelligent agent that is represented by one type of visible interface, and users are able to communicate with the agent through the interface (Shneiderman & Maes, 1997). In a situation where visible interfaces are symbolised in GUI forms, these visual representations are mostly anthropomorphic characters (Laurel, 1990). Similarly, when they are presented on hardware devices, they commonly exist in the forms of robots. For the purposes of this research, agent interfaces are defined as the external representations of intelligent agents, including GUIs and hardware interfaces.
With major developments of AI and computer graphics fields, studies on HCIs have emphasised the need to move beyond functional programmes in designing lifelike interfaces by studying target appearances and behaviours, such as facial expressions and gestures (Cassell, et al., 1999). Affective computing is another approach to AI interfaces (discussed further in Chapter 2). It not only develops certain computational functions, but also takes human emotions into account, especially those concerning affective reorganisation and expression during HCIs. Some studies related to affective computing built AI into specific devices, and examples of this include wearable computers that provide new methods to augment human abilities, such as boosting memory capability (Picard, 1997), and the robot Kismet, which is equipped with auditory, visual, and expressive systems designed to perceive and respond to human social cues as shown in Figure 7 (Breazeal, 2003). In truth, agent interfaces are the media that allow users to interact with agents, albeit the design of efficient agent interfaces is necessary irrespective of whether GUIs or hardware interfaces are employed. Negroponte (1995) 11 has proposed that agent-based interfaces should possess social and active functions to effectively help users in human-agent tasks. Since agent interfaces are the medium for users to communicate with agents, this might affect the flow of agent control. In addition, using human-like characters to resemble agents might impact human affective feelings during HCIs.
Figure 7. Social robot (Breazeal, 2003).
Nevertheless, when people interact with other human beings, physical attractiveness affects their beliefs and behaviours (Kalick, 1988). In addition, Reeves and Nass (1996) posit that people treat computers as social actors. Therefore, these phenomena have significant implications for agent interfaces. Similar to interpersonal behaviour, the physical appearances of agents also affect human beliefs and behaviours, given that the external features of agent interfaces attract or distract people, in the same manner as human appearances do. Gard (2000), the designer of the game Tomb Raider and its main character Lara Croft, has stated during a discussion about computer game character design that “a person’s first impression of a character will almost certainly come not from what they do, think, or say, but what they look like”. In truth, users typically grant greater attention to a character if the character has an appealing visual impression, and this verifies the aphorism that ‘first impressions last’. When this concept is applied to agent interfaces, their attractive visual features not only create a better impression in the mind of users, but it also encourages users to further interact with the agents.
Over the past few years, several studies have confirmed that agent interfaces influence user beliefs and behaviours in multiple ways. For instance, Baylor and Kim (2004) imply that agent interfaces that incorporate realism, gender, ethnicity, and instructional roles affect learners’ self-regulation and self-efficacy. Another frequently cited study by Koda and Maes (1996) has ascertained that a canine (dog/animal) character is generally more likeable than human characters, and a realistic (human) character can offer people comfort and the feeling of being more intelligent in their study. Moreover, in a similar review undertaken by Dehn and van Mulken (2000), incontrovertible proof that agent
12 interfaces affect user experience and behaviours was provided. They also lead to noticeable impacts on the outcomes of the user-agent interactions. It is worth noting that the effect of first impressions and the influence of agent interfaces over beliefs and behaviours of users occur both before and after interactions. Hence, agent interfaces comprise a critical element in the interaction between agents and users. This underlines that research on HCIs requires not only the development of programs that function effectively, but also the design of interfaces that positively influence interactions to ensure that they occur more smoothly.
This research conducted two surveys to establish the influence of ECA interfaces on users. Firstly, two chatbots with different hardware interfaces are employed to explore human affective factors as subjects respond to agent interfaces. A thorough review of contemporary literature has indicated the absence of studies, which compare agent interfaces on computers and portable devices to determine the distinct affective effects of agent interfaces on users. Secondly, various graphical characters are utilised to determine the precise human affective factors that are involved in learning tasks and the prevailing connection between character preferences, user engagement and user-agent relationships. In a similar research domain, Koda and Maes (1996) investigated various types of graphical characters (two realistic humans, two human caricatures, a line-drawn smiley, a dog caricature and an invisible man). However, these graphical characters were examined in the context of a poker game and not in an actual learning task. Furthermore, Koda and Maes (1996) failed to scrutinise the level of engagement or applied a similar graphic style to design agent characters. In a related field, a valuable study by van Vugt, Konijn, Hoorn, Keur, and Eliëns (2007) analyses the contribution of the effects of realistic agents on user engagement. Their research differs from this research because the agents used did not possess emotional expressions or offer interaction with users. In addition, Koda and Maes (1996) mainly address the realism of agent interfaces and not the character classifications of agent interfaces. Nevertheless, this research adopts the engagement measurement adjusted by van Vugt and his colleagues (van Vugt, Hoorn, & Konijn, 2004; van Vugt, Hoorn, Konijn, & de Bie Dimitriadou, 2006), namely I-PEFiC. In reality, agent interfaces will form one of the major issues in this study. Chapter 3 will address these issues in more detail and provide more related information about agent interfaces.
13 1.2. The Importance of Affective Factors
Affective factors represent another core component of this research, and typically address how humans feel and their underlying emotions. According to Russell (2003), affect as classified by psychologists generally covers feelings and emotions. Another commonly accepted definition of affect is as “an emotional response, a gut reaction, or sympathetic nervous activity” (Breckler, 1984, p. 1191). In addition, Corsisni (1999, p. 25) has described affective interaction as an “interpersonal relationship carried out on an emotionally-charged level”. As a result, it is patently clear that affect is intimately involved with human emotions. In fact, affective factors are those responses of human affected by human emotions and feelings, such as empathy, attitude, mood and motivation. Moreover, Forgas (2002) has stated that affect plays a strategic role in the thought processes and actions of humans in social situations. In truth, humans are not entirely rational or logical beings, given that feelings, moods, emotions, and other types of affective factors have a significant influence on people’s thoughts and behaviours.
In the context of affective factors, it is necessary to distinguish between two forms of affective factors, namely positive and negative affects. Just as the names imply, positive affects are positive human states and feelings when people experience upbeat emotions, such as happiness and pleasure. On the other hand, negative affects are those unenthusiastic human states and feelings, such as anger, sadness and fear (Diener, Larsen, Levine, & Emmons, 1985). Several pieces of research have already confirmed the association between positive affects and helpful influences on humans. One such study demonstrated that people in a positive mood performed better in early idea production (Kaufmann & Vosburg, 2002), and were more predisposed to promoting the exploration and enjoyment of new ideas (Isen, 1993). Even if people experience mild positive affects every day, positive states may have a systematic influence on cognitive processing (Ashby, Isen, & Turken, 1999). This is commonly manifested by people in positive states, thinking and executing actions according to their happiness and satisfaction levels. On the contrary, people in negative states address their individual states more rationally by seeking and using information to explain this particular situation (Schwarz & Clore, 1983). These above studies have proved that affective factors have a considerable impact on the thoughts of humans and are worthy of further investigation.
However, affective factors do not only impact human thinking, but also people’s daily routine and this has been depicted in several studies. In this domain, Winkielman, Knutson, Paulus, and Trujillo (2007) have investigated affective influences on judgment
14 and decisions with a biopsychological model and how they subsequently influence attention, memory, and perception. Similarly, Estrada, Isen, and Young (1997) discovered that physicians in an affect-group (as the experimental group) arranged information more efficiently than others in a control group, and did not engage in precipitate processing. Furthermore, affective factors influence people in their interactions with others, such as in bargaining and negotiations (Forgas, 2002). This underscores that affective factors influence humans in mental as well as physical aspects.
Affective factors not only affect humans in terms of their behaviour and thoughts, as they are also affected by external objects, events and other individuals. Norman (2002, p. 38) suggested that “pleasing things work better”, given users are attracted by beautiful products and express greater willingness to use those products. In fact, even a single external objective event can clearly alter humans’ affective states (Russell, 2003). These represent some of the reasons why Zhang and Li (2005) pointed out that utility and accessibility were not the only factors that Information Technology (IT) designers should pay attention to, as affective factors should be given due consideration.
In truth, affective factors have significant effects on wide variety of human experience, including attention, memory, judgment, decisions, bargaining, and negotiations. However, it is typically problematic to measure the affective factors of humans. This difficulty is often overcome through affective science 1 , which has several means for measuring human affective factors, in particular in the domain of emotions (Kring & Moran, 2008). As an illustration, it deploys physiological measures of skin conductance, muscle tension and hormone secretion. Additionally, self-reporting, which was adopted in two influential surveys, is the most widely employed by researchers to measure human affective factors (Larsen & Fredrickson, 1999). More theories and studies in relation to affective science will be addressed in greater detail in Chapter 4. It is critically important at this stage to ascertain users’ feelings during HCIs, because these affective feelings could further impact their mind, bodies, and interactions with agents. Further research related to affective factors will be elucidated in Chapter 4.
1 Affective science: Affective science is one type of scientific study that concerns emotions, moods, feelings and affects.
15 1.3. What are Anthropomorphic Agents?
ECAs form the other core component of this research, as well as being one type of anthropomorphic agents. This research aims to investigate the alterations of human affective states when users interact with ECAs that are represented by assorted affective interfaces. In order to improve HCI performance towards human-human interactions, computers are required to possess more human-like capabilities (Geven, Schrammel, & Tschelig, 2006). For example, computers are represented by visual characters as well as being designed with emotional facial expressions in order to express thoughts and communicate with humans naturally. Anthropomorphic agents generally are the computer systems displayed in human forms, and are also a type of affective computing.
Maes (1994) posits that building a successful embodied agent is likely to help users to understand the capacities, limitations, and operations of agents. In a similar vein, Koda and Maes (1996) claim that agents with faces and facial expressions were normally regarded as more likeable and attractive. In fact, adding a graphical character to an agent for expressing emotions makes users feel that they are interacting with an independent social actor (Sunder & Nass, 2000). Additionally, when computers are displayed in the form of social cues, users unconsciously apply social rules to the interactions with these agents (Nass, Moon, Morkes, Kim, & Fogg, 1997).
The benefits of the interfaces presented by anthropomorphic forms have been shown in several studies. Walker, Sproull, and Subramani (1994) found that enhanced user performance was realised when questions had been asked by a talking face rather than displayed in text on the screen. Moreover, Rickenberg and Reeves (2000) conducted a study in which personified characters monitor users on a website when they were working online. This study proves that monitoring of characters decreased user performance, although increased trust levels towards the website content. Besides providing information, ECAs are also exceptionally popular for applying anthropomorphic agents in the educational environment, such as in intelligent tutoring systems (ITS) and learning companion systems (LCS) (These concepts are addressed in more detail in Chapter 2). In this field, Moundridou and Virvou (2002) assert that personified interfaces in a learning environment enhance students’ learning experiences. In addition, when ECAs are applied to different domains, Koda and Maes (1996) state that personified interfaces not only improve user engagement in tasks, but also have applicability in the entertainment area. Even interfaces represented by anthropomorphic characters lessen the distance between humans and computers, given greater human
16 familiarity with interacting with computers that are portrayed by personified interfaces (Nass & Moon, 2000). However, there is a paucity of research indicating the specific impact of visual forms of agents on humans and better visual design for ECA interfaces. These reasons underline the necessity to undertake research that investigates how ECA interfaces may enhance HCIs.
1.3.1. Interfaces of Embodied Agents
Embodied agents are intelligent agents depicted by the visual forms, and anthropomorphic agents are one subset of them that possess human-like appearances. Recent developments in computer technology that have emphasised more social functions as well as advances in computer graphics have created more opportunities for embodied agents. Embodied agents are defined as those that “transform the experience of interacting with a computer, making it more social by explicitly referencing human- human interaction” (Zimmerman, Ayoob, Forlizzi, & McQuaid, 2005, p. 233). Nonetheless, not all embodied agents stimulate human external features. For instance, Clippit, the default Office Assistant (shown as Figure 8, left), is represented by a paper clip animation and appears whenever you click the help button in Microsoft Office (Dickinson College, 2007). A further example of this situation is Apple’s embodied agent Phil that possesses human-like representation, and Phil helped a professor to perform his work and life tasks in the Knowledge Navigator video (shown in Figure 8, right) (Sculley, 1989).
Figure 8. Left: Microsoft Office Assistant, Clippit (Dickinson College, 2007); Right: Knowledge Navigator, Phil (Sculley, 1989).
The aim of this research is to investigate the affective influences of agent interfaces on human affective factors, such as character preferences, user engagement and user-agent relationships. Several studies have shown that interfaces of embodied agents influence not only user engagement, but also other human affective factors. Takeuchi and Naito (1995) demonstrated that an agent with a face elicits greater engagement than one depicted by a 3-D arrow. Broadly comparable results were seen in Koda and Maes’
17 (1996) study, where an agent with a face is seen as more engaging than one without. In addition, a persona agent is more entertaining than the non-persona agent (van Mulken, André, & Müller, 1998) while an expressive agent has more credibility than a muted agent (Lester & Stone, 1997). In terms of the behavioural influence of agent interfaces, an agent with a face generally draws greater attention than a 3-D arrow (Takeuchi & Naito, 1995). Additionally, an expressive agent improves problem-solving outcomes more effectively than a muted agent (Lester, et al., 1997).
According to this research, it is evident that agents with visible appearances have affective effects on humans. In this research, the visual forms of agents deserve particular exploration. However, there remain limited studies conducted on interfaces with embodied agents compared to the amount of studies on other aspects of interface design, such as in websites, games, portable device applications and computer software. That said, the appearance of embodied agents continues to have a significant role in human affective factors. As a consequence, designers and researchers should give additional attention to better design of interfaces with embodied agents. This also highlights the need for this research to focus on the interface design of ECAs, and develop better interfaces, which could improve HCIs.
1.3.2. Embodied Conversational Agents (ECAs)
ECAs are intelligent agents equipped with conversational abilities to communicate with users via natural language and normally represented by anthropomorphic interfaces (Bickmore & Cassell, 2001). ECAs are the fine media for users to control agents. Additionally, Nass et al. (1994) state that humans willingly treat computers as other social actors. Moreover, ECAs have the potential to improve HCIs towards human- human interactions, and as a result this study deploys ECAs as the tool to investigate agent interfaces during HCIs.
ECAs have attracted substantial research and design consideration, particularly given their widespread applications, such as the Real Estate Agent (REA, as shown in Figure 9), which was designed by Cassell (2000). This enabled users to communicate with REA through natural speech in order to make queries or obtain real estate related information. REA, which is an advanced demonstration of ECAs, is not only displayed in a 3-D virtual character to express facial expressions, gestures, winks, nods, and sidelong glances, but is also equipped with a level of AI to understand natural speech (Cassell, 2000).
18
Figure 9. REA (Real Estate Agent) (Cassell, 2000).
In addition, ALICE is a well-known application of ECAs, and this adopted Artificial Intelligence Markup Language (AIML, discussed further in Chapter 5) as its conversational ability to communicate with humans via keyboard conversations. As shown in Figure 10, users type via an input frame and submit their entries, and ALICE can generate natural speech responses as well as offering relevant feedback. Furthermore, the eyes of ALICE can follow the cursor on the screen and depict ALICE in a more vivid form (ALICE AI Foundation, 2008).
Figure 10. Demonstration of ALICE (ALICE AI Foundation, 2008).
Another application of ECAs is Xiaoi, which is a popular chatbot in China, and was created to be executed on MSN Messenger in 2004 by the Incesoft Corporation. In fact, as shown in Figure 11, Xiaoi has already been developed for several different platforms, such as MSN Messenger, Yahoo Messenger, QQ, Google Talk, and other leading websites and applications. The convenience of these online systems has facilitated users to chat with Xiaoi anytime and anywhere via the Internet. Furthermore, Xiaoi has been developed to possess various functions, the most common of which is chatting while others include movie and music searches, meal ordering, translation facilities and other customer-based services. Additionally, Incesoft has published a desktop version of Xiaoi to allow more rapid, easier and intuitive use (Incesoft, 2008).
19
Figure 11. Web version of Xiaoi (Incesoft, 2008).
In addition to developing functional and various ECA applications, researchers have determined that ECAs have a positive effect on HCIs. In this field, a study applied ECAs to a memory performance task and proved that ECAs have a positive effect on retainable human memory (Beun, de Vos, & Witteman, 2003). In the same way, ECAs have a positive impact on humans’ impression of these representations because ECAs are equipped with the persona effect2 (van Mulken et al., 1998). Moreover, anthropomorphic interfaces on ECAs enable HCIs to perform as humans in face-to-face conversations. Further ECA-related background and information about functional agents will be presented in Chapter 3.
Although some ECA applications exist that have created and possess several design principles for developers and designers to follow, ECAs remain imperfect if developers apply all of those principles. This derives from other factors that require consideration, such as user demands and the precise missions conducted by ECAs (Tatai, Csordás, Kiss, Szaló, & Laufer, 2003). Furthermore, knowledge about how to improve user engagement by designing agent interfaces in learning tasks remains relatively limited. Thus, this field was given closer scrutiny in this research. Firstly, the hardware interface survey studied the human affective factors observed when subjects respond to different agent interfaces. Secondly, the character interface survey explored the human affective factors involved in learning tasks that might contribute to user engagement. Additionally, this research designed ECAs with different interfaces in order to investigate the relationship between various agent interfaces and users.
2 The persona effect has been defined as “the presence of a life-like character in an interactive learning environment … can have a strong positive effect on student’s perception of their learning experience” (Lester et al., 1997a).
20 1.4. Domain Analysis
This research focuses on investigating whether human affective states are affected by the affective interfaces of ECAs via hardware interfaces and character classifications. In interpersonal interactions, people’s feelings are generally affected by other people’s physical attractiveness. Similarly, agent interfaces determine human affective feelings during the interactions between users and agents, including character preferences and user engagement, and feelings further influence the relationship between users and agents.
1.4.1. Research Questions
The purpose of this research is discover existing developments and design guidelines of the affective interfaces of ECAs, ascertaining potentially missing components of existing theories, and ameliorating ECA interfaces to improve user engagement. This research will attempt to discover certain aspects about the affective influences of agent interfaces among different hardware interfaces and character classifications. The most important question is when ECA interfaces are represented in different hardware interfaces, with various character classifications, are there any variations in human affective states (character preference, user engagement, user-agent relationships and emotional influences) during HCIs?
Although the pervasive influence of computers in contemporary society has led to widespread usage of ECAs, portable devices are another type of technological devices employed to address daily tasks. This raises the question about what is the impact of agent interfaces on users in these varied domains. For instance, when ECAs appear in computers and portable devices, do ECAs have different affective influences on users (such as emotional influences and user-agent relationships)? In order to study these phenomena, two chatbots were developed for different devices (one for computers and the other one for portable devices) in the first survey. This survey aims to reveal human affective factors of users when they respond to these two agent hardware interfaces. When one chatbot appears on computers and the other one on portable devices, what are the differences of human affective factors as subjects/users responded to these two agent interfaces? Furthermore, what are the differences of emotional influences of these two agent interfaces on users? And what are the differences of relationships between these two agent interfaces and users?
21 Several studies (already mentioned in the previous sections) have found that users have different levels of affective responses towards different genders, ages and races of ECAs. However, few studies have addressed the classifications of characters, particularly in the performing in learning tasks. When beginner English learners undertake a language practice task, do they have different affective responses (such as character preference, user engagement and user-agent relationships) towards distinct character classifications of ECAs (humans, animals and creative creatures)? In other words, do learners possess differential character preferences toward distinct character classifications of ECAs? Do learners engage on different levels with distinct character classifications of ECAs? Moreover, do learners build different levels of user-agent relationships with distinct character classifications of ECAs? Additionally, what is the relationship, if any, among different affective factors during HCIs, such as character preferences, user engagement and user-agent relationships? In the second survey, three classifications of characters were designed as CompanionBots, namely a human, an animal and a creative creature. This survey intends to explore the affective influences of agent interfaces on human affective factors, such as character preference, user engagement and user-agent relationships. When CompanionBots appear with different classifications of characters, what are the human affective factors that are affected by agent interfaces? Furthermore, how does user engagement relate to character preference for agent interfaces? Finally, how does user engagement relate to user-agent relationships?
1.4.2. Research Hypotheses
Several studies mentioned in the review of Dehn and van Mulken (2000) confirmed that the appearances of ECAs do impact user experiences, behaviours and performance. Therefore, this research will hypothesise that ECAs, which are represented by different interfaces, such as those found with different hardware interfaces and various character classifications, are likely to influence human affective states differently.
In response to the research question concerning the affective influences of different ECA hardware interfaces, are examined based on the characteristics of portable devices and computers in terms of: 1) input devices, 2) output devices and 3) portability. This research will hypothesise that agent interfaces appearing on different hardware interfaces possess distinct affective influences on users, and that computer-agents have greater emotional influences on users because of the larger screens, and that portable device- agents build better relationships with users due to their convenience. 22 Moreover, in response to the research question in relation to the human affective states affected by different classifications of agent characters has been verified by several important research studies. This research will hypothesise that character classifications of ECAs are likely to affect human affective factors during learning tasks. The specific details about how distinct character classifications of ECAs affect character preference, user engagement and user-agent relationships differently form an important component of the investigations in this research. Additionally, there should be some connection between character preference, user engagement and user-agent relationships because affective factors are closely linked to other factors. This research will hypothesise that the level of user preference for an agent character directly impacts the degree of user engagement in interactions with the agent character. Hence, this leads to the development of better relations between users and agent characters.
In order to explore these questions that occur during the interactions between humans and agents, there are two particular domains that this research proposes to analyse. The first domain investigates the affective influence of ECAs on users (such as emotional influences and user-agent relationships) when ECAs are applied to computers and portable devices. The second domain explores the variations of human affective states when ECAs are represented by different character classifications. In the second domain, the relationship between user engagement, character preference and user-agent relationships are also studied. Two surveys were conducted to determine answers of these research domains.
Firstly, ECAs were employed as chatbots with two hardware interfaces (for computers and portable devices) to investigate the main components of agent interfaces that influence human affective factors. This experiment investigated some underlying human affective factors, such as emotions and relationships with ECAs, which were involved in the interactions with ECAs on different hardware interfaces.
Secondly, ECAs were used as CompanionBots with three classifications of character interfaces (a human, an animal, and a creative creature) to determine human affective factors that relate to user engagement as well as the components of agent interfaces that are involved in user-agent tasks. This experiment investigated human affective factors to determine the interactions between users and agents, such as involvement, distance, users’ willingness to spend time with agents, learning attitudes, and others that might be related to user engagement, including preference for and relationships with agents.
23 Two types of ECAs were developed to address these specific areas of investigation (shown in Table 1). Gary and Gavin were used as chatbots in the first survey. Subsequently, Andrew, Ben and Chris were used as CompanionBots in the second survey. In this endeavour, not only were the different agent interfaces affecting various human affective factors investigated, but the strategies in improving user engagement were also explored by designing agent interfaces. A multi-agent platform was developed for beginners to practice language online. The results of two surveys were adopted to design the agent interfaces in the platform intended for improving user engagement.
Table 1. ECAs developed for these two surveys.
ECAs Gary Gavin Andrew Ben Chris Experiment The hardware interface The character interface experiment experiment Variable Computer Portable Human Animal Creative device creature Task domain Talkative agents: chatbots Learning companion agents: CompanionBots Evaluation Emotional influences and Character preference, user engagement and user- user-agent relationships agent relationships Discussed in Chapters 5–6 Chapters 7–8
Table 2 provides a summary of overall research questions, and research hypotheses that are derived from these questions. Additionally, the table lists the research findings according to these questions and more related research finding will be addressed more detail in Part ϩ and Part Ϫ.
Table 2. A summary of research questions, hypotheses and findings. Item Content RQ When ECA interfaces are represented in different hardware interfaces, with various character classifications, are there any variations in human affective states during HCIs? RH ECAs, which are represented by different interfaces, are likely to influence human affective states differently. RF Two experiments proved that ECAs influences human affective states differently when they are represented by distinct interfaces. RQ1 Do ECAs have different affective influences on users? If any, what are the
24 differences of emotional influences and user-agent relationships? RH1 The computer ECA has greater emotional influences on users because of the larger screen, and the portable device ECA builds better relationship with users due to the convenience. ġ RF1 The hardware interfaces of ECAs may be related to affective responses of users. RQ2 Do beginner English learners have different affective responses toward distince character classifications of ECAs? RH2 Character classifications of ECAs are likely to affect human affective factors during learning tasks. RF2 Character classifications may be related to human affective factors. RQ3 What is the relationship among different affective factors during HCIs? RH3 There should be some connection between character preference, user engagement and user-agent relationships because affective factors are closely linked to other factors. RF3 There are significant relationships between character preference, user engagement and user-agent relationships.
RQ: Research Question; RH: Research Hypothesis; RF: Research Finding.
The purpose of this study is on developing better ECAs interfaces to improve user-agent interactions, such as emotional influences, user engagement, character preference and user-agent relationships. The domains of giraffe knowledge in the hardware interface survey and language learning tasks in the character interface survey are the experimental contents adopted to attract subjects to interact with ECAs and further experience the distinct ECA interfaces.
1.5. Summary
In this chapter, the motivations of designing better agent interfaces, which include looking for the variations of human affective states, are addressed. The connections between agent interfaces, affective factors, and ECAs in this research are also provided, and citing how important agent interfaces and affective factors are to the HCIs. These aforementioned factors should be considered when designing and developing ECAs.
The remainder of this thesis is divided into three main sections, namely a theoretical, experimental and practical part. The theoretical part involves a study of the affective interfaces of ECAs (Chapters 2 to 4), which contains literature reviews from technological, design and psychological perspectives. The experimental part includes 25 surveys on hardware and character interfaces of ECAs (Chapters 5 to 8), which describe the details of the two experiments. Moreover, the practical part encompasses the implications of adopting experimental results and existing guidelines to build a multi- agent platform as well as conclusions and possible further research in this field (Chapters 9 and 10).
Chapter 1. Introduction
This chapter outlines how the motivations of improving agent interfaces - by designing agent interfaces using the investigation on human affective factors as subjects responding to agent interfaces - can be framed. It presents some important and provisional definitions of the three core components of this research, namely agent interfaces, ECAs and affective factors.
Part Ϩ. Study of affective interfaces of ECAs
Chapter 2. From AI to ECAs
The core component, ECAs, is one substantial part of AI. This chapter presents the literature review for the technological component of this research. It presents the intelligent and functional parts, which relate to ECAs, including a brief history of AI, introductions about intelligent agents and affective computing, and studies in relation to ECAs and other agents.
Chapter 3. Interface design of agents
The core component, agent interfaces, is the visual part of ECAs. This chapter describes relevant academic theories and practices for the interface design aspect of this research. It offers a brief history of user interfaces, an introduction of agent interfaces, an evaluation of the importance of visual impact of agent interfaces and a description of some methodologies of the related agent interface studies.
Chapter 4. Affective factors of intelligent agents
26 One focus of this research is the human feelings towards agent interfaces, and these feelings are affective factors, which also is the core component of this research. This chapter presents a literature review for the psychological part in relation to the affective factors of this research. It also provides an introduction of affect and affective display, a thorough discussion of affective science and affective agents, and the relationship between affective factors and agent interfaces.
Part ϩ. Surveys on hardware and character interfaces of ECAs
Chapter 5. Method of hardware interface of chatbots
This chapter presents the hardware interface survey, which explores users’ feelings toward different ECA hardware interfaces. The methodology and development of the hardware interface survey are addressed in this chapter. It also introduces the instruments applied to this research, such as AIML and emotional expressions.
Chapter 6. Evaluation of hardware interfaces of chatbots
This chapter summarises and analyses the experimental results of the hardware interface survey.
Chapter 7. Method of character interfaces of CompanionBots
This chapter presents the character interface survey, which investigates the affective influences of different character classifications of ECA interfaces. Furthermore, the design and development of the character interface survey are addressed in this chapter. It also provides the measurements and materials applied to this survey as well as character design of the CompanionBots.
Chapter 8. Evaluation of character interfaces of CompanionBots
This chapter summaries and analyses the experimental results of the character interface survey.
Part Ϫ. Adopting results to build a multi-agent platform
27 Chapter 9. Development of a multi-agent website with affective interfaces
After conducting two surveys to investigate the affective responses of users towards ECA interfaces, this research adopted the existing design guidelines and experimental results to develop a multi-agent website. This chapter clarifies the design and development process of the website.
Chapter 10. Conclusions and future work
In the end of this research, this chapter summarises the results of the entire research, discusses possible implications of the results for a number of disciplines and charts the direction of future work for affective interface design of ECAs.
This chapter grants a general start for this research, by discussing the roles of these three core components in this research. In the following chapters, there will be more theoretical and experimental content related to these components. These subsequent chapters also address how affective interfaces influence humans on both mental and behavioural levels.
28 Part I. Study of Affective Interfaces of ECAs
The previous chapter addresses motivations for investigating the affective interfaces of ECAs and generates research questions and hypotheses. The thesis comprises three sections, namely a theoretical, experimental and practical part. This part addresses core theories and mainly investigates existing research related to the three fundamental components, specifically ECAs, agent interfaces and affective factors. In the course of this research, literature reviews of technological, design, and psychological factors thoroughly explore these three core components.
There are three chapters in this part. Technology-related studies are presented in Chapter 2 to deliver general background information about ECAs, including a history of AI and ECA-related subfields of AI as well as introducing intelligent agents, affective computing and ECAs. In Chapter 3, a literature review of agent interfaces is presented. This chapter also contains an introduction and assorted definitions of various user interfaces, a study of agent interfaces, statements of the importance of visual impacts as well as methodologies of agent interface studies. Subsequently, a literature review of other affective factors is undertaken in Chapter 4 as well as introducing the value and meaning of affect, affective science, affective influence and affective agents.
In the following two parts, two surveys are presented, which explore the hardware and character interfaces of ECAs, including the methodologies, procedures, and results of these two experiments, and these encompass the experimental part running from Chapter 5 to 8. In the practical part, the practical use of agent interfaces as well as possible contributions and conclusions of this research are discussed in Chapters 9 and 10.
29
Chapter 2. From AI to ECAs
This research intends to design better ECA interfaces in order to improve user engagement during HCIs. This chapter conducts a literature review of the three core components of this research in the ECA domain. ECAs form one specific application of Artificial Intelligence (AI), and AI provides ECAs with the computational functions to execute their missions. The relationship between AI and ECAs forms the major part of this chapter while other aspects include interface design of agents and affective factors, which will be addressed in the next two chapters.
This chapter undertakes a technological literature review in the field ranging from AI to ECAs. This chapter has been divided into four main sections as follows: 1) Brief Introduction of AI, 2) Intelligent Agents, 3) Affective Computing, and 4) Embodied Conversational Agents. Since ECAs have originated from AI, the first section discusses the development of AI from virtual imaginations to practical applications. Because ECAs represent only one type of intelligent agents, the evolutions as well as the categories of intelligent agents are established in the second section. The ECAs utilised this research are similar to affective computing, and both of them are involved in human emotions, therefore, the third section discusses certain issues and ideas related to affective computing concerning the combination of emotions and AI. The fourth section presents the core component, ECAs, which are agents that integrate natural language processing and visual characters.
Since ECAs have originated from AI, this chapter appraises previous studies related to AI and ECAs as well as reviewing those technological elements, which were employed in the subsequent surveys. Other academic studies in relation to agent interfaces and affective factors will be presented in Chapters 3 and 4.
2.1. Brief Introduction to Artificial Intelligence (AI)
AI represents the foundation of ECAs and better understanding of the components and origins of AI will enable the detection of possible future developments of ECAs. The following sub-section gives a general introduction of AI related history, elements, functions, applications and markets. Artificial intelligence (AI) may be defined “as the branch of computer science that is concerned with the automation of intelligent behavior” (Luger & Stubblefield, 1998, p. 1). Given that AI is a component of computer science, it
31 is ordinarily applied to computers or other machines to execute activities that require human intelligence, such as reasoning and learning through experience (Adam, 1998). Moreover, AI has been employed in various fields, including expert systems, social intelligences, natural language processing, general intelligence and robotics (Thrun, 2002).
AI has become an important field of contemporary research and modern technology, notwithstanding that it has been studied for many decades and had been created from human imaginations in previous centuries. However, it was not until World War ϩ that work in the AI field started to intensify and be applied practically (Mahoney, 1988). In earlier eras, numerous terms were employed to refer to machines based on human intelligence, although John McCarthy first coined the phrase Artificial Intelligence in 1956 (Nilsson, 1998). In fact, multifarious definitions to describe how AI works and can be applied have been offered. Russell and Norvig (2003) organised AI into four separate systems, namely systems that think like humans, systems that act like human, systems that think rationally, and systems that act rationally. This important piece of research has posited that the entire spectrum of skills and actions of AI are developed in order to achieve rationality. While AI is one of the most innovative areas of research, it has existed within the human imaginations for a long period analogous to the ECAs, which were mentioned in the prologue.
2.1.1. AI Come from Human Imaginations
Humans are creatures with a full range of imagination, and these creative abilities have inspired researchers and scientists to design superior and more beautiful living environments. AI is one specific example of a creation spawned from human imaginations, which has subsequently been implemented practically (Haynes, 2003). In truth, the Greek gods are the earliest examples of the formation of artificial creatures. These creatures were conceptualised in the mythology of Hephaestus where Pandora was the first man-made being. The origin of this creature is when Zeus ordered Hephaestus to create Pandora to punish mankind for accepting Prometheus’s theft of the secret of fire. Accordingly, such types of mythologies have acted as an inspiration for human thoughts.
In 1818, British author Mary Shelley wrote a novel entitled Frankenstein about a scientist, Dr. Frankenstein, who created man-made beings. Frankenstein intended to build a creature in our likeness, albeit larger and more powerful than the average human. In this 32 novel, Frankenstein’s monster did not have a name, although people have generally tended to refer to the monster as Frankenstein. In fact, the tale of Frankenstein has been passed down from generation to generation, and it has inspired many creative works in various areas, such as in movies, music and novels.
Nevertheless, popular culture has typically associated high tech AI with robots even robotics is but one subdivision of AI, and this has permeated the public consciousness (Thrun, 2002). Contemporary robotics remains greatly influenced by Isaac Asimov’s Three Laws of Robotics (Craig, 1986). In reality, Asimov’s laws have deeply influenced other writers and thinkers in their treatment of robotics while many of Asimov’s stories have also been made into films, such as Bicentennial Man (Columbus, 1999) and I, Robot (Proyas, 2004). As the mythology of robots appears frequently in the film industry and popular culture, scientists and researchers have begun to pay more attention and interest to the development of AI (Bartneck, van der Hoek, Mubin, & Mahmud, 2007). One of the most advanced manifestations of AI depicted within film may be The Matrix where an intelligent machine controls the whole of mankind and employs humans as an energy source (Wachowski & Wachowski, 1999). In fact, countless AI depictions exist throughout the entertainment industry. The prologue and this chapter has already pointed out that ECAs are applications of AI and both of them are generated from human imaginations. This underscores the necessity to gain further knowledge about the history of AI in order to better understand potential developments in the ECA arena.
2.1.2. Early History of AI
Despite the fact the concept of AI has been conceived from human imaginations, scientists and researchers have endeavoured to apply AI practically.
Before AI research was firmly established, many people attempted to build intelligent devices, such as the Turk, a one-of-a-kind chess-playing machine (Buchanan, 2005). In the period between 1940 and 1950, various researchers began to explore the possibility of creating AI for computers or machines, specialising in range of fields, such as mathematics, philosophy, psychology, computer engineering, linguistics and economics (Buchanan, 2005). This research led to Warren McCulloch and Walter Pitts creating the first recognisable AI work in 1943 (Russell & Norvig, 2003).
33 In 1950, a mathematician and logician named Alan Turing promoted the Turing test as a measure to examine machines’ level of true intelligence (Buchanan, 2005; Russell & Norvig, 2003). In the same year, Marvin Minsky and Dean Edmonds built the first neural network computer, the SNARC (Russell & Norvig, 2003). Thereafter, at Dartmouth, John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon organised an AI workshop that further propagated AI into different fields of research and encouraged cross-faculty development (Russell & Norvig, 2003). Later, McCarthy moved to MIT and defined the high-level language, Lisp, as well as publishing a seminal paper entitled Programs with Common Sense (Negnevitsky, 2002; Russell & Norvig, 2003). In 1958, McCarthy was joined by Minsky at MIT with whom he co-founded the MIT AI Lab (Negnevitsky, 2002; Russell & Norvig, 2003).
Prior to the first AI winter, which was a period that endured from the mid 1960s to the mid 1970s, Joseph Weizenbaum built an interactive program called ELIZA, and this program has had a long-term influence on gaming (McCorduck, 2004). This program had been inspired by psychotherapy research and conversations between therapists and patients. In 1967, Edward Feigenbaum, Joshua Lederberg, Bruce Buchanan, and Georgia Sutherland developed Dendral, the first domain-specific knowledge program at Stanford (Negnevitsky, 2002; Russell & Norvig, 2003). The particular work was designed to solve programs and identify unknown organic molecules by a mass spectrometer.
Throughout the early 1980s, AI entered the commercial age with the first commercial expert system, R1, being successfully marketed (Russell & Norvig, 2003). Meanwhile, Japan’s Ministry announced the Fifth Generation Computer Systems (FGCS) project that aimed to develop a supercomputer with AI capabilities. In response to Japan’s FGCS, the United States established the Microelectronics and Computer Technology Corporation (MCC) (Russell & Norvig, 2003). However, both projects failed to achieve their original goals, and by the mid 1980s, a second AI winter began.
The 1990s witnessed further development of AI research largely thanks to developments in related areas of technology. Furthermore, researchers started to build a complete agent, such as the 1993 creation of the first mobile robot, Polly, by Ian Horswill at the MIT AI Lab (Horswill, 1993). Polly possessed the ability to move at animal-like speeds and navigate through use of computer vision. In addition, the Deep Blue chess program defeated the reigning world chess champion, Garry Kasparov, in 1997 and this marked a milestone in AI history while proving AI could exceed human intelligence (Goodman & Keene, 1997; Nilsson, 1998).
34 Although AI has not been limited to robots, the general population continues to pay greater attention to these manifestations of AI. In 1997, Sony introduced its first AI pet, AIBO. Similarly, Honda developed an AI human-like robot in 2001 named ASIMO, which resembled an astronaut wearing a backpack, and has the capability to walk or run on two feet (Sakagami et al., 2002). More recent research on intelligent robots not only focuses on creating robots with human-like skills and intelligence, but also centres on the entertainment and personal amusement sphere.
2.1.3. Main Subfields of AI
AI is a multi-discipline research field, which requires different demands, and it is based on a number of foundations, including philosophy, mathematics, economics, neuroscience, psychology, computer engineering, control theory, cybernetics and linguistics. Similarly, as different approaches, applications, majors, and perceptions from different fields contribute to the area of knowledge, AI has developed into several subsets. The most notable subfields are Natural Language Processing (NLP), General Intelligence (known as Strong AI), Social Intelligence (also called Affective Computing), Expert System, Robotics and Machine Learning. The following sections address two ECA- related subsets of this research.
Natural Language Processing (NLP)
ECAs are renowned for their ability to communicate with users with natural language because they are equipped with Natural Language Processing (NLP) (Sing, Wong, Fung, & Depickere, 2006). NLP is a system that computer scientists have developed to simulate human language processing. NLP has been defined as “several level and sub-processes are more and less simultaneously engaged in the analysis and the generation of natural language” (Handke, 1995, p. 47). Designing machines to solve problems of automated generation and understand natural human languages is the major research area of NLP (Nilsson, 1998). Manning and Schütze (2000) state that there are two approaches of NLP, one, which is a rational approach and the other, which is an empiricist approach. The former works by duplicating the functions of human brains, similar to human intuitions in response to language while the latter approach focuses on specifying the appropriate general language model, through the use of linguistic performance.
35 NLP has several main goals in this research, of which Statistical Natural Language Processing represents one of the most challenging tasks. Statistical NLP is a linguistic approach to solve the difficulties of long sentences by using stochastic, probabilistic, and statistical methods (Manning & Schütze, 2000). Moreover, Information Retrieval (IR) is another subset of NLP, which entails “finding documents that are relevant to a user’s need for information” (Russell & Norvig, 2003, p. 840). It utilises search information from related databases or even the World Wide Web. Machine Translation (MT) is broadly known as mechanical translations of text or speech from one natural language to another, and is another subset of NLP and defined as “the automatic translation of text from one natural language (the source) to another (the target)” (Russell & Norvig, 2003, p. 850).
In addition to those subsets mentioned above, there are also other areas, such as Question Answering, Speech Recognition, Information Extraction, Automatic Summarisation, and Optical Character Recognition. However, the main purposes of NLP are natural language generation and natural language understanding. The former functions by converting information from computer databases into natural human language and represents the decision-making process used to conceptualise something during NLP. Alternatively, natural language understanding mainly works by converting sources of human language into more formal depictions that are easier for machines to manipulate (Moore & Swartout, 1991). These two purposes also form the primary goals that ECAs intend to achieve. In this endeavour, ECAs, in this research, have applied one type of NLP, namely AIML (discussed more in Chapter 5), as the language database for users to communicate through natural language.
Social Intelligence
Social intelligence differs from other subsets of AI, as it involves human emotions. In fact, most social intelligence research addresses emotional and social skills, which humans apply to interact with others in social situations. Miura and Matsumura (2009, pp. 140- 141) have described social intelligence in social intelligence design (SID) as being “grounded in acquisition of abilities through interaction with the environment and the development/evolution within this process. These abilities are not only learned from public teachings, but also from the everyday environment”. Furthermore, Miura and Matsumura (2009) contend that social intelligence is an applied research field for engineering researchers and social scientists to cooperate and implement in the real world. 36 Social intelligence, similar to affective computing, is a multi-discipline research area spanning several fields, including computer science, psychology and cognitive science. The obvious difference with affective computing from other AI research areas is human emotions. While scientists previously developed computers for practical purposes, such as problem-solving, reasoning, and algorithms, contemporary researchers aim to build computers equipped with human emotions. The rationale for this new direction in development and research involves a number of factors. Firstly, the manner in which humans interact with computers is not simply on the basis of it as a machine, but also as a social actor (Nass et al., 1994). Secondly, researchers intend to improve the interaction between humans and computers to achieve more natural and comfortable ways. Therefore, in order to advance HCIs, affective computing can recognise the emotions of users and in turn express emotions. The ECAs, in this research, were designed with affective emotions, and they form one type of affective computing. The major issues relating to affective computing will be addressed in the section on Affective Computing.
2.1.4. Applications of AI
AI is a broad area of research that typically utilises technologies and theories from other fields. AI researchers have developed various tools and programs to solve numerous intractable contemporary problems and created various inventions to augment daily life.
Recently, the emotional dinosaur pet toy, Pleo (as shown in Figure 12, left), has attracted popular attention, as it is an autonomous pet toy that can feel hunger, fatigue and respond to its master as well as using its body to express actions, such as grazing, napping and waddling (UGOBE Inc, 2008). In truth, AI is generally impressive and well-received when it is applied to the domains of education and entertainment, especially amongst children. The first robot pet, Furby (as shown in Figure 12, middle) was released by Tiger Electronics in 1998, and it became popularised due to its ability to speak some words, interact with its master, and even fall asleep (Tiger Electronics, 1998). Later, Sony released a robotic dog, AIBO (as shown in Figure 12, right), equipped with intelligence and with the ability to walk and recognise the environment through camera use (Sony, 1999). The success of these and other products has resulted in increased use of AI in entertainment products.
37
Figure 12. Artificial toys; Left: Pleo (UGOBE Inc, 2008); Middle: Furby (Tiger Electronics, 1998); Right: AIBO (Sony, 1999).
Another common area of AI applicability is within business and commerce. This has been applied where shopping websites use AI programs to suggest products to their customers. For example, when a customer selects a book on Amazon.com, there will be list of ‘Customers Who Bought This Item Also Bought’ below the introduction of the book (Amazon, 2011). Other commercial applications include banks, which have implemented AI systems to arrange their operations, such as investment, insurance and account management (Chou & Chou, 2000). Another industry that has deployed AI programs is the aviation sector, for example Air Operations Division use AI for training simulators (Heinze, Smith, & Cross, 1998).
AI applications have not only been developing with programs, but also with robots. However, AI robots are used in a number of other fields. For instance, robot arms are utilised in construction industries due to their effectiveness for conveying bulky objects (Yokoi, et al., 2003). Mobile robots are also employed in hospitals to help deliver food or medicine (Evans, 1994). In fact, the increasing application of AI products in contemporary life has raised convenience levels and comforted human minds in mental manners.
2.2. Intelligent Agents
ECAs are one type of intelligent agents, which possess communication abilities and visual representations. An intelligent agent is an AI program that can operate autonomously to gather information or perform some service without users’ immediate presence. In truth, as mentioned in the prologue, intelligent agent research had been initially inspired by Bush’s Memex and this is commonly regarded as the field’s foundation (Detlor, 2004).
38 While an agent is generally thought of as a representative who acts on behalf of other persons, in terms of intelligence it is defined as “anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators” (Russell & Norvig, 2003, p. 32). Brenner and Zarnekow (1998) imply that agents can be categorised into three major categories, namely human, software and hardware agents. The duties of human agents mainly entail helping humans to address a particular task, such as travel agents, which assist customers to book flights and hotels, and organise visas. Similar to human agents, software and hardware agents can execute a number of tasks. Intelligent agents are one type of program analogous to software agents or otherwise defined as “a computer system that is capable of flexible autonomous action in order to meet its design objectives” (Jennings & Wooldridge, 1998, p. 4). In this research, the term agent will be employed as an abbreviated form of intelligent agent.
2.2.1. Categories of Intelligent Agents
Intelligent agent applications have grown rapidly in various fields and service provision areas. Most contemporary intelligent agents are based on different programs and goals, and can be categorised in various areas. Some categories related to affective computing and ECAs will be presented in the following sections.
In general, the most well known category of intelligent agents is software agents. These agents are similar to intelligent agents in that both are programs with a degree of intelligence for a particular purpose, and equipped with the abilities to interact with their environment. Brenner, Zarnekow and Wittig (1998) have defined a software agent as a software program that was developed for a specific task and one that exhibits intelligence. Additionally, software agents can be classified into several types depending on their different task areas. The first type is an information agent designed to search for information according to user requests. The second type is a cooperation agent that aims to solve problems by using communication and cooperation methods. The third type is a transaction agent whose functions including processing and monitoring transactions.
Russell and Norvig (2003) outline intelligent systems that are grouped into four categories of agents, namely simple reflex, model-based reflex, goal-based and utility- based agents. All the above agent programs have the capacity to convert into learning agents, which can absorb information from their environment and users. This can be achieved by a number of methods. Firstly, simple reflex agents can be developed with
39 limited intelligence, and they only decide actions on current perception and ignore the remainder of perception history. Secondly, model-based reflex agents mostly perform internal functions depending on the perception history, although they react according to the perception history, and also update internal state information of current state to facilitate better decision-making. Thirdly, goal-based agents decide what should be done with reference to their goal information. Finally, utility-based agents are equipped with utility functions for making rational decisions, and if there are conflicting goals, the utility functions of utility-based agents will identify the appropriate trade-off in all situations.
Nowadays, intelligent agents are operated in most digital fields, such as in business, education, entertainment and various industry sectors. Scientists have developed various categories of intelligent agents based upon diverse purposes and theories. In general, most applications of intelligent agents are not chosen at random, though they usually merge intelligent agents for better utilisation.
2.2.2. Applications of Intelligent Agents
ECAs form one major application of intelligent agents, and there are many intelligent agents having significant usefulness for information systems as well as improving the performance of various research fields. According to dissimilar objectives and foundations, the applications of intelligent agents can be grouped into commercial intelligent agent applications and personal intelligent agent applications.
Intelligent agents when applied to the business domain are regarded as commercial intelligent agent applications. In fact, buyer agents are one type of commercial agents, which have the ability to use a search engine to search for product information as well as provide users with the price and product supplier data (Brenner et al., 1998). Some product-related websites provide buyer agents for users to search information about certain goods. Similar to buyer agents, a price watcher is another commercial agent application. Fong, Sun, and Wong (2001) developed price watchers to retrieve competitors’ product prices through the Internet while also aiding rivals to monitor competitors’ pricing strategies without their knowledge or attention. In addition, Bensaid and Mathieu (1999) designed a shop navigation system that utilised multi-agent technology for navigation in a virtual shop. The shop navigation system not only guides
40 customers to shop in the virtual space, but also built a new system to monitor and learn customers’ behaviours.
In contrast to commercial agents, intelligent agents that are used in private areas are referred to as personal intelligent agent applications. Search engines are the most well- known application of this form of intelligent agent, and typically employ specific information or knowledge search according to user queries. This generally works by users selecting keywords for search engines, which then will collect keyword-related information and knowledge on the Internet or on its database. For instance, Google (2008) facilitates its search engines to seek various categories of information, such as books, news, images, blogs, university documents and scholarly papers. Moreover, users can search for what they want based on different demands. Another type of personal agents collects the usage frequency, remembers information about users’ preference and provides suggestions. Alternatively, users can inform agents of their personal preferences and interests. For instance, some media players, such as iTunes, are equipped with the capability to recognise music played frequently and suggest possible playlists for users (Apple, 2011). These forms of agents are normally based on users’ preference history and this enables the provision of personalised customer service.
In contemporary society, humans live in a fast-paced information era where Information is received from the Internet, television programs, newspaper, and radio daily and almost instantaneously. Furthermore, information filter agents are another form of intelligent agent designed to assist humans eliminate useless or unwanted information. Search engines typically provide numerous interfaces for users to setup and personalise their searches in an attempt to aid this endeavour. Google Reader is a typical information filter agent where users simply select the form of information that they want to receive, and the agent constantly revises favoured news sites and blogs with fresh content (Google, 2008).
In fact, modern lifestyles are more complicated than heretofore, and although people have developed skills to handle information and multitask, intelligent agents are vital for processing abundant resources via technological innovations.
ECAs can also be applied for use in both the personal and commercial area, although the ECAs in this research have been developed for personal use. In the hardware interface survey, ECAs as chatbots are used for personal chatting with users on computers and portable devices. In addition, ECAs as CompanionBots in the character interface survey are adopted as language practice companion for private learning.
41
2.3. Affective Computing
The ECAs in this research were designed with emotional expressions, given that the most vital feature of affective computing is emotion. Therefore, the ECAs, in this research, symbolise one type of affective computing. As mentioned in the above section covering social intelligence, affective computing is a subdivision of AI and generally is a multi- disciplinary research domain spanning several areas, such as computer science, psychology and cognitive science. It differs from other AI research because affective computing not only focuses on usability but also considers social factors. In truth, most social factors of affective computing are related to emotions while the term affective is generally thought to relate to human emotion. Picard (1995) coined the term Affective Computing through her 1995 publication of a paper of the same name. Alternatively, the term affective in affective computing has been defined as the natural and social components of human communications that humans have adopted in order to interact with computers (Picard, 1997). In addition, affective computing refers to the computers equipped with the abilities to recognise and express affect when they interact with humans (Picard, 1995). In other words, during the interactions between affective computers and humans, these computers cannot only recognise the emotion of the person with whom they interact, but also express responding emotion.
While previous generations of scientists developed computers for practical purposes, such as problem solving, reasoning, and algorithms, contemporary researchers give notable consideration to emotions as part of AI. In truth, emotions have a major influence on human cognitive processes. Picard (1995, p. 14) proposed that “emotions play a necessary role not only in human creativity and intelligence, but also in rational human thinking and decision-making”. Hence, researchers have adopted human emotions as the main factor of AI to develop computers to not only function productively, but also to more closely resemble humans. In this area, Reeves and Nass (1996) state that the responses of humans towards computer are typically social and natural. Furthermore, Reynolds and Picard (2001) point out that an affective HCI is a natural and comfortable situation because computers can recognise human emotions when users encounter emotional information.
Picard (1997) describe affective computing as equipped with the abilities to recognise, express, and possess emotions and have emotional intelligence. However, it is not necessary to equip all four abilities, as computers with one or more of these abilities 42 could be an example of an affective computer. Picard (1995) categorised the development of affective computing into four stages based on affective expression and recognition as follows: 1) Computers cannot express affect nor perceive affect, 2) Computers can express affect but cannot perceive affect, 3) Computers cannot express affect but can perceive affect, and 4) Computers can both express and perceive affect.
Affective computing displays and perceives emotion in several ways, such as emotional speech, facial expressions, body gestures, multimodal systems, and affect understanding and cognition. This indicates that understanding of how affective computing recognises and expresses emotion is imperative. In reality, the affect model of affective computing is based on numerous sensor-captured pieces of information. For example, affective computing recognises human emotions by acoustic sensors via the sensors analysing features of human voice acoustics, synthesising a system of affective computing to obtain the pertinent data and recognise which emotion the user is experiencing (Tao & Tan, 2005). Additionally, optical scanners can capture facial features from human faces, thereby helping to distinguish emotions. In summary, in terms of expressing emotions, affective computing can express facial emotions and gestures through onscreen graphics and robotic bodies. More related affective measures will be discussed in the section entitled Affective Science of Chapter 4.
Nevertheless, it is the goal of modern researchers to develop the abilities of affective computing so as to not only express emotion, but also to recognise emotion, and ultimately equip computers with emotional intelligence. Tao and Tan (2005) assert that affective computing may potentially absorb human emotions based on a variety of sensor- captured information, and several systems exist for affective computing in order to equip them with the emotional intelligence. The personalised computing systems of affective computing are likely to be equipped with the abilities to perceive and interpret the emotion of users in order to give users intelligent, sensitive and friendly feedback (Martinho, Machado, & Paiva, 2000). On the other hand, multimodal systems of affective computing are equipped with additional abilities to display emotion, including speech, facial expressions and gestures (Tao & Tan, 2005). In addition, affective understanding modules contain functions, such as the capacity to absorb emotional information, retain information, and model current human moods. In truth, the emotional segment is the greatest distinguishing factor between affective computers and other computers (Tao & Tan, 2005). Furthermore, the higher speed of hardware and intelligence of AI is likely to provide affective computing with increased ability to understand and generate emotions.
43 The increasing attention on user-centred design has led to the steady improvement and development of affective computing. This situation is vital, given that affective computing not only satisfies human desires for the functional requirements of technology, but also considers the mental processes necessary for users. In this way, computers no longer transmit cold feelings to humans, as instead warm social manners are conveyed during HCIs. In the longer term, the development of affective computing will probably deliver humans a more colourful computing world.
In this research, two chatbots were developed with four of the most common facial emotions and one neutral emotion in order to investigate human affective factors, and subject/user responses toward affective agent interfaces. Secondly, three CompanionBots were developed with six basic facial emotions plus one neutral emotion to explore affective influences of agent interfaces on users. The ECAs in this research were designed with emotional expressions in order to build affective connections with users, such as engagement, relationships and preference. One central purpose of this research is to investigate the emotional influence of ECAs on users. Therefore, the ECAs in this research were not equipped with the ability to detect human emotions, but they were able to display emotional expressions. Furthermore, the hardware interface survey studied whether the emotions of users were affected by ECAs when ECAs wore emotional expressions.
2.4. Embodied Conversational Agents (ECAs)
Agents have been developed in numerous domains based on different purposes and functions. When agents are depicted as embodied characters, these agents are defined as embodied agents. Additionally, when agents are developed with NLP and have the ability to chat with users, these agents are conversational agents. ECAs are these embodied agents combined with NLP where NLP (as mentioned in previous section) is a system developed by computer scientists to simulate human language processing. Therefore, ECAs are not only depicted by visual characters, but are also equipped with the abilities to communicate with users through natural language. Moreover, they are innovations that permit people to interact with computers in a natural and intuitive manner. This research focuses on embodied talkative agents (chatbots) and learning companion agents (CompanionBots) as specialised forms of ECAs. Several types of agents related to ECAs are described below.
44
2.4.1. Conversational Agents
Intelligent agents have been applied to a broad range of commercial, educational, and entertainment areas, though researchers continue to develop better intelligent agents for use. Conversational agents are an example of an invention that combines intelligent agents and natural language utilisation and these forms of agents mostly function in the customer service and support domain. According to Lester, Branting and Mott (2004, p. 220), conservational agents have the capacity to “exploit natural language technologies to engage users in text-based information-seeking and task-oriented dialogs for a broad range of applications”. Nevertheless, two requirements to build a conversational agent exist, one is a natural language capability and the other is enterprise delivery, which requires a degree of intelligence to function as well as to express reliability to the users (Lester et al., 2004). In fact, some technologies have been developed for conversational agents to enable them communicate with humans through natural language. One example of this technological advance is Artificial Intelligence Markup Language (AIML), which will be addressed in greater detail in Chapter 5.
Since conversational agents communicate with humans via natural language, this has led to their application in customer service, education and entertainment domains. Lester et al. (2004) assert that conversational agent applications used by businesses are most commonly founded on information-seeking and task-oriented dialogues. These applications have been grouped into five main categories, specifically customer service, help desk, website navigation, guided selling, and technical support (Lester et al., 2004). As an example, conversational agents of customer service focus on general Q&A service, agents in the help desk category maintain internal employee queries, website navigation agents help users by guiding them on a complex website, agents of guided selling provide answers and guidance during the sales process while technical support agents respond to users technical problems. This highlights the different capabilities of each form of conversational agent and the demarcation that exists between them.
In addition to the business field, conversational agents can also be utilised in education. Intelligent Tutoring Systems (ITSs) are conversational agents that apply AI techniques to computer-assisted instruction as well as allowing a fully Socratic mode to develop with individualised instruction (Sarrafzadeh, Alexander, Dadgostar, Fan, & Bigdeli, 2008). Another conversational agent that has been successfully deployed in education is the Intelligent Tutoring Agent (ITA) (Day et al., 2005). An ITA works by equipping an
45 interface to interpret natural language and facilitate communication between the student and the teacher. Furthermore, students can communicate with the ITA about learning related information through MSN Messenger. During the course of English learning, the ITA plays two roles. Firstly, it acts as an assistant to the teacher and records the process of the lecture and answers questions from students. Secondly, the ITA acts as a learning partner for students. Similar to the ITA, TutorBot was developed as an assistant for learning processing of English for second language students (Lu, Chiou, Day, Ong, & Hsu, 2006). Moreover, TutorBot combines the NLP tool and AIML into an instant message-based chatbot in order to enhance learning.
Conversational agents are equipped with the ability for users to converse through natural language, and this is the most natural way for humans to communicate with technology. As a result of enhanced knowledge about the development of conversational agents, it is likely to facilitate improved design of affective ECAs.
2.4.2. Embodied Agents
There is a growing interest in developing interfaces for computers that exhibit more human-like characteristics in order to improve the interaction between humans and computers in a natural and direct manner. Examples of such improvements may include developing computers with the ability to communicate through natural language, equipping computers with bodies to express thought through gestures, and furnishing computers with facial expressions to display emotions.
This factor underlines the underlying reasons why researchers have depicted computers with visual characters. In fact, Reeves and Nass (1996) state that people treat computers as more than mere tools and programs. Furthermore, people accept computers that have been equipped with personalities and genders as well as computers that can evoke humans’ emotional responses and influence memories. As a consequence, Reeves and Nass (1996, p. 251) declare that “people’s responses to media are fundamentally social and natural”. This finding has compelled researchers to invent embodied agents in order to improve HCIs and ensure that it more closely resembles daily human-human interactions. This situation has personified by reflecting that embodied agents “have some kind of graphical representation and have more or less human look and feel” (Geven et al., 2006, p. 135).
46 In reality, some embodied agents are equipped with bodies to express thoughts and display emotions through gestures and facial expressions. The faces of agents play an essential role in the human-computer communication process, such as a smile expressing an agent’s happiness as well as behaving as a greeting (Poggi & Pelachaud, 2000). The addition of a graphic persona to an agent to display facial expressions makes users feel as though they are interacting with a social actor. In fact, several advantages of applying embodied agents to HCIs exist. Koda and Maes (1996) point out that results indicate that agents with faces and facial expressions are more likeable and attractive. Similarly, Geven et al. (2006) state that embodied agents offer sense to an abstract system and make the experience of users more meaningful and effective.
The most widely known embodied agents are Microsoft’s Office Assistants, such as Clippit, Marlin and Links (Dickinson College, 2007). While these embodied agents seemed to grant many advantages to HCIs, Microsoft removed these Office Assistants from Office 2007 (Microsoft, 2011). This points up that if embodied agents offer numerous many positive impacts, why has Microsoft removed the Office Assistants from its latest Office Suite. In this domain, Doyle (1999) claims that applying embodied agents to the Office Assistants are not a negative idea nor are Office Assistants poorly designed. The main problem with the Office Assistants is that Microsoft had chosen to apply them in an inappropriate domain. In addition, Schaumburg (2001) commented that the primary reason that users rejected the Office Assistants is that they often interrupted ongoing tasks. This verdict of users has some truth, given that nobody wants software suggestions while typing a report, letter, essay or assignment. Indeed, embodied agents can actively assist the interaction between humans and computers, although the precise demand of users and the tasks being processing require enhanced consideration.
2.4.3. Embodied Conversational Agents (ECAs)
Embodied conversational agents (ECAs) are the combination of embodied agents and conversational agents, which display personified characters and possess the ability to communicate with users through natural language. Cassell (2000) states that conversation is an essential skill for humans and generally learned exceptionally early in a person’s life. Additionally, the human body is suitably equipped to support conversation. As mentioned above, agents are represented by embodied characters to enliven HCIs. Hence, ECAs equipped with conversational ability and expression equipment are likely to offer a powerful approach for humans to interact with computers. This capability derives
47 somewhat from ECAs acting as “autonomous agents with a human-like appearance and communicative skills” (Pelachaud, 2005, p. 683). In truth, ECAs are inventions that allow people to interact with computers in a natural and intuitive manner.
The developments of ECAs are leading HCIs towards human-human interactions, although achieving the goal that the HCIs resemble face-to-face conversations necessitates ECAs to be imbued with certain properties. Several studies of ECAs have provided some noteworthy suggestions for better ECA developments. For example, ECAs should first be able to recognise verbal or nonverbal inputs from the user. Secondly, ECAs should be capable to address conversational functions, such as feedback and turn taking. Thirdly, ECAs should have the potential to generate verbal and/or nonverbal responses to users. Finally, ECAs should be equipped to point out the state of conversation (Cassell, Bickmore, Cambell, Vilhjalrnsson, & Yan, 2000). In addition, Goh, Fung, Wong, and Depickere (2006) assert that a domain-specific knowledge base is better for ECAs design, given that it may provide a more particular answer for users’ questions, and it is likely to offer higher response satisfaction levels in the process of conversations. In terms of the conversation that ECAs participated in, Oviatt and Adams (2000) comment that certain challenges exist for ECAs, as conversations in human society are not only the procedures of information exchanges, but also form a social activity. Therefore, well-designed ECAs also have to perform exceptionally effectively in social activities.
In the hardware interface survey, two ECAs were developed as chatbots to undertake keyboard conversations with the subject. A chatbot is one type of intelligent agent that permits communication with humans via natural language and has been defined as “a conversational agent that interacts with users turn by turn using natural language”ġ (AbuShawar & Atwell, 2003, p. 25). Chatbots have been applied in different domains, such as language learning, customer service, online assistance, and for entirely entertainment purposes. In fact, it is not necessary for chatbots to appear with anthropomorphic forms since the aim of chatbot is to function as a human-computer dialogue system, which employs spoken or text communication with users. However, this research applied anthropomorphic characters to chatbots for express emotions as one type of ECAs.
Most research about ECAs has been based on computers and there is has been an absence of studies conducted on portable devices. However, since Kadous and Sammut (2004) conceived the first ECA for mobile phones, a few other ECAs have developed for
48 portable devices and most of them are fitness applications (Ståhl, Gambäck, Turunen, & Hakulinen, 2009; Johnson, LaBore, & Chiu, 2004; Chittaro, Buttussi, & Nadalutti, 2006). In spite of these advances, limited ECAs research on portable devices address human feelings while there has been almost no attention given to comparing the affective influences of ECAs on different devices on users. Thus, the hardware interface experiment, which compared the human affective factors towards a computer chatbot and a portable device chatbot during HCIs, can overcome some of these current research shortcomings.
2.4.4. Learning Companion Agents
In the character interface survey, ECAs were developed as learning companion agents, which form one particular type of educational agents. Tecuci and Keeling (1999, p. 222) offered a simple definition of educational agents as “a class of agents that assist a user in an educational-related task”. Moreover, educational agents are typically grouped into two major categories, namely Intelligent Tutoring Systems (ITSs) (Kearsley, 1993) and Learning Companion Agents (LCAs) (Hietala & Niemirepo, 1998). One of the benefits of ITSs is that they are generally more active than learning companion agents, and this facilitates users, such as students, to gain new knowledge by this method. By contrast, learning companion agents do not possess the same active teaching capability as ITSs, and as a result they mainly serve an assistive function while users practice. In fact, learning companion agents act as companions for students in the learning process, as it operates on a similar intellectual level as the student involved in the learning task. Both forms of agents can aid the exchange of ideas and accomplishing practice tasks in learning material(s) assigned by a teacher (Chan & Baskin, 1990).
Sarrafzadeh, Alexander, Dadgostar, Fan and Bigdeli (2008) have developed Affective Tutoring Systems (ATSs), which is an augmented type of ITS. These systems are capable of dealing with the affective states of students. One realistic ATS called Eve possesses the capacity to adapt to students’ affective states and display appropriate emotional responses through facial expressions while it is also equipped with a gesture analysis system and emotion detection abilities. Furthermore, Eve has not only being applied in the education field through individualising interactions with the students, but has also been employed in the sales arena. In an online sales environment, Eve is deployed as an online sales assistant, which is able to guide buyers to browser relevant products.
49 In contrast to ITSs, a learning companion agent has been defined as “a computer- simulated character, which has human-like characteristics and plays a non-authoritative role in a social learning environment” (Chou, Chan, & Lin, 2003, p. 258). Researchers have developed learning companion agents to assist users as they learn, although unlike tutors they do not actively lecture students. In reality, learning companion agents play an assistive role, helping students to learn at the time and place of their choice.
The character interface survey intends to investigate the friendly relationship between users and agents. Therefore, this survey developed ECAs as learning companion agents and focuses on the effects of agent interfaces on human affective factors. In the character interface survey, learning companion agents have been classified as CompanionBots. These CompanionBots have been designed as language practice partners to exhibit anthropomorphic representations and conversational abilities in order to assist users as they practice conversation as part of language study. This are will be covered and discussed in further detail in Chapter 7.
This chapter has presented the core component, ECAs and related technologies with a detailed literature review of research in this area. AI forms a critical foundation of ECAs, and this chapter provided a general understanding of AI, including AI-related history, various derivative areas and applications. This chapter has also elucidated the connection between intelligent agents and ECAs, the concept of affective computing, which inspired the ECA emotional expressions of this research, and the major role of ECAs in this study. This research has adopted ECAs as the primary tool to investigate the affective factor of users towards computers. Therefore, ECAs along with its related fields plays an important role in developing the basis of this study. The following chapter will present another core component of this research and its related literature review, namely the interface design of agents.
50 Chapter 3. Interface Design of Agents
Interface design of agents represents one of the core components of this research, as it intends to investigate affective influences of agent interfaces. This chapter follows the literature review on ECAs in the previous chapter, and continues by undertaking a detailed literature review in relation to agent interfaces. Agent interfaces form the main external features of ECAs, which create the first impression for users, and further determine the interactions between users and agents. As a result, this is primary reason that agent interfaces form one of the key focus areas of this research.
This chapter has been divided into four sections as follows: 1) User interfaces, 2) Interfaces of embodied agents, 3) Visual impacts of agent interfaces, and 4) Methods of agent interface studies. Agent interfaces represent one type of user interfaces and the chapter will address the following areas. The first section gives a general overview of user interfaces in terms of its history, elements, design-related aspects and HCIs. Then, the second section addresses several issues regarding the representations of agent interfaces, including face, gestures and beauty. Subsequently, the third section analyses previous studies about the visual impact of agent interfaces on user experience, behaviour and performance. Finally, the fourth section presents numerous general methods that existing agent interface studies have adopted, and evaluates the advantages and disadvantages of these particular approaches.
Agent interfaces are the visual representations of ECAs, and they typically leave a visual impact on users. This chapter reviews previous academic studies concerning interface design, since this is a major component, which this research intends to explore. Moreover, academic theories in relation to affective factors will be presented from a psychological perspective in next chapter.
3.1. User Interfaces
Agent interfaces, which form the core component of this research, are visual representations of ECAs, and represent one type of user interface. An interface is a medium of communication between two systems or two work groups. Nowadays, technology assists humans to improve life quality in various ways, and in this manner, interfaces are a major factor that affect individual’s capacity to successfully utilise such technologies conveniently. At this point, it is useful to define interfaces and this research
51 will employ the one offered by Laurel and Mountford (1990, p. xii) of an interface as being “a contact surface…. that reflects the physical properties of the interactors, the functions to be performed, and the balance of power and control”.
A user interface is the medium where interactions occur between humans and hardware or software, although generally speaking, it refers to the physical component (hardware interface) and logical component (software interface). For example, input devices and output devices are two types of hardware interfaces whereas command line and GUI functions belong to software interfaces. User interfaces in contemporary application are typically referred to as GUIs, which are those graphic interfaces visible operating in software, websites, operating systems, widgets and applications for electronic devices. GUIs employ graphic items, such as windows, icons, menus and cursors to assist users’ interaction with computers. In this research, interfaces are particularly addressed in the hardware interface survey focus while GUIs are tackled in the character interface survey.
3.1.1. Hardware Interfaces
Hardware interfaces enable people to operate electronic devices, such as panels on automobiles, microwaves and remote controls. This is achieved by translating an individual’s commands to machines as well as providing a means of control over these devices. In this field of study, hardware interfaces of computer systems are predominately referred to input devices and output devices. Each of these devices have specific functionality with input devices, such as keyboards or a computer mouse allowing humans to control the systems. Alternatively, output devices like screens and speakers display the end results.
Initially, user interfaces designed for computers focused on the hardware interfaces (Grudin, 1990). However, developments of computer technology have facilitated operating systems to evolve from programming tasks to WYSIWYG (what you see is what you get) visual operations (Gentner & Nielsen, 1996). While these advances have offered noteworthy gains for users, hardware interfaces remain necessary to help users input commands to machines and display information of machine output to users. Two types of user interfaces for two devices (computers and portable devices) were adopted in this research with a view to study human affective factors and responses towards ECAs. This research addresses the different human affective factors observed among test subjects and their responses towards these two interfaces, including emotional influences
52 and user-agent relationships. This research implies that hardware interfaces refer to the visual representations on different hardware screens.
3.1.2. Graphical User Interfaces (GUIs)
GUIs are one type of user interfaces with visual graphical icons, visual indicators and special graphical elements. Individual interaction with computers is primarily accomplished through a computer mouse launching programs by clicking on graphical icons and controlling software through identifiable graphical elements. As a consequence, GUIs enable users to understand the functions of programs and provide an easy way to control computers. In truth, the use of characters to depict computational systems is a relatively new approach of GUIs. In this area, it is beneficial to examine the history of the GUIs to understand how they have evolved.
Brief History of GUIs
GUIs are the most common interfaces that users encounter when utilising a variety of modern digital devices, including computers, the Internet, mobile phones and music player devices. Prior to the development of GUIs, most interfaces were command line interfaces, which were problematic for inexperienced users to deploy and manipulate. As mentioned in the prologue, the Memex idea of Bush (1945) inspired the practical use of GUIs. In fact, Engelbert invented a computer, oN-Line System, which was controlled by a mouse during the 1960s (Reimer, 2005). The oN-Line System was also the first system to utilise a GUI input device, namely the mouse. Subsequently, a team led by Alan Kay developed the first GUI system, having being heavily influenced by Bush’s ideas. This work also resulted in the creation of one concept of a notebook computer called Dynabook, first conceived in 1968, which was a portable device based on the hypothetical Memex (Kay, 1990).
53
Figure 13. The oN-Line System display, keyboard and mouse (Reimer, 2005).
Kay (1990) asserts that computers could be used as a communication medium rather than as a tool. This concept was also applied to two computers that Key’s team developed at Xerox RARC, the Alto and Star (Barnes, 2000). The Alto offered all the familiar graphical components of the GUI, such as icons, windows, scrollbar and sliders, when it was developed in 1973. In contrast, the Star, the first GUI personal computer, which was developed in 1981, was a failure as a result of being underpowered, slow and overpriced (Raymond & Landley, 2004). Although these computers had contrasting fortunes, both were developed according to Kay’s interface concept. Kay (1990, p. 196) states that “doing with images makes symbols”, which refers to the communicative abilities of images and symbols, albeit humans have enhanced capacity to understand symbols and they also offer augmented problem-solving and cognitive processing abilities.
Figure 14. The Xerox Star (Raymond & Landley, 2004).
Even though Star proved a market failure, it remains the foundation of GUIs and its underlying concepts were subsequently applied by Apple in their products’ GUIs. Mandel (1997, p. 83) insists that an effective interface “should be consistent with real-world objects and metaphors”. Hence, the desktop metaphor was applied in the Apple Lisa and Macintosh. Similarly, numerous computer operations have adopted GUIs as their
54 interface, such as Graphical Environment Manager (GEM) of Digital Research, MS-DOS of Microsoft, and a series of Windows of Microsoft as well as Mac OS of Apple. The recent development of GUI technology has witnessed the application of 3-D effects in window management, for example Windows Flip 3-D (Microsoft, 2008), Compiz (Compiz, 2007) and Exposé (Apple, 2007b).
Touch user interfaces display graphical interfaces on touchscreens, thereby allowing users to interact with devices intuitively (Sasaki, Kuroda, Manabe, & Chihara, 1999). These features may account for the increasing popularity of touch user interfaces. This has been particularly germane after Apple introduced their first mobile phone product, the iPhone, in 2007 (Apple, 2007a). The final goal of the development of user interface is expanding the significant power of interfaces to allow users direct manipulation of computer systems. Therefore, interfaces should not form a barrier between computers and users. Instead interfaces should assist users to control computers as though they give the command directly.
Elements of GUIs
While characters represent one vital element of GUIs, more commonly the main elements of GUI are referred to by the acronym WIMPs, which includes windows, icons, menus and pointers (Mandel, 1997). These GUI elements were first established by Xerox (Raymond & Landley, 2004). A window refers to a display area for information on screen and allows subdivision of the screen space so that it can show users several different interfaces while facilitating contemporaneous activation of assorted operations. Furthermore, windows provide the ability to have more than one view on a single object (Sutcliffe, 1989). Additionally, icons are small graphical forms that are used to represent objects or functions, and with superior design, they can also provide information about the functions and significance of objects (Mandel, 1997). They also permit users to make an informed judgment about their meanings instantaneously owing to reduced required learning time (Sutcliffe, 1989). Alternatively, menus provide a list of choices for users to select and execute commands. Menus are one type of simple interfaces for inexperienced users because they present limited choices on a screen at once. Nevertheless, the optimal amount of choices is nine or less, as users may experience information overload and increase the amount of time spent searching menus (Sutcliffe, 1989). Moreover, pointing devices are tools used to input pointer location in order to execute commands, and can
55 include such devices as a mouse, touchpad, trackball or touch panel. In reality, those particular elements are important components for users to control computer system.
In addition to WIMPs, GUIs provide another approach for presenting information in the form of virtual characters. According to Reeves and Nass (1996), individuals generally treat media figures as they treat real life interactions, and as a result life-like characters have become another graphical interface strategy. For example, Microsoft released a software program called Bob in early 1995, which was an embodied character and designed mainly for inexperienced computer users. As shown in Figure 15, users can simply create an account and will be taken to their room where an embodied dog with a natural language menu assists in operating the program (Andy, 2005).
Figure 15. Our Room - The Study (Andy, 2005).
The primary aim of this research is to investigate the affective interactions between users and life-like characters. Additionally, this research attempts to discover whether users prefer character-presented or other forms of interfaces and what type of character interfaces users prefer. Thus, character-presented interfaces form a crucial aspect of this study.
Interface Design
As already stated, interfaces consist of two main types, namely hardware and software interfaces. Most computer interfaces that users interact with are software interfaces, such as colourful graphics, sounds, speech and animations, given they are excellent means to express ideas and information on computers (Hakansson, 1990). In fact, interface design and usability mainly determine the functionality of a program or electronic device. Therefore, there is growing attention on different aspects of interface design.
56 In the hardware interface survey, experimental interfaces were designed for two devices. Furthermore, the rising popularity of mobile devices has encouraged designers to consider the composition of screens and the size of the handset. Nonetheless, Shneiderman and Plaisant (2005) claim that designers face the challenge of providing different interface design for varied devices. These devices range from portable ones, such as mobile phones with portable display, to private interaction with ubiquitous television screens, and even multiple users controlling characters in video games. In order to provide a better environment for HCIs, researchers and designers have endeavoured to develop interface for several devices. In fact, considerable design guidelines exist for interface design, and some recognised and general design suggestions for the majority of user interfaces are presented in the following paragraph.
The primary suggestion for interface design has focused on user-centred design. Norman (1990, p. 217) ponders “What are computers for?” and answers “The user, that’s what- making life easier for the user”. Therefore, the foremost concern for interface designers is considering the needs of users and what utility they can offer. In the same vein, Mandel (1997) contends that the optimal approach for designers to improve the interface is to factor the user’s beliefs, wants, needs, experiences and expectations into the design process. Accordingly, the paramount issue in interface design is how these elements affect users.
Since interface has been created to serve users, Sutcliffe (1989) recommends three approaches to address user-centred design. The first approach involves user-participative design where participation during the design process is likely to narrow the gap between what the design team envisions and what users require from an interface in terms of value and content. The second approach is the aforementioned suggestion, namely user-centred design where designers should primarily consider the needs of users rather than the functional processing requirements and hardware limitations. The final approach entails iterative design, which involves the development of prototypes during the design process with a focus on improving human-computer interactions. The intention of inventing these varied systems and devices is to assist users and to eliminate confusion. Consequently, every aspect of interface design should consider the user’s precise needs and capabilities.
As mentioned above, symbols facilitate easier user understanding of the system content (Kay, 1990). Therefore, another enhancement for interface design is to design easy to understand and clear metaphors for graphics. The idea of creating interface metaphors is the main factor to assist humans to associate user understanding of computers to familiar
57 concepts. As a consequence, metaphors are a powerful tool for helping both inexperienced and expert users to rapidly become familiarised with new situations (Mountford, 1990b). In truth, metaphors not only appear in user interfaces, but also are present when people speak, write and think about abstract concepts. In this domain, Erickson (1990) explains that metaphors allow users to give structure to abstract concepts by augmenting users’ knowledge and experience. Moreover, metaphors provide methods to understand abstract concepts about systems based on existing knowledge and experiences, and help to improve understanding of how things actually work.
In fact, the interface design suggestions have broad similarities. However, the design recommendations are simply used as a reference point for designers to improve interfaces, given that interfaces are the medium for users to communicate with devices and software. However, the principles of interface design vary according to the precise device and/or software employed.
Web interfaces are one of the most common forms of software interface design that millions of people daily encounter when using the Internet to gather information. People typically communicate with websites through web interfaces to facilitate shopping, searching and gaining the most up-to-date information. This is accomplished through various search engine websites, such as Google and Yahoo!, shopping websites like Amazon and eBay, news websites including CNN and BBC, company websites as Apple and Microsoft, and various other types of websites. As a result of the advantages provided by the Internet, such as convenience, concurrence, and unlimited space and time, increasing numbers of enterprises are transferring their services to the Internet. Furthermore, the popularity of the Internet has drawn greater attention to web interface developments, and this trend is evidenced by some financial firms satisfying customer demands to employ GUIs when offering services online (Magee, 2008).
According to Mandel (1997, pp. 413-414), three main conclusions for web interface design exist. Firstly, users “have very little patience for poorly designed web sites”. In fact, given billions of websites are available on the World Wide Web for users to browse, it is unlikely users will tolerate poorly designed interfaces and overloaded web pages. Secondly, Mandel points out that “users don’t want to scroll information”, as they prefer to see what is needed easily and rapidly on the current web page. Finally, most users eschew reading, and in the scenario where texts and graphs co-exist on the same web page, users prefer to see graphs.
58 In addition to web interfaces, game interfaces are other popular interfaces that humans commonly come across, in particular in the gaming and entertainment sphere. Furthermore, designers typically place entertaining users as the primary principle when creating game interfaces. As a result, “game interfaces have advanced much faster than business software interfaces” (Lager, 2008, p. 29). Therefore, Crawford (1990) implies that some general lessons can be learned from game interfaces, including using additional graphics and sound, and emphasising the effect of these interactions. In truth, game interfaces differ from other user interfaces because they are principally designed for entertainment. This results in users experiencing greater pleasure and enjoyment when they interact with these game interfaces, and this phenomenon is replicated by other interfaces.
The growing contemporary prevalence of portable devices has led to an increasingly important role for mobile interfaces. The major distinction between mobile and software interfaces is the screen size. In reality, recent research has demonstrated that mobile phones are the most valuable possession for individuals, not the laptop as may be commonly assumed. Additionally, greater numbers of people purchase mobile phones than those buy laptops while individuals use mobile phones to undertake some of their most vital tasks (Lager, 2008). However, interface design for mobile phones is limited by the screen size and the efficiency of palm-sized devices. In fact, efficient mobile interface means that users can deploy mobile devices conveniently, easily and intuitively to control the most frequent functions (Jarnstrom & Kiljander, 2003). In order to achieve these objectives, Lindholm and Keinonen (2003) advise that metaphors are exceptionally effective tools to apply to mobile interface design, even with only limited wording. Furthermore, in the case of habitual application, users can be encouraged to relate their common knowledge to mobile interfaces and instinctively function the devices in various situations.
In addition, the growing scale of mobile phone use is illustrated by the number of handsets globally, which reached 3.3 billion by the end of 2007, a figure encompassing almost half of the world’s population (The Daily Telegraph, 2008). There were more than 5 billion mobile phone connections worldwide in 2010, and the penetration even exceeded 100% in many regions where there was more than one connection per person. In fact, this figure has been predicted to rise to six billion mobile phone connections worldwide by the middle of 2012 (BBC, 2010). In order to satisfy these customers, mobile phone services are developing at an increasing rate, and as a result many popular websites, including Amazon, Facebook, Google, Youtube and the BBC, have been
59 redesigned for mobile phones. These websites have been developed to offer high- resolution and palm-sized displays suitable for viewing downloading to a handheld device. In order to further develop the research, more practical mobile interface design will be presented in Chapter 5. In this case, the hardware interface survey compares ECA interfaces in terms of computers and portable devices. Consequently, it will be possible to ascertain the differences of human affective response towards these two ECA hardware interfaces.
3.1.3. Human-Computer Interactions (HCIs)
This research aims to study the alterations in human affective feelings that occur during HCIs. Thus, HCI design is also crucial to the conduct and findings of this research. Additionally, the increasing use of technical devices has increased public and academic interest in HCIs over the course of several years. Booth (1989, p. 4) offered a simplified definition of the term HCI being “the study of the interaction between humans and computers”.
In truth, interactions variously depend on different interfaces for users to implement, including GUIs, speech-based, touch screen, multimedia and wearable devices. Sharp (2007) suggests four fundamental types of HCIs, namely instructing, conversing, manipulating and exploring interactions. Instructing interactions are the processes where users tell systems what to do in order to perform their tasks, although these forms of interactions are limited in that computers simply obey orders. On the other hand, conversing interactions are based on a two-way communication process where users engage in conversations with computers and computers respond to users in a conversational way. A third form, manipulating interactions allow users to manipulate objects through physical gestures and movements, and these interactions not only apply to computers but can also be used in electric robots and toys. Finally, exploring interactions refer to users investigating virtual or physical environments, and these forms of interactions allow users to move and navigate through existing spaces to examine and interact with virtual or physical environments. On the whole, different modes of the HCIs can be utilised to achieve a number of different outcomes while being applied to diverse systems and devices based on their missions and processes. In order to develop better HCIs, many researchers suggest a number of principles for improved interaction design. Similar to user interface design, the main factor of interaction design is user-centred, and entire sets of interactions are designed for humans, and “users should set the pace of an 60 interaction" (Raskin, 2000, p. 8). In most pieces of literature discussing HCIs, attention has predominately addressed user-centred design and user needs (Blair-Early & Zender, 2008). Furthermore, Sharp (2007) points out developers can achieve a more detailed understanding of users’ goals by involving real users throughout the development of interaction design.
In fact, more recent developments in HCI design have moved beyond user-centred design to user-involved design and learner-centred design (Mandel, 1997). Thus, HCI design is based upon users through a more comprehensive understanding of their need and wants as well as offering better services. One of the underpinnings of this trend is that smooth and lively interactions between users and computers facilitate more productive work and diminished time to achieve positive results. Interaction design generally involves “designing interactive products to support the way people communicate and interact in their everyday and working lives” (Sharp, 2007, p. 8). Therefore, interaction design is pivotal for designers and researchers at every step of development.
During HCIs, humans typically generate and respond to emotions. As a result, developers have invested considerable time and effort to conceive intelligent robots and computer- based systems, which behave and appear like humans and other creatures in order to ensure that users feel more comfortable as they interact with computers. In fact, Sharp (2007) explains that it is natural for people to create designs for technology, which exhibit more human-like attributes. Therefore, anthropomorphism has become more widespread in interaction design through the employment of intelligent agents and interactive toys. As an illustration, users chat with anthropomorphic characters on the Internet or interact with personified robots through physical contact. In summary, these products are primarily developed in order to guarantee that the interactions between humans and computers more humanised.
There are numerous methods to improve the interactions between humans and computers. One clear method is through sound effects that can enhance the affective influence on users. In this field, Lee and Nass (2003) investigate the personality of synthesised voices and how they affect users’ feelings. In reality, this experiment ascertained that users have a stronger sense of social presence when the synthesised voice matches the message of the content, although when the tone and content are incompatible, users felt a diminished social presence, thereby negatively affecting their ability to interact with the computer. Hence, the senses of sight and hearing are the most obvious senses that users accept from computers.
61 This research aims to explore affective HCIs, and further develop HCIs complete with more affective phenomenon to achieve the human-human interactions. In truth, computers have a significant role in building affective characteristics during HCIs, which provide humanoid and warm interactions.
3.2. Interfaces of Embodied Agents
Agent interfaces form one of the core components of this research. Although several manners of agent interfaces exist to express information, such as gestures, voice and facial expressions, this research mainly focuses on the facial expressions and character classifications of ECAs. In the domain of future developments, Mountford (1990a) mentions that humans are likely to communicate with computers by means of speech and gestures, control multimedia by hand, and perceive the status of a computer through colour or animated images of internal processes. In fact, social user interfaces contain the entire range of communication methods and allow users to converse with computers by speaking in their natural language or gesturing with certain body parts. Negroponte (1990) posits that proscenium and embodied agents are likely to appear on desktop computers according to the users’ individual desires. Moreover, Kurtenbach and Hulteen (1990) point out using gestures as a means of computer input improves orientation, connection, function and other advantages of communication. Furthermore, users could enhance their expressiveness through gestures. Alternatively, speech is a specific, efficient and rich medium for sharing information and knowledge (Mountford & Gaver, 1990). One of the drawbacks of graphical images is that they are limited by space while a sound can be heard in various locations simultaneously. Therefore, speaking and gesturing are extremely propitious forms of communication.
Another means of communication for the social user interface is the embodied agent. Embodied interface agents are defined as “transform the experience of interacting with a computer, making it more social by explicitly referencing human-human interaction” (Zimmerman et al., 2005, p. 233). Oren, Salomon, Kreitman and Don (1990) point out that users assume that graphic guide figures had been created with individual personalities, although the system had not been developed as characters. Additionally, Laurel (1990, p. 360) states that “an interface agent is a prime example of user-centred interface design”, because embodied agents possess the capacity to socially interact with users. Indeed, embodied characters equipped with AI can communicate with users through natural language and by recognition of users’ gestures. When embodied 62 characters interact with users in tight synchronisation with natural language, embodied interfaces provide a wide range of behaviours to perform communicative functions (Cassell, Bickmore, Campbell, Vilhjálmsson, & Yan, 2001).
In addition to numerous agent interface studies, which have been addressed in the previous content, several studies in relation to different expressive manners and visual parts of agent interfaces are presented below.
3.2.1. Face
The face is a key factor for the research of interfaces with embodied agents, as face includes two main elements, namely facial expressions and eye contacts. Furthermore, facial expressions have the ability to express the emotions of embodied agents and eye contacts can indicate directions of current information in a given task.
In truth, several studies have been conducted that address design aspects of facial features for embodied agents (Gong & Nass, 2007; Sproull, Subramani, Kiesler, Walker, & Waters, 1996). One such investigation by Walker et al. (1994) discovered that facial expressions affected users’ experiences and performances. This study also found that stern expression created a bad impression but led to greater engagement. Furthermore, the eye gaze of agents also affects users in the tasks, as the natural eye gaze of agents helps direct the attention of users from one agent to another during a multi-agent task (Colburn, Cohen, & Drucker, 2000). Similarly, Vertegaal, Slagter, van der Veer and Nijholt (2001) also believe that eye gaze contacts are the greatest predicator of conversational attention during multiparty conversations. In fact, eyes are the most obvious cue within the face that allows the imparting of information. Furthermore, eye gaze not only enhances emotional expression, but also assists directing users’ attentions during particular tasks to augment user engagement.
Face is the most common application for embodied agents functioning within interfaces, given that it is the most obvious bodily form that presents human features, and it is the easiest approach to build a connection with humans. Therefore, Cosi, Fusaro and Tisato (2003) developed a high-quality 3-D Talking-Head Model, called LUCIA, for embodied agents to animate. In the same domain, RUTH is another Talking-Head Model, which can be animated to synchronise speech and lip movements (DeCarlo, Stone, Revilla, & Venditti, 2004). Those face-based animation systems help the interfaces of embodied
63 agents to perform naturally in real-time. This research will also focus on building agent faces to express emotions in order to develop affective connections with users.
3.2.2. Gestures
Similar to real humans, embodied agent interfaces use body language to improve interactions with others. This is accomplished through embodied agents applying physical models, such as arms and hands, to indicate directions, enhance emotions, and attract the attentions of the users.
Cassell and Vilhjálmsson (1999) examined autonomously generated hand gestures of physical models of embodied agents and claimed that embodied agents exhibiting such gestures are more natural and able to increase the perceived expressiveness of users. Similarly, Hartmann, Mancini, and Pelachaud (2006) also developed a gesture modelling language in their animation system for embodied agents. These gesture-modelling systems improve the interaction between agents and humans while also affecting user feelings towards embodied agents during HCIs.
3.2.3. Voice
Voice is another feature that enables embodied agents to humanise representations (Gong & Lai, 2003). Furthermore, Louwerse, Graesser, Lu and Mitchell (2005) ascertained that humans preferred embodied agents with natural voices, such as male agents with male voices. Moreno (2001)ġ demonstrates that the voice of agents is responsible. In terms of learning, no difference exists between a pedagogical agent and a voice-based agent (Rajan et al., 2001; Atkinson, 2002). Atkinson (2002) discovered that learners experience less difficulty to learn from a speaking agent as opposed to a text-based agent. Additionally, a pedagogical with human voice led to greater learner perceptions (Baylor & Ryu, 2003). In fact, when embodied agents presented themselves as personified interfaces, users typically infer agents are likely to act as humans, including displaying facial expressions, body gestures and voices.
64 3.2.4. Beauty
In terms of the beauty of embodied agents, Norman (2002, p. 36) states that “attractive things work better” and during the interactions between humans and embodied agents, the beauty of the embodied agents has a considerable impact on users. van Vugt et al. (2006) declare that the beauty of embodied agents may enhance user engagements, albeit the good looks of embodied agents did not affect the intention of users to use the character. van Vugt et al. (2007) also proved that users felt more involved with beautiful embodied agents. This is likely to derive from individuals feeling about beautiful products, where the sell rate and satisfaction is typically far higher than with less aesthetically pleasing products.
3.2.5. Realism
The degree of realism of embodied agents tends to have different outcomes on users. On the one hand, Koda and Maes (1996) found that users rated realistic agents more intelligent and engaging as well as increasing task involvement. On the other hand, van Vugt, Konijn, and Hoorn (2009) discovered that the realism of embodied agents is insignificant in the case user involvement, as other design features have a more influential effect on user performance.
In summary, according to the above studies, it is not particularly problematic to establish that the appearance and expressive manner of agents affect humans. However, limited research has focused on the visual form of agents. Gulz and Haake (2006, pp. 327-328) addressed two possible explanations as to why visual appearance is commonly neglected in research on embodied agents, given “it cannot be readily approached with existing research methodology” and “the influence of look on emotional and intellectual processes is not readily accepted, although empirically well established”. In addition, compared to the amount of studies on other aspects of interface design, such as websites, applications on portable devices and computer software, a paucity of investigations have been conducted on interfaces with embodied agents. However, the interfaces of embodied agents continue to play a significant role in user engagement and overall feelings. As a result, designers and researchers should grant additional attention to improving design of the interfaces with embodied agents. In truth, the two surveys were undertaken to
65 determine the connections between human affective factors and agent interfaces, including ECA hardware interfaces and character classifications.
3.3. Visual Impact of Agent Interfaces
The various studies, cited above, have proved that various agent interfaces deliver diverse influences on HCIs, in particular the visual appearances of agent interfaces affect users who utilise computational systems. Gulz and Haake (2006) posited that poor visualisation of educational animated agents has impacted users, despite the successful functioning of agents. In addition, this study proposes that visual and aesthetic factors should be considered when researchers or designers develop animated pedagogical agents. In fact, this situation pertains, as “appearance is a critical component of how people access ECAs. It is perhaps not surprising that individuals prefer to look at or even interact with ECAs that are more “attractive”” (Nass, Isbister, & Lee, 2000, p. 397). The embodied pedagogical agents positively influence learning in various ways (Clarebout, Elen, Johnson, & Shaw, 2002; Krämer & Bente, 2010).
Numerous studies have indicated that embodied agents have the ability to improve HCIs, although failed applications like the aforementioned Microsoft Office Assistants may periodically occur. In truth, multifarious suggestions have been generated for designers and developers to create better agent interfaces in order to avoid such situations eventuating. Koda and Maes (1996) state that ECAs portrayed as animal characters are generally regarded as more likeable than their human counterparts. In a similar vein, Oviatte and Adams (2000) hint that children prefer to talk to ECAs that are represented by animals because they viewed them as friends, rather than aspirants or teachers. In addition, Forlizzi, Zimmerman, Mancuso and Kwak (2007) explored the relationships between the visual features and performances of embodied agents, and concluded that users favour human forms over non-human forms. Additionally, users typically choose embodied agents with female forms as opposed to male ones. In fact, the visual presence and appearance of these agents impact users in many ways, such as beliefs, interest, attitude and feelings (Baylor, 2009). Actually, agent interface related studies under various mission with dismiss purposes offer dissimilar suggestions for interface design. Therefore, appropriate agent interface design should consider the precise mission being undertaken and the surrounding environment.
66 Furthermore, beautiful items generally attract people’s eyes, and this phenomenon also prevails with agent interfaces. In reality, embodied agents designed with beautiful appearances are more convincing and possess the ability to obtain greater cooperation from their users (Nass et al., 2000). Moreover, the appearance of embodied agents influences the feelings of users. In this field, Koda and Maes (1996) assert that humans favour embodied agents with realistic faces to those with abstract ones while people consider that realistic embodied agents possess greater intelligence.
In fact, similarity is another feature of agent appearance that impacts users. For example, students prefer pedagogical agents of the same ethnicity as their own (Baylor, 2005). Similarly, agents that are similar to users in terms of appearance-related characteristics have more influence on users (Bailenson, Blascovich, & Guadagno, 2008). Additionally, Fox and Bailenson (2009) established that agents with similar appearances to users and where users watch them exercising and losing weight in the virtual world, users will subsequently exercise more and try to lose weight in the real world.
In addition to the effects of beautiful and realistic ECAs, studies have recorded that the sizes and forms of embodied agents similarly influence users. Partala, Surakka and Lahti (2004) researched the proximity levels between embodied agents and humans. They found that preferred levels for individuals to communicate with an embodied agent are the personal proximity level (46-122 cm) or the social proximity level (122-366 cm). Additionally, the size of face displays elicits some psychological responses among users, such as mental energy, memory and judgments (Reeves & Nass, 1996). In truth, agent interfaces comprise the attached elements of vision on intelligent agents and appearance is the most significant factor, which motivates users to utilise embodied agents (Baylor, 2009).
According to Dehn and van Mulken (2000), the effects of animated agents on users consist of three types of effects, namely users’ subjective experiences, users’ behaviours and user performances. The following paragraphs will summarise the work of Dehn and van Mulken (2000) and assess numerous empirical studies evaluating the visual impact of agent interfaces on users’ experiences, behaviours and performances.
The employment of visual characters to represent agents may potentially influence user attitudes towards agents. In fact, the empirical evidence regarding the visual impact of agent interfaces on user experiences includes intelligence, believability, likeability/social evaluation, activity, engagingness/entertainment, comfort, usability/utility, reported attention, enjoyment, appreciation and felt support. Prior to interaction, the majority of
67 users had assumed agents that resembled humans were more intelligent (Koda & Maes, 1996; King & Ohya, 1996). However, after the interaction, the visual representations of agents normally did not affect user feelings about agent intelligence (Sproull et al., 1996; Koda & Maes, 1996; Geven et al., 2006). In the dimension of credibility, agents with visual presences or expressive emotions are typically more believable (Lester, et al., 1997; Brave, Nass, & Hutchinson, 2005; Rosenberg-Kima, Baylor, Ashby Plant, & Doerr, 2008). In addition, agents that are represented by visual empathic characters and similar to users are more preferred (Koda & Maes, 1996; Brave et al., 2005; Berry, Butler, & Rosis, 2005), and a rabbit-like agent is more preferred by users compared to a humanoid- looking agent (Sträfling, Fleischer, Polzer, Leutner, & Krämer, 2010). Similarly, with reference to engagement, agents that are designed with animated faces and beautiful features seem to encourage users to increase their commitment in HCIs (Koda & Maes, 1996; Baylor & Ryu, 2003; van Vugt et al., 2006).
The visual impact of agent interfaces on users has been primarily investigated with regard to user behaviours while interacting with agents, such as recognition, attention, flow of communication and self-presentation. Koda, Ishida, Rehm and André (2009) discover that the recognition accuracy of agent negative expressions is higher than positive expressions. In addition, users are willing to pay more attention to agents with faces than those without (Takeuchi & Naito, 1995; Sproull et al., 1996). In the sphere of self- presentation, users rated higher self-presentation scales on agents that are developed with speaking faces or similar to users (Sproull et al. 1996; Rosenberg-Kima et al. 2008).
These study findings demonstrate that the representations of agents have certain influences on user behaviour. Results from this behavioural data reflect the outcomes of HCIs in terms of performance, problem solving, learning and memory. According to Geven at al. (2006), subjects performed better when they indicated with GUIs than with a character presented agent. However, the study was a singular case, as subjects were asked to indicate a direction under the interpretation of computer interfaces. Results showed that GUIs provided clearer instruction of direction over the character presented agent. The study of Mumm and Mutlu (2011) on motivational interfaces drew that human-like agents are likely to enhance the social presence of the interface and improve user performance. In the dimension of learning, studies have shown that different character presented agents and expression methods led to distinct learning outcomes (Baylor & Kim, 2004; Baylor & Ryu, 2003; Baylor & Kim, 2005; Baylor & Kim, 2008).
68 Table 3 succeeds the review of Dehn and van Mulken (2000) and summarises the visual impact of agent interfaces on users’ experiences, behaviours and performances of assorted empirical studies.
Table 3. The visual impact of agent interfaces on user experiences, behaviours and performances. Variable Authors Results Intelligence After interaction Sproull et al. (1996) I (text with speaking face) = I (text) Koda & Maes (1996) I (caricature face) = I (no face) Geven et al. (2006) I (no character) = I (cartoon character) = I (realistic character) Before interaction Koda & Maes (1996) I (caricature face) < I (realistic face) King & Ohya (1996) I (geometric shapes) < I (human shapes) Believability Lester et al. (1997a) B (muted agent) < B (expressive agent) Lee & Nass (2002) B (animated agent)> B (static image), B (text- box) Baylor & Ryu (2003) B (animated image) = B (static image) Brave et al. (2005) B (empathic emotion) > B (no empathic emotion) Geven et al. (2006) B (no character) = B (cartoon character) = B (realistic character) Rosenberg-Kima et al. B (visual presence) > A (voice alone) (2008) Likeability/Social evaluation Sproull et al. (1996) S (text with speaking face) < S (text) Gong (2008) More anthropomorphic agent receives more social responses. Koda & Maes (1996) L (face) > L (no face) Brave et al. (2005) L (empathic emotion) > L (no empathic emotion) Berry et al. (2005) L (face) > L (no face) Sträfling et al. (2010) L (cartoon-like) > L (human-like) Attractive Lee & Nass (2002) L-A (animated agent) > L-A (static image), L- A (text-box) Nowak & Rauh (2005) L-A (Femininity agent) > L-A (masculinity agent) Preference Nowak & Rauh (2005) L-P (same gender as subjects) > L-P (differ gender from subjects) Activity King & Ohya (1996) A (geometric shapes) < A (human shapes) Sproull et al. (1996) A (text with speaking face) < A (text) Politeness Hoffmann et al. (2009) A-P (character) > A-P (paper-and-pencil questionnaire) Engagement/ Entertainment Takeuchi & Naito (1995) E (face) > E (arrow) Koda & Maes (1996) E (face) > E (no face) Lester et al. (1997a) E (mutes agent) = E (expressive agent) Baylor & Ryu (2003) E (animated image) > E (static image) van Vugt et al. (2006) E (beautiful character) > E (ugly character) Involvement van Vugt et al. (2007) E-I (realistic agent) = E-I (unrealistic agent) Distance van Vugt et al. (2007) E-D (realistic agent) = E-D (unrealistic agent) Intention to van Vugt et al. (2006) E-I (beauty agent) = E-I (ugly agent) interact (continued)
69 Variable Authors Results Comfort Sproull et al. (1996) C (text with speaking face) < C (text) Koda & Maes (1996) C (face) < C (no face) Perceived caring Brave et al. (2005) P-C (empathic emotion) > P-C (no empathic emotion) Friendly Sträfling et al. (2010) C-F (cartoon-like) > C-F (human-like) Usability/Utility Takeuchi & Naito (1995) U (arrow) > U (face) Lester et al. (1997a) U (muted agent) < U (expressive agent) Cassell & Thórisson U (verbal-only), U (verbal/emotional) < U (1999) (verbal/structural) Berry et al. (2005) U (face) > U (no face) Rosenberg-Kima et al. U (male) > U (female) (2008) Reported Attention Koda & Maes (1996) Caricature face attracts attention Attracts attention Rosenberg-Kima et al. AA (visual presence) > AA (voice alone) (2008) Distracts Koda & Maes (1996) Caricature face does not distract Enjoyment Bartneck (2003) E (emotional robot) > E (none-emotional robot) Bartneck (2003) E (screen character) = E (robotic character) Appreciation Baylor & Ryu (2003) P-L (animated image) > P-L (static image) Felt support Brave et al. (2005) F-S (empathic emotion) > F-S (no empathic emotion) Mitigating Baylor & Rosenberg- M-F (agent) > M-F (text-box) frustration Kima(2006) Recognition Koda et al. (2009) R (negative expressions) > R (positive expressions) Attention Eye contact Takeuchi & Naito (1995) EC (face) > EC (arrow) Response time Takeuchi & Naito (1995) RT (face) > RT (arrow) Sproull et al. (1996) RT (text with speaking face) > RT (text) Skipped items Sproull et al. (1996) SI (text with speaking face) > SI (text) Flow of communication Conversation Takeuchi & Nagao (1993) CI (face) = CI (textual descriptions) index Repetitions Cassell & Thórisson FR (verbal-only), FR (verbal/emotional) > R (1999) (verbal/structural) Hesitations Cassell & Thórisson H (verbal-only), H (verbal/emotional) < H (1999) (verbal/structural) Overlaps Cassell & Thórisson O (verbal-only), O (verbal/emotional) < O (1999) (verbal/structural)
(continued)
70 Variable Authors Results Self-presentation Social desirability Sproull et al. (1996) SD (text with speaking face) > SD (text) scale Altruism scale Sproull et al. (1996) SA (text with speaking face) > A (text) Self-worth scale Sproull et al. (1996) SW (text with speaking face) = SW (text) Self-efficacy Rosenberg-Kima et al. SE (visual presence) > SE (voice alone) (2008) Rosenberg-Kima et al. SE (similar to users) > SE (dissimilar to (2008) subjects) Self-regulation Baylor & Kim (2004) SR (male) > SR (female) Baylor & Kim (2004) SR (mentor) > SR (motivator) SR (expert) Performance Geven et al. (2006) P (character) < P (no character) van Vugt et al. (2007) P (character) = P (no character) Problem-solving Simple problems Lester et al. (1997b) SP (mutes agent) = SP (expressive agent) Complex problems Lester et al. (1997b) CP (mutes agent) < CP (expressive agent) Learning Lester et al. (1997b) L (mutes agent) = L (expressive agent) Baylor & Kim (2004) L (realistic agent) > L (cartoon agent) Baylor & Kim (2004) L (black expert) > L (white expert) Instructor-like Baylor & Ryu (2003) I-L (animated image) > I-L (static image) Baylor & Kim (2005) Expert led to increased information acquisition Baylor & Kim (2005) Motivator led to increased self-efficacy Baylor & Kim (2005) Mentor led to overall improved learning and motivation Baylor & Kim (2008) Facial expression valuable for attitudinal learning Baylor & Kim (2008) Gestures valuable for procedural learning Memory Berry et al. (2005) M (emotional agent) > M (none-emotional agent) Note: Capital letters in the results column are abbreviations of the variable names
There are various advantages between these particular agents, which are presented by various characters. Nevertheless, it is almost impossible to make the advantages outweigh the disadvantages by using one character in all application domains. Hence, careful attention is required to design the front-end representations of agents (Baylor, 2009), and the advised approach is to carefully assess both advantages and disadvantages of all characters and chooses the most appropriate one for the application domain. This research aims to design suitable ECA hardware interfaces for the hardware interface survey and agent characters for the character interface survey. In the end, it is likely that adopting the existing and experimental design guidelines can be extremely helpful in designing better agent interfaces for a multi-agent website.
71 3.4. Methods of Agent Interface Study
While multifarious research methods exist, the most common methods in agent related studies involve experimentation, creating better agents and developing research framework. Most methodologies of agent interface studies entail experimentations, although some other agent studies address the creation of better agents or developing frameworks. The following paragraphs will cite some advantageous examples that employ these methods.
For the purpose of creating better agents, Bickmore (2003) has developed relational agents in his doctoral research, and these agents have the capacity to build and maintain long-term and social-emotional relationships with users. This involved an application developed for assisting users through a month-long health behavioural program. This method has developed relational agents to build durable relationships with users, but also conducted several pilot studies to improve the foundations of relational agents. In summary, Bickmore’s relational agents possess the ability to establish improved long- term relationships with users.
In terms of developing research frameworks, Konijn and Hoorn (2003) have formulated an integrative theoretical framework otherwise entitled Perceiving and Experiencing Fictional Character (PEFiC) that explains the (dis)liking of fictional characters in media. Subsequently, van Vugt and his colleagues (van Vugt et al., 2004; van Vugt et al., 2006) have slightly adjusted the PEFiC model with interface agents to examine the affective effects of relevant factors in the human–character interactions, and this is referred to as Interactive PEFiC (I-PEFiC) while several studies have also conducted to test this amended model.
In addition to creating enhanced agents and developing research framework, experimentation adoption is pervasive in agent-interface-related research. This scientific method is employed to answer questions or investigate problems. Koda and Maes (1996) employed experimentation as their research method in order to investigate the effect of facial features on an interface. This was achieved by developing a web-based poker game with seven different graphical representations, namely a realistic male face, a realistic female face, a caricature male face, a caricature female face, a caricature dogface, a line- drawn Smiley face and the Invisible Man which had no face. In the computer poker game, agents played against each other and a user. In fact, five web-based experiments were undertaken, and between-subjects design was adopted in their study with more than 1,000 subjects accessing the poker game of which 157 of them completed the questionnaire. 72 The questionnaire entailed four questions about the subjects’ background and four to eight questions in relation to subjects’ impressions of the agent interfaces. Each subject who completed the experiment played a poker game for 15 rounds and the process endured for between ten to 15 minutes. The results were analysed quantitatively in order to provide useful data about opinions concerning agent-interfaces.
In another research project, Partala at al. (2004) varied the proximity of agents by exposing subjects to speech messages that were given by an anthropomorphic conversational agent, albeit also employing experimentation as their methodology. This particular experiment adopted within-subjects design, and eight subjects participated in the experiment where each subject was asked to face one agent with four proximity levels of distances. Partala et al. (2004) calculated the simulated sizes of the agent head whose height on the screen were 27.6cm (intimate level), 13cm (personal level), 4.5cm (social level) and 2.4cm (public level) respectively. In fact, each proximity level of agent played two negative, two neutral and two positive messages at random, and 24 messages were displayed to each subject. Subsequently, subjects were asked to rate their subjective experience after each speech message and choose the proximity level experienced as the most natural for an agent. The results in this study were also analysed quantitatively.
Furthermore, Geven et al. (2006) studied how three different interface styles, namely a traditional GUI, a cartoon-like agent and a realistic agent, assist users in a spatial task through a specially designed experimental investigation. The experiment involved two tasks with the first task being a hide-and-seek scenario, and the second one a 3-D puzzle. The experiment employed within-subjects design with 12 participants in the experiment with each subject requiring between 45 to 75 minutes to complete their actions. In truth, the system was simulated to observe and interpret user actions, though none of the subjects recognised that they were not interacting with an agent. After the completion of each task, subjects were asked to draw a line with the length representing their affective feelings toward the agents, thereby enabling data to be collected and analysed quantitatively. Additionally, this experiment also utilised an open interview to determine the feelings of subjects toward agents, their preferences and the reasons.
In the experimental study of Baylor and Kim (2009), a 2x2x2 factorial design was employed to investigate the effects of various types of instruction (procedural module, attitudinal module), deictic gesture (presence, absence) and facial expression (presence, absence) on student perception of pedagogical agents. The experiment was designed with two different instructional modules with the first one being a procedural instructional
73 module to teach the subjects how to use a web-based software program while the second one involves an attitudinal instructional module that was developed to elicit more desirable attitudes in students towards intellectual property rules and laws. Both modules employed between-subjects design with 120 participants in the procedural instructional module and 117 subjects in the attitudinal instructional module. Subsequent to the completion of the instructional modules, subjects were asked to complete an online questionnaire, which necessitated around 20 minutes to complete the module phase and another 20 minutes to finalise the questionnaire. Afterwards, all the data collected in the online questionnaire was analysed quantitatively.
Numerous agent interface studies have been undertaken through experimental methods, given that experimentation is a reasonable and scientific approach to realise how interactions occur between users and agents. This particular research employed experimentation as the method to explore the affective influence of agent interfaces on users as well. In reality, it involved the conducting of two experiments to investigate the affective feeling of users toward different agent interfaces. Questionnaires were designed for subjects to quantify their affective feelings toward agent interfaces while semi- structured interviews were held after interactions between subjects and ECAs to gain greater insight into the answers supplied in the questionnaires. More details about the experimental design and result findings will be presented in next part of this thesis, the experimental part, which runs from Chapters 5 to 8.
This chapter offers an agent interface related literature review of prior research. In fact, agent interfaces descend from user interfaces, thus, this chapter presents the relationship between user interfaces and agent interfaces from a historical, elemental, design and HCI perspective. Additionally, this chapter addresses how the visual impact of agent interfaces affects users in terms of experiences, behaviours and performances. The chapter also illustrates the most common methods that are adopted in agent interface related studies. Agent interfaces also form an important component of this research and they are also critical in HCI-based work on the above cited studies. This research attempts to build upon previous research experiences and intends to study unknown affective interactions between users and agent interfaces. This chapter also continues certain academic theories in relation to the core components that formed the previous chapter, which presented ECAs related literature review and addressed other core components, in particular agent
74 interfaces. The other core component, affective factors, will be presented in next chapter by means of a detailed literature review.
75
Chapter 4. Affective Factors of Intelligent Agents
This research aims to investigate a number of human affective factors that users respond to affective interfaces of ECAs. Literature reviews about ECAs and agent interfaces related to this research have been presented in the previous two chapters, while the focus of this chapter is the other core component, namely affective factors. This chapter provides these affective feelings related literature review from a psychological aspect.
This chapter is divided into five sections as follows: 1) What is Affect?, 2) Affective Displays – Emotions, 3) Affective Science, 4) Affective Factors and Influences, and 5) Affective Agents. Affective factors represent one specific subgroup of psychology, and some theories in relation to affect recognition in psychology, such as Tomkins’ nine affects and the relationship between affect and cognition, are discussed in the first section. While emotions represent the main approach of humans to express feelings, the second section evaluates from affect to emotion and points out the dissimilarities between emotions and moods. Similarly, human feelings are delicate and problematic to measure objectively. The principal focus of the third section is the various scientific measurements of affective factors. Subsequently, relevant affective factors in this research and their linkages and influences are discussed in the fourth section. Moreover, ECAs, in this research, were developed with affective emotions, and the fifth section demonstrates other agents, which are designed with affective features, such as relational and believable agents.
This chapter, which is the end of the theoretical component, mainly addresses affective factors related literature review. The following part encompasses the experimental aspect, entailing four chapters to present experimental design, actual procedure and the experimental results of two surveys, the hardware and character interface surveys.
4.1. What is Affect?
The purpose of this research is to investigate the alterations in human affective feelings when users interact with various agent interfaces, and these fundamental affective factors, including engagement, empathy, preference and user-agent relationships. In truth, affective factors have originated from the concept of affect in psychology. The word affect is defined as a noun in psychology to mean “emotion or desire as influencing behaviour” (Oxford University Press, 2011a), while affective is an adjective in
77 psychology “relating to moods, feelings, and attitudes” (Oxford University Press, 2011b). Alternatively, Neuman, Marcus, Grigler and MacKuen (2007, p. 9) classify affect as “the evolved cognitive and psychological response to the detection of personal significance”. In general, as mentioned in Chapter 1, affect as a noun typically refers to an emotional or psychological state. This research addresses variations of human affective factors when users encounter different affective agent interfaces.
For two hundred years, many psychologists have been aware ‘the trilogy of the mind’ comprises cognition, affection and conation (Hilgard, 1980). While cognition oversees perception, storage and processing of information, affect addresses the evaluation of information, and conation manages the interaction of cognition, affect and actual behaviour (Neuman et al., 2007). Although cognition is ascendant as a result of the current decline of the affective-conation dimensions in contemporary psychology, affect remains recognised as a major classification of mental events (Hilgard, 1980). This indicates that affect has been identified and investigated for a considerable time in the psychology domain. However, affect is not only a component of mind, for example Breckler (1984, p. 1191) regards affect, behaviour and cognition as distinct components of attitude, and defined affect as “an emotional response, a gut reaction, or sympathetic nervous activity”.
Affect generally receives significant comment from researchers in psychology in the literature. One example is Kelly’s (2009) definition of affect as a biological event, and a normal function of central nervous system (CNS), such as a knee-jerk reflex. Furthermore, Nathanson (2008) explains several affect-related experiences in order to introduce Affect Imagery Consciousness (AIC) as defined by Tomkins. In fact, such affect refers to “a group of nine highly specific unmodulated physiological reactions present from birth”, whereas feeling refers to people aware that an affect has been activated (Tomkins, 2008, p. xiv). Tomkins has extended Charles Darwin’s research by collating facial expression to posit the theory of nine affects, where distinct facial patterns are categorised into positive, negative and resetting affects (Kelly, 2009). In general, categories such as interest- excitement and enjoyment-joy belong to the positive category, surprise-startle is typically placed in the resetting category, while distress-anguish, fear-terror, shame-humiliation, disgust, dissmell and anger-rage are in the negative category (Kosofsky-Sedgwick, Frank, & Alexander, 1995). Additionally, core affect is another psychological term that Russell (2003, p. 145) has defined as “the heart of emotion, mood, and any other emotionally charged event and states experienced as simply feeling good or bad, energized or enervated”. Moreover, core affect is commonly regarded as a feeling, and is an example
78 of a neurophysiological state that is heavily influenced by feelings, moods and emotions. This particular research studies human emotion related affective factors, such as preferences, relationships and engagement.
The character interface survey in this research involves a language practice task, and studies user engagement, character preferences and user-agent relationships, as these types of affective factors are commonplace during language learning. In addition, other researchers in the psychological domain have defined affect-related terms in relation to language learning. In this area, Gardner and MacIntyre (1993) describe affective variables of second language acquisition as relevant emotional characteristics that influence the human response to any situation. Similarly, Schumann (1975) pointed out that affective variables in adult second language acquisition might possess more importance than biological maturation. Actually, several affective variables, such as attitude, motivation and empathy, have been considered critical in second language learning.
Most theories consider affect to be related to cognition, ands on the basis of the traditional approach, it is regarded to be post-cognitive. A prior cognitive process reacts on an affective reaction. There are several approaches and features that are examined for their value and weighted for their contributions. After completing the analytical task, an overall affective judgment is generated (Lazarus, 1982). In other words, affect is generated after a certain amount of cognitive processing has been accomplished, although between cognitive and affective components, the cognitive may be dominant in some cases. Alternatively, cognitive and affective components may be equal to each other, and in some cases, the affective factors may be more important than cognitive factors in other cases (Bettman, 1982).
Given the theories described above, it is apparent that cognition and affect interact with each other. However, Zajonc and Markus (1982) stated that despite recognition having the potential to influence affective ratings of stimuli, it was not an essential factor. In fact, affective reactions may occur without extensive perception and cognition, and may be generated sooner and with greater confidence than cognitive judgments (Zajonc, 1980). This indicates that affect, as defined above, can vary independently of other factors. While cognition may transpire independently of other aspects, affect is typically closely related to other affective issues. Furthermore, this research does not intend to investigate cognition as the primary affective factor during HCIs.
79 The underlying interdependence is shown by Russell (1980, p. 1161) who states that evidence exists that indicates “affective states are not independent of one another, but are related to each other in a highly systematic fashion”. In truth, affective factors are generated from human affective feelings, such as attitude, motivation and empathy. Moreover, given humans are not always rational beings, decisions and judgments are heavily influenced by emotional states. For example, Norman (2002, p. 38) claims that “pleasing things work better”. Therefore, when individuals are faced with difficult tasks, a positive affect is likely to enable persons to accomplish this task more rapidly and easily. On the contrary, a negative affect has the capacity to render simple tasks extremely complicated. Furthermore, affective factors influence internal human feelings, and might be altered by external events, as Russell (2003) hints that a single external objective event can easily alter human states. Similarly, affect impacts social interaction as well as the major currency in social interactions (Zajonc, 1980).
According to the above statements, it is clear that affect factors occur independently, albeit with close linkages to other factors. However, humans are not entirely rational, meaning that human affective states might be altered by external events and affective factors might impact other affective reactions. This research intends to investigate how agent interfaces affect human affective internal feelings during HCIs, and further influence user engagement, character preference, user-agent relationships and emotional influences.
4.2. Affective Displays – Emotions
Emotional expressions are pivotal in developing interpersonal relationships (Ekman, 1999a), and this is particularly simple for humans, given that even an infant is able to learn to impersonate expressions within a few hours of his/her birth (Rosenberg & Ekman, 1999). Moreover, emotional expressions of agents might impact the interactions between users and agents. The hardware interface survey investigates how agent emotional expressions influence user emotions. Therefore, agent affective displays are crucial in this research. Affect psychology includes several different ideas about the origins of emotions (Kelly, 2009). This concept is important, as humans display affect with observable expressions of emotions called affective displays, such as facial expressions, hand gestures and tones of voice.
80 In general, face forms an integral part of the affect system. Tomkins (1995, p. 217) claims that “affects are primarily facial behaviors and secondarily outer skeletal and inner visceral behavior”. As a consequence, the optimal way to distinguish discrete emotions is through facial expressions. In fact, the idea that emotions have distinguishable features is traced back to Darwin (1998) who discovered that facial, physiological and behavioural processes are connected to different emotions, and are comparable between humans and animals. In addition, Tomkins (1962) stated that a limited number of basic emotions exist called affect programs. Therefore, affect programs are determined and distinguishable products on the basis of humans’ bodily, behavioural and expressive features.
In terms of human emotion, cognition is another affect component that theorists give crucial prominence. As previously addressed, Zajonc and Lazarus have argued considerable interplay exists between affect and cognition (Kleinginna & Kleinginna, 1985). Zajonc and Markus (1982) pointed out that the expressive aspect of emotions is the primary driver of affect. Additionally, recognition influences affect, however, affect comes as the first experience in some circumstances. Alternatively, emotions may occur as the primary experience in some cases, but could also be affected by cognition as the foremost post-cognitive factor. In contrast to this theory, Lazarus (1982) contends that cognition arises prior to emotion and highlights the cognitive processes in emotions. In reality, cognitive appraisal occurs in specific stages through the evaluation of information processing from the environment, and the appraisal is undertaken in a deliberate, rational and conscious manner. In addition, Batson, Shaw and Oleson (1992, p. 298) state affect “has tone or valence (positive or negative) and intensity (weak to strong)”, and claim mood and emotion are forms of affect that also contain complex cognitive processes. This underlines that whether cognition is involved in affect processes or not, emotion is the primary external form of expression. As stated previously, cognition is not the affective factor that this research focuses on. Furthermore, affective factors exist with valence and intensity. This research studies short-term affect in the hardware interface survey and long-term affect in the character interface survey in order to develop agent interfaces with more positive affective influences.
Two main categories exist in the affective phenomenon, namely emotions and moods. In general, moods are long-lasting affective states that are somewhat sub-conscious, and their effect on human thoughts and behaviours tend to be of higher potential and more consistent (Forgas, 2006). On the other hand, unlike the more insidious moods, emotions are more intense and pertain to some cognitive content. This is likely because emotions emanate from specific knowledge, although their influences tend to be less consistent and
81 highly situational (Forgas, 2006). Nathanson (2008) also defines emotion and mood, where the term emotion is used to describe an affect that has been triggered through memory of previous experiences, and the expression mood refers to an affect that has been triggered through immediate experiences and reminds the individual of a similar historical experience. In fact, both emotion and mood are forms of affective responses and involve tone and intensity. In contrast, Batson et al. (1992, p. 301) distinguish emotion from mood in terms of where “mood reflects a change in expectation about the general likelihood of positive or negative affect in the future, emotion reflects the existence of a specific goal or of perceived change in one’s relation to a specific goal in the present”. Therefore, mood refers to an affective response of immediate experiences and potential influences on future affects, while emotion refers to an affective response of previous experiences and impacts existing affects. Moreover, Brave and Nass (2003) differentiated emotion from mood through their object-directedness, given that the occurrence of emotions typically involves a specific object. Although a particular object may trigger shifting of moods, they are not fully engaged in the specific object, albeit experienced more generally. For instance, individuals may be sad about something (an emotion) and then feel depressed (a mood). Short-term emotions are specifically investigated in the hardware interface survey, such as the immediately emotional influences. Additionally, long-term moods are scrutinised in the character interface survey, for example agent interfaces further influence user engagement and user-agent relationships.
Several studies have demonstrated that a relationship exists between emotional expressions and feelings (Hanger & Ekman, 1983). An emotional response involves tone and intensity in human internal states, and it also involves multiple somatic parameters, such as endocrine, visceral, autonomic, and musculoskeletal factors (Adolphs, 2005). However, this underscores the exact cause of emotions and moods. This research aims to investigate affective interfaces of ECAs and create better design of agent interfaces for a language-learning platform. As a result, enhanced understandings of the way in which emotions and moods’ originate are likely to assist the creation of more productive interfaces. Brave and Nass (2003) stated that emotions may originate from specific needs, goals, appraisal theories, contagion, moods, sentiments, and previous emotional state, while moods may be triggered by contagion, colour, and other effects. This demonstrates that emotions and moods not only influence human differently, but are also triggered by a variety of means.
82 This research applies emotional expressions to ECAs. A brief history of facial expression research begins with Tomkins - inspired by Charles Darwin - introducing his nine-affect programs (surprise, interest, enjoyment, anger, fear, distress, shame, disgust and dissmell). Subsequently, Ekman, Friesen and Ellsworth (1972) tested Tomkins’ hypothesis and produced the facial expressions of six basic emotions, namely anger, happiness, fear, surprise, disgust and sadness. The universal recognition of six emotions was the product of several decades’ worth of cross-cultural research (Neuman et al., 2007). In addition, Ekman (1999b) provided more than nine forms of evidence to prove that facial expressions of emotion are universal.
In addition, human affective factors are closely linked to other factors, such as emotions. Emotions are signals for other people to notice our reactive emotions and sentiments (Oatley, 2004). Some emotions are likely to influence one affective process, and other emotions are able to impact another affective process in the mean time (Ekman & Davidson, 1994). Moreover, an individual’s emotions can influence the emotions, thoughts, and behaviours of others, and in turn their response can influence future interactions - a phenomenon that persists until it generates an emotional cycle (Hareli & Rafaeli, 2008). The hardware interface survey studies how agent facial expressions affect user emotions while the character interface survey investigates user moods during HCIs and the influences on affective connections between users and agents. Thus, the emotional expressions of agents form an important part of this research. More emotion design for ECAs in this research will be addressed in Chapters 5 and 7.
4.3. Affective Science
In general, human affective feelings are difficult to measure objectively. Affective science is the scientific study of affects or emotions, and involves the causes of emotions, emotional expressions, emotional experience, and recognition of emotions in its scope (Kring & Moran, 2008). The major focus of affective science is measuring the subjective aspect of emotions and feelings in an objective manner.
Emotion is linked to other psychological branches, such as clinical and social sciences, personality, cognitive psychology and behavioural neuroscience (Ekman & Davidson, 1994). Additionally, affective science is an interdisciplinary domain that integrates psychology, sociology, anthropology, ethology, neuroscience, science, psychiatry and other domains. In truth, increasing attention in affects and emotions is addressing
83 praxeological, biological and social science aspects, although research indicates that investigating human behaviours and social activities cannot be entirely understand without considering affective factors. However, certain limitations within affective science may hinder its ability to reach a thorough understanding of emotions.
As discussed previously, the primary challenge is the argument of the relationship between emotions and cognitions. In fact, debate remains whether emotions occur more primary than cognitions or whether emotions are post-cognitions. Larsen and Fredrickson (1999) addressed a number of other issues, such as timing, context, reliability and validity, in terms of emotions. Firstly, timing is an issue because emotions are dynamic processes that occur, linger, and then dissipate over time, meaning it is difficult to isolate the targeted emotion period. Secondly, context in relation to emotions refers to the consideration of psychological factors and bodily experience, as these contexts without doubt impinge on measuring emotions. Thirdly, reliability and validity are concerning issues within affective research, and both are important factors for all measurements. In truth, emotion is an exceptionally complicated affective state, and simple correlations and associations cannot be used to measure the reliability and validity of emotion. However, major challenges within the interdisciplinary domain of affective science and the study of emotion remain. Though several scientific methods are used to measure emotions, not all of these methods are suited for every type of situation.
The measurement of human emotions provides a clear insight into what a person is feeling. Monitoring physiological responses (such as heart rate and galvanic skin responses) and collecting verbal reports and facial expressions are some of the approaches to measure affective emotions (Breckler, 1984). The existing approaches are categorised into three types, measuring, observing and self-reporting.
Measuring approaches include those methods that use scientific instruments to measure human physiological responses, such as skin conductance, heart rate and finger pulse volume (Kring & Moran, 2008). For example, one method employs an electron- cephalogram (EEG) to measure neurological changes (Brave & Nass, 2003). Additionally, two meta-analyses, positron emission tomography (PET) and functional Magnetic Resonance Imaging (fMRI), are utilised to collect emotion-specific neural circuitries and correspondence between each emotion (Phan, Wager, Taylor, & Liberzon, 2002).
Furthermore, observing approaches refer to those methods where observable external affective displays are analysed, including facial expressions, vocal speeches and bodily gestures. In fact, facial expressions are the most obvious way to detect others’ emotional 84 states. Paul Ekman, a pioneering psychologist in the field of cross-cultural emotions and facial expressions, started his research on facial expressions and body gestures in 1954, and expanded his interest in an evolutionary and semiotic frame of reference (Paul Ekman Group, 2011). Ekman, Friesen and Ellsworth (1972) described the six basic emotions, and Ekman and Friesen’s (1976) Facial Action Code (FAC) and advanced version, the Facial Action Coding System (FACS) (Ekman & Friesen, 1977), further describes a specific set of muscular movements for each emotion and provides a comprehensive descriptive system for measuring facial action, consisting of 46 action units (AUs). FACS is likely to measure the visible distinguishing facial actions successfully as its creators proposed (Hanger & Ekman, 1983). Some AUs refer to the movement of specific facial muscles, and some are the observable changes in the face (Ekman & Friesen, 1977). For example, as shown in the Figure 16, AU 1 refers to the raising of the inner brows, AU 4 means that the brows are drawn together and lowered, and AU 1+4 indicates that the outer brows are lowered and the medial portion of the brows is raised and pulled together (Tian, Kanade, & Cohn, 2001). This facial coding is useful to measure emotion and is one of the most widely adopted for facial expression recognitions. Furthermore, Bartlett, Hager, Ekman and Sejnowski (1999) applied three approaches, holistic spatial analysis, explicit measurement of features and estimation of motion flow fields, into FACS as a hybrid system, and its best performance achieved 92% correct for classifying six upper facial actions.
Figure 16. Upper unit action units and some combinations (Tian et al., 2001).
85 In addition to facial expressions, voice and gesture are other indicators to observe human emotions. Human voice conveys some information related to emotion through rhythm, pitch range and amplitude while speech provides more information than the verbal message, such as the background and emotional state of speaker (McIntyre & Göcke, 2008). In summary, human external expressions can be observed from facial expressions, voice and speech as well as gestures to recognise the emotional states of individuals.
Finally, self-reporting is the method that measure human affective states through an interview with the subject or questionnaires. Although self-reporting is only capable of measuring the conscious experience of emotion and mood, it remains the most direct way to measure affective states (Brave & Nass, 2003). Larsen and Fredrickson (1999) declare that two techniques of self-reports exist, namely single-item measures and multiple-item measures. Single-item measures are the method used where research subjects are invited to rate their feelings on a single emotional construct. For example, asking subjects a question like ‘How happy do you feel?’ where subjects could answer from ‘not at all happy’ to ‘extremely happy’ (unipolar scale) or from ‘unhappy’ to ‘happy’ (bipolar scale). Moreover, multiple-item measures refer to various assessment instruments, and most of them contain lists of adjectives describing emotional states. For instance, the checklists for subjects to check all the affective states that he/she is feeling, and the rating tasks for subjects to rate the specific emotion for the degree to which he/she is feeling. Since self- report measures have been widely adopted to measure emotions, several emotional adjective rating scales have been developed to measure human emotions. One of the most employed rating scales is the Positive Affect Negative Affect Schedule (PANAS), which focuses on positive affect (PA) and negative affect (NA), and contains ten items on each of the two scales (Watson, Clark, & Tellegen, 1988). While PA reflects human emotions, such as enthusiasm and attentiveness, NA refers to the feelings like distress and displeasure.
Although Ekman and Davidson (1994) claimed that researchers should not rely on self- reporting too much, Ekman, Friesen and Ancoli (1980) discovered that subject’s self- reports of happiness were directly connected to the specific smiling action. In fact, there is a precise and refined relationship between different instinctive expressions and distinct emotions (Hanger & Ekman, 1983). Rosenberg and Ekman (1994) demonstrated that there was a tough correlation between expression and self-reporting when subjects provided the means of retrieving memories for specific emotional experiences in time. Therefore, self-reporting is suggested to immediately self-report after an emotional episode (Ekman & Davidson, 1994).
86 Although affective states are difficult to measure scientifically, various methods have been developed to recognise human emotions. Among those methods, the self-reporting method is the most widely employed by researchers to identify human feelings as well as being perhaps the most efficient and easiest technique to measure human affective states (Larsen & Fredrickson, 1999). Thus, this research adopts self-reporting methods as the measure to ask subjects about their own affective feelings toward ECAs in these two surveys. Furthermore, questionnaires have been designed to ask subjects to self-report the affective influences of ECAs on themselves. The following sections will address the scope of affective factors and what influences may result related to this research.
4.4. Affective Factors and Influences
This research intends to investigate the affective factors between agent interfaces and users, and especially focuses on emotional influences, character preference, user engagement and user-agent relationships. In terms of affective factors, it is necessary to discuss what affect is. Affect in psychology refers to human emotions and moods, and it is defined as “the emotional feeling, tone, and mood attached to a thought, including its external or physical manifestations” (Durrenberger, 1999, p. 169). As mentioned in Chapter 1, affective factors are those responses affected by human emotions and feelings, such as empathy, attitude, mood and motivation.
Brave and Nass (2003) pointed out all attention, memory, performance and assessment are effects of affect. In truth, emotion could completely absorb the subject and is likely to influence attention. For example, becoming angry with an onscreen agent may ignore other information onscreen. In addition, emotional reactions can occur when others are not in the attendance, such as react to music, thunder and the loss of physical support (Ekman, 1999a). According to Brave and Nass (2003, p. 85), “emotion’s effect on attention also has implications for memory”. Humans typically have better recall mood- incongruent at certain times, such as a pleasurable memory while in a bad mood. Furthermore, affect has also been found to influence performance and decision-making. As an illustration, physicians in the affect-induction group, in which they received a small package of candy, arranged information more efficiently than others in the control group (Estrada et al., 1997). In addition, users browsing online shopping website in an optimistic mood are more likely to judge products or services positively. Oatley (2004) also concluded that moods, sentiments and preferences are included into the family of emotion. Sentiments are long-term emotional states relating to others while preferences
87 are silent emotions that will be revealed when humans make decisions. In short, affective factors are related to other human emotions and feelings affect and influence them.
This research aims to investigate the influence of agent interfaces on human-related affective factors, such as emotional influences, user-agent relationships, character preferences, and focuses specifically on user engagement. Poggi (2007) defines engagement as “the value that a participant in an interaction attributes to the goal of being together with the other participant(s) and continuing the interaction”. Additionally, Chapman (1997, p. 3) stated that “something that ‘engages’ us is something that draws us in, that attracts and holds our attention”. In fact, engagement can be affected by some human affective factors, for example, people might pay attention to something because personal interest or curiosity.
Engagement is one of the most popular features of a quality multimedia product (Barker & King, 1993). Furthermore, Chapman, Selvarajah, and Webster (1999) advise that engagement should be integrated into software, the appropriate aspects of which can be engaged by the users. For these reasons, engagement forms a critical part of a technological product, especially a multimedia one. In addition, engagement in user- computer interaction “is a category of user experience characterized by attributes of challenge, positive affect, endurability, aesthetic and sensory appeal, attention, feedback, variety/novelty, interactivity, and perceived user control” (O'Brien & Toms, 2008, p. 941). As a result, this clearly indicates that engagement is essential to HCIs and is influenced by several affective factors. According to van Vugt et al. (2007), increased engagement with an interface agent can improve user performance. Moreover, engagement is an important clue in predicting user satisfaction with HCIs. As mentioned earlier, the majority of human affective factors are closely linked to others factors, and user engagement is one of them. Hence, the preferences of users for agent interfaces affect user engagements, which in turn affect the relationships developed between users and agents.
Several studies have developed engagement measurements for various purposes. Playfulness is an appropriate component to measure user engagement when users use spreadsheets, word processors and presentations (Laurel, 1991; Webster & Ho, 1997). Additionally, attentive and emotional processes are two other components that lead to user engagement (Peters, Asteriadis, Karpouzis, & de Sevin, 2008). Chapman et al. (1999) measured engagement in multimedia training systems through attention focus, curiosity and intrinsic interest. Furthermore, user engagement could be enhanced by
88 aesthetics of agents (van Vugt et al., 2006). As is widely known, usability is the primary requirement for designing a product. However, design for a more engaging experience is a trend in HCI studies beyond usability. In the view of O’Brien and Toms (2008), engagement is an experience that includes more components than usability, such as aesthetics, novelty and endurance.
This research also intends to investigate human affective factors that relate to user engagement, such as character preference and user-agent relationships. In this research, the term user engagement applies to people’s experiences with computer technology and is defined as the value that a user in an interaction attributes to the willingness to spend time with the agents and continue the interaction. Furthermore, the interaction between users and agents in the character interface survey was a learning task. Learning attitude is a task-related component that augments user engagement as a means of boosting users’ learning engagement.
User engagement is primarily measured using questionnaires or interviews, where users recount their experiences, attitudes and behaviours (Gulz & Haake, 2006). Several human affective factors are involved in user engagement during HCIs. In the past few decades, studies have addressed numerous components for measuring user engagement based on different conditions and purposes. In this work, van Vugt et al.’s (2007) framework is adopted as part of the measurement. In reality, van Vugt and his colleagues (van Vugt et al., 2004; van Vugt et al. 2006) adjusted the framework of Hoorn and Konijn (2003), PEFiC, into an integrative framework, I-PEFiC, to explain user engagement with embodied agents. This engagement framework consists of parallel processes of involvement and distance being congruent tendencies to approach and avoid agents. Konijn and Hoorn (2005) provide evidence showing that the most appropriate explanations of liking a virtual character are both involvement and distance experiences. The involvement process is where a user feels engaged with an agent, and one of the measured issues, for example, is ‘This agent appeals to me’. On the other hand, the distance process is the tendency to avoid agents where one of the measured issues is, ‘I dislike this agent’. Furthermore, involvement and distance processes occur in parallel with each other because general effects are mediated further by valences of positivity or negativity. For example, the irrelevant facial expressions of agents may not be added to the distance process but may decrease the involvement process.
As mentioned above, human affective factors are closely linked to other factors, as is user engagement. One affective factor in relation to engagement that this research intends to
89 explore is emotion. In reference to this research, emotional influence refers to internal feelings that originate from human goals, thoughts, and concerns felt toward ECAs during HCIs. The hardware interface survey intends to study how the agent emotional expressions influence user internal emotions and feelings.
Another affective factor that this study aims to investigate is relationships, as defined by Kelley (1983) as interdependent behaviours between people. Furthermore, Kelly, who is the first Training Director of The Silvan S. Tomkins Institute, developed the Blueprint for Intimacy as follows (Nathanson, 2008, p. 9):
1) Intimacy requires the mutualisation and maximisation of positive affect. 2) Intimacy requires the mutualisation and minimisation of negative affect. 3) Central to intimacy is the requirement that we disclose our affects to each other. 4) Anything that increases our power to accomplish these three goals for intimacy, anything that reduces this power is bad for intimacy.
In the context of this research, relationship refers to humans generating a number of interdependent behaviours to ECAs during HCIs.
The other affective factor that this study aims to investigate is preference, defined by Zajonc and Markus (1982) who have claim that a preference is the behavioural tendency where a person does not think or say much about the object, but looks at how the person acts towards it. Moreover, affective factors play a major role in the maintenance of preference, and this also proves that affective factors are linked closely to other factors. In this research, preference refers to users generating a predisposition in favour of the ECAs during HCIs.
This section addresses those affective factors of user engagement, character preference, user-agent relationships and emotional influences that this research addresses. More specific measurements that this research adopted will be presented in Chapters 5 and 7.
4.5. Affective Agents
This research focuses on the affective responses of users towards ECAs with different interfaces. The ECAs that were developed for this study were equipped with emotional expressions. Haake (2009) describes affective agents as the agents that have the capacity to detect and react to human’s emotional states and behaviours. The domain of intelligent 90 agents focused on the ability to recognise and generate emotional expressions referred to as affective computing, which has been discussed in Chapter 2. Affective agents are one subgroup of affective computing, which employs affective factors on AI, including agents, robots and computerised systems or devices. Additionally, affective agents consider intelligent agents that are able to recognise and/or express emotions, especially on embodied agents.
Similar to affective computing, studies of affective agents have also concentrated on recognising human emotions and generating their own emotion and express them. Minsky (1985, p. 163) addressed the issue of agents needing emotions as being “not whether intelligent machines can have emotions, but whether machines can be intelligent without any emotions”. When virtual characters interact with users, the usability of agents is the first aspect to be considered. While functional agents, such as librarians and travel agents, are likely to be more rational and logical without emotional involvement, HCI researchers have been interested sought to increase agent credibility. This has been achieved by investigating emotions and personalities of agents, such as the studies of Bates (1994) and Reilly (1996). In fact, emotions and personalities of agents are the factors that provide agents with additional humanity. In addition, Stern (2002) pointed out that users have the potential to build emotional relationships with virtual characters – relationships that more than readers for the fictional characters in traditional stories, and resemble real friendships. In truth, several artificial characters with emotions and personalities have appeared in literature and science fiction, ranging in identification from good to bad guys, such as Pinocchio, R2D2, Frankenstein and HAL900. This shows that humans often have meaningful feelings toward virtual characters and can build delicate relationships with them.
As agents continue to become more humanised, users hope agents would be more considerate and thoughtful towards them. In this endeavour, it is likely to be exceptionally beneficial for agents to be furnished with as many positive affective features as possible. In fact, agents are likely to gain affective responses from users mostly because agents are equipped with personalities that are similar to users or that users are likely to sympathise with. Moreover, emotions and personalities are the two features make agents tend to social computing (Ball & Breese, 2000). Emotional expressions play an important role in this research since it intends develop better agent interfaces, which have the capacity to further affective connections with users.
91 Numerous affective agents have been developed worldwide. Rational agents (Bickmore, 2003) are one example of affective agents, and these are developed to maintain long-term emotional relationships with users. Those agents are able to save information in relation to past interactions with users, and further utilise the information in the computational models for maintaining relationships with users. In addition, poker-playing agents in Koda and Maes’ (1996) studies are additional examples. These agents were permitted to have emotions, and ten emotional expressions each, namely neutral, pleased, displeased, excited, very excited, anxious, satisfied, disappointed, surprised and relieved. Furthermore, subjects in this study generated different levels of preference toward those poker-playing agents. Moreover, Reilly (1996) designed a methodology for agent builders to develop social behaviours for credible agents through developing seven agents to drive and contextualise a framework, including believable emotional and social agents.
Despite increasing user awareness that virtual characters are not real, these characters may feel real to users. For instance, people often have negative feelings when characters suffer and feel joy when characters succeed. According to (Stern, 2002, p. 353), “designers should be aware that man-made characters have the potential to have a powerful effect on people”. Functional agents have impacts only on usability for users, whereas affective agents have a potential effect on individual mental states.
In this thesis, the focus is on human affective factors towards affective interfaces of ECAs. This chapter succeeds the former two chapters to continue to address affective factors in the literature review. In this chapter, the definition of affect by several psychological theorists, the differences between emotions and moods, the methods to measure affective factors scientifically, the impact of affective factors on human internal and external aspects, and the combinations of affective factors and AI are offered. This chapter not only covers a literature review of the affective domain in this research, but also addresses the relationship of these fields to this research to demonstrate the important roles in HCIs. The following sections involve the experimental part, which is contained from Chapter 5 to 8 and two experiments that were conducted to investigate affective interfaces of ECAs on users, including methodologies, measurements and results will be presented.
92 Part II. Surveys on Hardware and Character Interfaces of ECAs
After investigating existing developments and design guidelines of affective interfaces of ECAs from technological, design and psychological perspectives, and undertaking literature reviews in relation to these three core components, it was discovered that two missing sections necessitated consideration among affective interfaces of ECAs. These areas of concern determine if any distinct affective responses (such as emotional influences and user-agent relationships) towards ECAs occur when ECAs are applied to various devices (such as computers and portable devices), and whether ECAs have any different affective influences (such as character preference, user engagement and user- agent relationships) on users when ECAs were represented by varied character classifications (a human, an animal and a creative creature classifications). Furthermore, the relationship between the various human affective statements (character preference, user engagement and user-agent relationships) in HCIs should be ascertained.
Therefore, two surveys containing two particular experiments were conducted in this experimental part in order to investigate the affective influences of ECA interfaces, encompassing different devices with varied characters. This part presents two surveys on hardware and character interfaces of ECAs. The first survey entails a hardware interface survey that investigates the human affective feelings towards various ECA hardware interfaces. Its experiment design and the developments of its experimental system and ECAs will be addressed in Chapter 5, and its actual procedure and analytical findings will be presented in Chapter 6. Subsequently, the second survey encompasses a character interface survey that studies the affective influences of agent character classifications on users during HCIs. In a similar vein, its experiment and agent design will be covered in Chapter 7, and its procedure and results will be addressed in Chapter 8.
After detailing the theoretical and experimental parts, the final aspect comprises the practical part. This intends to tackle the practicality of adopting existing and experimental guidelines on agent interface design on a multi-agent website as well as summarising and concluding this research.
93
Chapter 5. Method of Hardware Interfaces of Chatbots
The purpose of this research is to examine the alterations of human affective feelings towards various agent interfaces during HCIs. This section commences the experimental part, in which two surveys are presented to explore the affective influences of agent interfaces on different devices with various character classifications. This part from Chapters 5 to 8 will encompass the designs of experiments, ECAs and systems, and the experimental results. Additionally, this particular chapter presents the experiment and ECAs design of the hardware interface survey. The hardware interface survey aims to investigate the various user emotions towards different hardware interfaces of the ECAs. One experiment has been conducted to discover answers to the research question of when ECAs appear in distinct hardware interfaces (computers and portable devices), and when do ECAs’ level of affective influences (emotional influences and user-agent relationships) on users fluctuate significantly?ġ
This chapter comprises four sections as follows: 1) Method, 2) Measures, 3) Materials, and 4) Hardware Interface Design of Chatbots. The first section discusses the experimental design of the hardware interface survey, such as various variables, conversational interface design and chat flow of chatbots. The second section addresses the measurements of the questionnaire, including categories of emotional influences and user-agent relationships. The third section contains an introduction of the concept of AIML, which is the NLP employed in this research and the emotional designs of ECAs. Finally, the fourth section will encompass issues, such as the importance and employment of hardware interfaces.
This chapter predominately focuses on the experimental design aspects of the hardware interface survey while additional actual procedures and results of this survey are discussed in Chapter 6. Subsequently, another survey in the experimental part, the character interface survey, will be presented in Chapters 7 and 8 along with its precise details, actual procedure and experimental results.
5.1. Method
The hardware interface survey has been designed to ascertain what type of hardware interface of ECAs is the optimal form to develop interactions with modern users. This survey has adopted experimentation as its research methodology, as mentioned in Chapter 95 4, given that this type of research is the most scientific manner to study agent related interfaces. This experiment aims to discover the emotional influences of different hardware interfaces of ECAs. Thus, the hardware interfaces of ECAs as the independent variable, and the attributes are computers and portable devices. In this case, the dependent variables were the degrees of emotional influences of chatbots on users and the relationships between chatbots and users. Possible considerations included the ability to deliver emotional influences to users and build closer relationships with users. Control variables encompassed using similar animal characters, conversational database, user conversational interfaces and adopting keyboard conversations as communication approaches. A one-factor-at-two-levels3 (computer interface – portable device interface) counterbalanced within-subjects design 4 was employed in this study. In fact, this experiment necessitated subjects to compare two types of ECA hardware interfaces. Therefore, within-subjects design was adopted for subjects to interact with both ECA hardware interfaces as well as offering further comparison between them. Furthermore, this experiment involved two chatbots, which were introduced to the subjects on two different devices. The independent variable in this survey is the devices that subjects operated during the experiment. One level referred to computers which as the control group, and the other level referred to portable devices which as the experimental group in this experiment. As shown in Table 4, one character referred to as Gary was designed for computers on the World Wide Web, and the other one known as Gavin was developed for portable devices on the Mobile Web. In fact, the purpose of these chatbots was to assist subjects to complete two tasks in the hardware interface experiment.
3 The devices were an independent variable designed in this experiment. The independent variable consisted of two discrete levels where one level referred to computers and the other level to portable devices.
4 Within-subjects design: within-subjects design refers to a type of experimental design in which all subjects are served in every treatment. In this experiment, all subjects experienced keyboard conversations with both the computer chatbot and the portable device chatbot.
96
Table 4. ECAs as chatbots in the hardware interface survey. Name Gary Gavin
Character
Hardware device
Computers: Portable devices: iPhone Apple MacBook Air or ASUS F8Sa or iPod touch
The conversational user interfaces for the experiment are similar in both devices, and the differences between them are mainly in the domain of size, chatbot characters and names. Figure 17 presents the conversational user interface of Gary on a computer. The program displays character facial expressions as well as an introduction, an output frame, an input frame and a submit button within the interface. This approach enables subjects to type sentences in the input frame, and then press the submit button to send questions to chatbots. Subsequently, chatbots will respond to the questions in the output frame and display a facial expression in the top right-hand corner simultaneously. In truth, this user conversational interface imitates the online conversation layout of existing popular Instant Messenger interfaces, such as MSN Messenger and Skype. Moreover, the familiar conversation layout is likely to assist users to easily commence keyboard conversations with the chatbots.
97
Figure 17. The conversational user interface of Gary.
Two tasks were involved in the hardware interface survey with one task being the keyboard conversation with Gary on computers, and the other being with Gavin on portable devices. Additionally, this research adopted keyboard conversation as the user- agent communication manner because human voices are delicate for computers to identify and other environmental sounds may also disturb voice recognition. Users were also asked to converse with chatbots through typing rather than speaking in order to control the environmental variables during the experiment. During the first task, subjects undertook a keyboard conversation with Gary in the consultation room. In the second task, subjects were allowed to carry the iPhone or iPod touch to have keyboard conversations with Gavin around the consultation room. After subjects conversed with each chatbot for at least ten minutes, they were asked to complete a questionnaire in relation to these interactions. Subsequent to completing both tasks of the experiment, subjects were also invited to partake in a face-to-face interview.
Gary and Gavin were designed as representations of giraffes, thereby facilitating subjects to chat with the chatbots about the individual characteristics of the chatbots (such as their birthday, gender and age) as well as answering questions regarding actual giraffes (such as habitat, diet and daily life). Moreover, giraffe forms were specially designed for chatbot representations, and giraffe-related information was gathered for conversation database development. This specific approach was selected given that a domain-specific knowledge base is more appropriate for ECAs design (Goh et al., 2006). In reality, the chatbots have the capacity to express four basic emotional expressions, namely
98 joy/happiness, anger, sadness and fear. These particular emotions have been selected, as they are four of the most common ones appearing on many theorists’ list (Kort, Reilly, & Picard, 2001; Oatley, 2004). In addition, one neutral facial expression was designed to aid the expression of chatbots emotions (emotional expressions of chatbots will be discussed more in the section of Materials).
The chat flow of chatbots is displayed in Figure 18. There are two distinct phases in the chat flow, namely a textual response phase and a facial expression phase. In the textual response phase, two AIML-Bots, the Giraffe-Bot and PI-Bot, were specifically designed for this experiment to answer user questions. In this case, AIML-Bots are the conversation database developed with AIML format. This form allows users to input questions through the input frame of the conversational user interface. In a situation where the question is about giraffe knowledge, Giraffe-Bot will generate a textual response according to the question. Alternatively, PI-Bot will answer questions concerning personal information or generate a random response. After generating the textual response, Emotional Keyword Filter, one facial expression is selected based upon the emotional keywords in the textual response (emotional keywords are listed in the Appendices). Emotional Keyword Filter is a program that triggers chatbots’ facial expression exchange if there was any emotional keyword in the textual response. Finally, both the textual response and the facial expression are presented in the conversational user interface.
Human
input output
ChatBot
Ready reference Yes Giraffe materials Bot Emotional Conversational Keyword User Filter Interface Personal Yes PI Information Bot
Textual Facial Response Expression
Figure 18. The chat flow of chatbots.
99 5.2. Measures
The two experiments conducted in this research were undertaken in order to investigate the optimal form of affective interface design for ECAs. In the hardware interface survey, two main factors of analysis with respect to HCIs were examined in terms of the emotional influences of chatbots on humans and the user-agent relationships. In addition, the similarities and differences between two devices of ECAs were compared, namely between computers and portable devices.
A self-reporting measure to design questionnaires has been adopted in order to analyse human affective factors in this survey. As mentioned in Chapter 4, the self-reporting measure is a method that asks users directly about their affective states, such as design questionnaires to ascertain the emotions, feelings and thoughts of the users. Although questionnaires are only able to analyse the conscious experience of affective states, it is the most direct way to measure the emotions and moods of humans (Brave & Nass, 2003). Furthermore, semi-structured interviews and auto-recorded keyboard conversations were used for further observations in this experiment. Therefore, this experiment adopts a hybrid methodology with both quantitative and qualitative analyses employed in the survey.
As mentioned previously, interactions between users and agents, and users sense of interfaces are mostly in relation to factors, such as preference, usability and efficiency, given these feelings affect the emotions of users. According to Oatley (2004), emotions are caused by important factors, including goals, concerns and aspirations. When applied to this experiment, the emotional influences of subjects generate internal feelings that are closely linked to their goals, thoughts and concerns towards ECAs during HCIs, and they are specifically impacted by agent emotional expressions. In fact, the emotional influences of ECAs on users are one factor that was measured in the hardware interface experiment, the other being the user-agent relationships. As mentioned in Chapter 4, relationships are connected directly to affect, and Kelley (1983) defined that relationship as the interdependent behaviour between people. Similarly, Bickmore (2003, p. 19) classified relationship as “a unique pattern of interdependent behavior”. In this experiment, a relationship refers to the human generated interdependent behaviours displayed towards the ECAs during HCIs.
In order to analyse affective influences of ECAs on humans during the hardware interface experiment, two coding schemes were developed to analyse questionnaire with one for emotional influences and the other one for user-agent relationships. 100 The emotional influences coding consists of one prerequisite and three categories. In truth, emotional appearance is the prerequisite in this coding scheme because it is the first component of four for a computer to possess emotion 5 (Picard, 2002). Furthermore, sentiment, attention and assessment are based on the description of emotion stated by Brave and Nass (2003). The emotional influences coding scheme is shown in Figure 19.
Emotional Influence
Emotional Appearance
Sentiment Attention Assessment
Figure 19. Coding scheme of the emotional influences of chatbots.
Emotional appearance: a prerequisite that identifies whether subjects felt the emotions of ECAs based on facial expressions. In this area, expressions, voices and gestures are the most common techniques utilised to create emotions on computers. In this experiment, facial expressions should give the appearance that agents have emotions.
Sentiment: a category that helps to determine whether subjects relate to ECAs. Brave and Nass (2003) pointed out that although sentiment is not emotion, it is important in HCIs because it motivates users to return to using particular products.
Attention: a category that examines whether subjects focus attention on ECAs when ECAs change facial expressions. Fenske and Raymond (2006) discovered that emotions have notable influences on attention in the human brain, especially in terms of negative emotions.
Assessment: a category which investigates the influences on judgement and decision making, including the willingness of subjects to return to talk with ECAs and their
5 Picard (2002) discovered that “a computer that “has emotions,” in the sense that a person does or will be capable of: 1. Emotional appearance, 2. Multilevel emotion generation, 3. Emotion experience, and 4. Mind-body interactions”.
101 impressions of the content when ECAs show emotions. This is important, as one of the significant influences on emotion is rational decision-making (Picard, 2000).
Additionally, attention and assessment in this scheme are based upon Brave and Nass’s (2003) consequences of affect. Two more elements are employed in the consequences of affect, namely memory and performance. However both consequences require subjects to interact with agents for a prolonged period of time to adequately observe the implications for memory and cognitive performance. In the hardware interface experiment, subjects were required to interact with ECAs for a short period of short conversations to study the immediate consequences of affect, namely attention and assessment.
The relationships coding consists of three categories, namely trustworthiness, likeableness, and closeness. Trustworthiness and likeableness are the source variables of the Elaboration Likelihood Model of Persuasion (ELM) (Petty & Wehener, 1998) while closeness is based on the research of close relationships conducted by Kelley (1983). The relationships coding scheme is illustrated in Figure 20.
Relationship
Trustworthiness Likeableness Closeness
Figure 20. Coding scheme of the user-agent relationships.
Trustworthiness: a category that assists to determine, and to what extent, whether a subject trusts ECAs. Bickmore (2003, p. 22) stated that “trust is one of the most important qualities of any relationship”. Hence, it is evidently clear that trust is a major factor for ECAs to develop relationships with humans.
Likeableness: a category that scrutinises the attractiveness of ECAs to subjects. Petty and Wehener (1998) pointed out that attractiveness has different degrees of impact on subjects when the topics have different levels of relevance.
Closeness: a category that examines the degree of closeness between subjects and ECAs. Kelley (1983, p. 13) stated that “people in close relationships have a great deal of impact
102 on each other”. Therefore, closeness is an indicator to show the depth and breadth of ECAs’ impact on subjects in a relationship.
The above categories were employed as measurements in the questionnaires. Subsequently, face-to-face interviews were conducted after subjects completed the keyboard conversations and questionnaires. The main purpose of the interviews was to gain further insights into the questionnaire results. It is also likely that the questionnaires and interviews would complement each other by ascertaining the causes and effects of affective influences of ECA hardware interfaces on users.
5.3. Materials
In this research, the chatbots and CompanionBots (the learning companion agents developed for the character interface survey) are presented as tools to explore or visualise the influences of different types of agent interfaces on users. These ECAs all utilised Artificial Intelligence Markup Language (AIML) to create the dialogue database. In terms of agent emotion design, the evaluation from the 22 emotions of OCC mode to Ekman, Friesen and Ellsworth’s six basic emotions are offered in this section.
5.3.1. AIML
AIML is an Extensible Markup Language (XML) dialect language, which was developed by Wallace and the Alicebot free software community during the period from 1995 to 2000 (Wallace, 2003). AMIL was formed for the first Alicebot, ALICE, the Artificial Linguistic Internet Computer Entity (ALICE AI Foundation, 2010). Furthermore, all the English conversation patterns of ALICE are stored in AIML format files and users have the ability to communicate with chatbots based on the Alicebot technology with natural language, where input dialogue patterns are translated into conversations.
AIML contains several elements that are consist of units called categories, patterns and templates as follows:
My name is Gary. What is yours? 103
In fact, categories are the fundamental units in AIML, and possess two types of elements, patterns and templates for matching inputs and generating outputs. As shown above, when users ask the input ‘Who are you’, the AIML-Bot will respond ‘My name is Gary. What is yours?’. A pattern in the category is employed to match the user’s input, and a template is arranged to convert to the AIML-Bot answer.
Moreover, an AIML pattern comprises only of words, spaces and the wildcard symbols _ and *. In this case, a pattern like ‘what is your name’ matches only one input. However, a pattern containing a wildcard symbol *, like ‘what is your *’ matches various inputs, such as ‘what is your name’, ‘what is your job’ and ‘what is your favourite food’. A pattern may also include another wildcard symbol _, like ‘_ Andrew’ matches inputs that used the keyword Andrew as the suffix of a sentence. Accordingly, a pattern as ‘Andrew _’ matches to those inputs that used Andrew as the prefix of an input sentence. Finally, a pattern analogous to ‘_ Andrew _’ matches Andrew as an infix of an input sentence. At every node, the wildcard ‘_’ is given the highest priority and a ‘*’matches the lowest priority.
An AIML template identifies a response to a matched pattern. More generally, the template consists of only plain, unmarked text, such as ‘My name is Gary.’ However, templates may also use variables, including ‘My name is
In the hardware interface experiment, AIML was adopted to develop the Giraffe-Bot and PI-Bot conversation database. The Giraffe-Bot contains ready reference materials in relation to giraffe knowledge, including giraffe’s food, habitat, behaviour and other vital information. Here is a simple example:
I am really excited to talk about food. I can eat about 30 kilograms of leaves and twigs daily. Of course, I also eat some fruit and grass as snacks.
104 The PI-Bot consists of personal information in relation to the chatbots Gary and Gavin, such as name, gender, age, birthday and family size. A category like:
My name is Gary, and I have the other twin brother, Gavin, appears on portable devices. How nice to meet you!
5.3.2. Emotional Expressions of Chatbots
The importance of emotions has been mentioned in previous chapters, including several on the roles of emotions in human problem solving, decision-making and cognitive processes. In recent years, the topic of emotions has drawn increasing attention to the area of HCIs. Following the studies of Bates (1994) and Picard (1997), Bates has constructed three emotional creatures in the Oz project and Picard appeals to the concept of affective computing. As a result, it is common to observe human-like emotions appearing on intelligent agents in both academic and commercial domains.
During HCIs, emotions play an essential role in the design of the agent interface (Picard, 1997) as humans interact with computers as though they were social actors (Nass et al., 1994). Furthermore, emotion is an important element to develop the credibility of intelligent agents interacting with humans (Koda & Maes, 1996; Reilly, 1996). According to Affective Computing by Picard (1997), agents need an emotional model to express their emotions. In truth, agents are also likely to need to able to detect the emotional mode of users, although this aspect does not form the focus of this study.
It is widely known that body language, including gestures, facial expressions and body movements, is an essential component of emotional human-computer interactions. This study only addresses emotional facial expressions because facial expressions provide natural and constant feedback regarding the status of the communication to users and therefore plays a critical role in the design of agent interfaces (Picard, 1997; Koda & Maes, 1996). Additionally, it has been demonstrated that agents with social facial expressions, such as smiling, are able to trigger some human brain regions to improve HCIs into human-human interactions (Schilbach, et al., 2006).ġ
105 In terms of the emotional categories of facial expressions, Ortony, Clore and Collins (1990) have established the OCC model as the standard model for emotion synthesis. The OCC model categorises various emotional categories based on positive or negative reactions to events, actions and objects. In fact, 22 emotional categories have been classified in the model and therefore the model offers a sufficient level of complexity and detail to cover most situations an emotional intelligent agent might have to tackle.
As a result of these capabilities, numerous pieces of research have adopted the OCC model to generate emotions for their intelligent agents. Nevertheless, it is problematic for intelligent agents to present 22 different emotions when they interact with humans because agents do not possess the ability to express 22 different emotions clearly and identifiably on their faces. Hence, Ortony (2002) acknowledges that the OCC model might be too complex for an emotional agent. These limitations led to the creation of ten emotional categories that consist of five positive categories (joy, hope, relief, pride, gratitude and love) and five negative categories (distress, fear, disappointment remorse, anger and hate) for developing emotional agents.
Although this shortened categorisation may facilitate emotional expression, ten emotional categories may remain an excessive amount for an agent to convey individual emotion clearly. As a consequence, a significant amount of studies have applied the Ekman, Friesen and Ellsworth’s (1972) six basic emotions to develop their emotional agents. Fridlund, Ekman and Oster (1987) reaffirmed these six basic emotions. The six basic emotions (anger, happiness, fear, surprise, disgust and sadness) can be communicated efficiently and recognised across a number of cultures (Ekman, 1992). These six emotions can be distinguished from other emotions, although fear and surprise are not always capable of being distinguished from each other (Ekman, 1999b). Accordingly, CompanionBots in the character interface experiment adopted these six basic emotions to design their facial expressions. However, chatbots in the hardware interface study took four basic emotional expressions (joy, anger, sadness and fear), which are four most common emotions that appear on leading experts’ list (Kort et al., 2001; Oatley, 2004) for small talk with chatbots.
In terms of the expression design of agents, several studies have been conducted on the facial expression design of agents. Pelachaud and Poggi (1998) investigated the signal part of a performative where the muscular actions of the face are distinguished into action units. Figure 21 shows some expressions of the output performatives, such as ‘imploring’, ‘ordering’ and ‘suggesting’. For instance, in the case of ‘suggesting’, the eyebrows are
106 more or less raised and accompanied by a wide eye opening. Subsequently, Pelachaud and Poggi (2002) also researched nonverbal communication systems, including some expressions shown in Figure 22. For example, the expression of ‘surprised’ is presented via facial expression by raising the eyebrows, enlarging the eyes aperture and an open mouth.
Figure 21. Faces of ‘imploring’, ‘ordering’ and ‘suggesting’ (Pelachaud & Poggi, 1998).
Figure 22. Faces of ‘joy’, ‘surprise’ and ‘mild’ (Pelachaud & Poggi, 2002).
One of the most cited papers of embodied agents is “Agents with Faces: The Effect of Personification” by Koda and Maes (1996). This paper compared the impact of face and facial expressions in an interface with several different graphical representations. In this study, each agent possesses ten facial expressions and Figure 23 shows the various facial expressions of a caricature female face.
107
Figure 23. Ten facial expressions of a Caricature Female (Koda & Maes, 1996).
Above are some examples of facial expressions of embodied agents. They employ certain facial features to express emotions, such as eyelids, eyebrows, lips and ears. An alternate range of sample expressions exhibited by Kismet is shown in Figure 24. Kismet is a sociable humanoid robot, developed by the MIT Media Lab, capable of generating a continuous range of expressions by blending the basis facial postures in real time (Breazeal, 2003).
Figure 24. A range of sample expressions of Kismet (Breazeal, 2003).
It is commonplace to observe that eyelids, eyebrows and lips are employed to express a variety of emotions through embodied agents and robots. In this research, those facial features are also adopted to design various emotions for the chatbots in the hardware interface survey and CompanionBots in the character interface survey respectively. However, graphic styles from particular cultures result in different levels of recognition within cultural participants. Koda (2007) investigated character facial expressions designed for use in Japanese comics and found that the drawing techniques utilised by
108 these artists have a higher level of recognition with Japanese participants than for participants from other Asian countries. The users of the character interface survey will focus on inexperienced English learners, most of who originate from various Asian countries. Consequently, most of these users are familiar with both Eastern and Western cultures.
In reality, both cultures comic drawing techniques were adopted in order to design cross- cultural emotional expressions for ECAs within these two experiments. In fact, “there is agreement across culture in judgements of the expression of emotion in the face, voice, and music” (Keltner & Ekman, 2003, p. 412). However, Figure 25 and Figure 26 portray the distinct comic style expressions that Koda et al. (2009) employed to study cultural differences in character expression interpretations. Each figure shows twelve facial expressions of one character where Figure 25 employs Japanese comic style and Figure 26 utilises American comic style. The expressions for both figures are depicted in the top row from left to right are as follows, happy, sad, approving, disapproving, proud and ashamed as well as in bottom row from left to right, namely grateful, angry, impressed, confused, remorseful and surprised.
Figure 25. Japanese comic style facial expressions (Koda et al., 2009).
Figure 26. American comic style facial expressions (Koda et al., 2009).
This study adopts both Western and Eastern cultural drawing styles to design cross- culture emotional expressions for ECAs, as the majority of users for this research are inexperienced English learners from assorted Asian countries who are receptive to each culture.
109 Table 5 shows the emotional expressions of the chatbots in the hardware interface survey. Each chatbot possesses five facial expressions, including one neutral face and four most common emotional faces, namely joy, anger, sadness and fear.
Table 5. Facial expressions of the chatbots in the hardware interface survey. Neutral Joy Anger Sadness Fear
Gary
Gavin
5.4. Hardware Interface Design of Chatbots
The hardware interface experiment intends to investigate various affective responses of users towards chatbots on computers and portable devices. The increasing popularity of mobile phone usage has resulted in more people paying greater attentions to the Mobile Web. The Mobile Web, under the World Wide Web, refers to ability to access the Internet via mobile devices and is well known as “the platform to access applications and services in the Web through mobile phones used by the under-privileged and the masses” (Talukder, 2007, p. 1).
Moreover, the adoption of mobile devices and the usage of Mobile Web, judged by the number of subscribers, data consumed and page viewed, is increasing daily. In truth, on some websites and applications, mobile users outnumber non-mobile users by two is to one. As an illustration of changing usage patterns, there are in excess of “150 million active users currently accessing Facebook through their mobile devices” and Facebook users “on their mobile devices are twice as active on Facebook than non-mobile users” (Facebook, 2010). Therefore, the Mobile Web not only changes users’ behaviour patterns to access the Internet, it also creates new markets for developers, researchers and individuals to explore.
110 Opera Software is renowned for its Opera family of web browsers involved in promoting Web standards through participation in the W3C6. According to the State of the Mobile Web of Opera in August 2010, unique users, pages viewed and data consumed have all increased substantially. As shown in Figure 27, the data consumed between July and August rose by 9.9% and the data traffic increased 134.4% since August 2009 (Opera Software, 2010). Thus, usage of the Mobile Web has grown appreciably, as access opportunities have improved.
Figure 27. Total data consumed by Opera Mini users per month (Opera Software, 2010).
In terms of the handsets, a leading study undertaken by mobile industry researchers found that Nokia filled 12 of the top 20 phones that accessed mobile networks in August 2009. At the top of the list, Apple iPhone occupied the number-one position, followed by the first Android phone and the Nokia N70 (Gohring, 2009). Another report performed by Opera ascertained that the iPhone was the number one handset in the United States, the United Kingdom and Germany (Opera Software, 2010). In this survey, the hardware interface of portable device was specifically designed for the iPhone or iPod Touch in order to deliver the optimal level of visual resolution. This does not prevent other portable devices equipped with browsers to have the ability to access the portable device chatbot for chatting purposes.
In reality, certain limitations exist with mobile devices when compared to computers. In this case, end-users commonly complain that poor browsers and hardware prevent them from spending more than a moderate amount of time online. Despite these problems, users of iPhones remain the heaviest Mobile Web users on the market since it offers an easy-to-use browser and larger screen than most other handsets. Therefore, Android and
6 W3C: “The World Wide Web Consortium (W3C) is an international community that develops standards to ensure the long-term growth of the Web” (W3C, 2009).
111 Palm as well as other phones and operating system developers have followed some of the approaches and systems devised by Apple for the iPhone (Gohring, 2009). These developments with the iPhone and Android phones have raised Mobile Web usage significantly, as users of these phones have become habitual Mobile Web users when they become accustomed with the operating systems.
It is clearly evident that the Mobile Web will dramatically alter people’s lives in many ways through increased information access and instant communication across media, particularly as mobile Internet access becomes more widespread. As a result of the size and mobility of such devices, it suggests that more people are likely to have access to the Internet than ever before. In this domain, Jon von Tetzchner, the co-founder of Opera Software, stated that he deems “the mobile Web has the power to bring tremendous social and economic change to the inhabitants of many countries” (Opera Software, 2010). Therefore, Mobile Web not only affects personal online habits but may also transform social and economic trends.
Furthermore, some guidelines for developing web sites specifically for mobile devices have been established. First of all, developers should avoid using Flash, JavaScript and AJAX, as these may not be supported on all mobile devices. Furthermore, it is advised to eschew the use of background images, given this may cause contrast issues, and also to abstain from highlighting content in colour, as mobile devices are often used in outdoor conditions where screen contrast may be an issue (Scott, 2010).
In the hardware interface experiment, two experimental versions of chatbots were developed where one version was presented on computers and the other one on portable devices, specifically for the iPod Touch or iPhone. The experiment was undertaken on two types of devices, the first one on a computer (one Apple MacBook Air with 13.3-inch LED screen and one ASUS F8Sa with 14-inch LED screen were used) and the second one on portable devices (two Apple iPod Touches with 3.5-inch widescreen Multi-Touch displays). Moreover, the size of the text and the characters made the demonstration easily readable on each device and no sound accompanied the experiment.
Subjects were instructed to input sentences via computer keyboards and the Multi-Touch screen of the iPod Touch. The interaction scripts included textual responses and non- verbal expressions for chatbots (such as joy and sadness). In addition, chatbots were run on a server under the Windows Server 2003 operating system with Microsoft software MSSQL Server 2005 Express Edition and .Net Framework 2.0 (same as the CompanionBots of the character interface study). Additionally, the keyboard 112 conversations between users and chatbots were automatically recorded in the server. After subjects completed the keyboard conversations with chatbots, a mobile phone (Nokia 5800 XpressMusic) was used to record the interviews. Three types of data were obtained during the experiment for further analysis, namely user-chatbot keyboard conversations, interview records and questionnaires.
This survey aims to study how ECA emotional expressions influence affective factors of users during HCIs. This chapter mostly presents the experimental design of the hardware interface survey, including the measurements and AIML database developments. This survey also attempts to develop an environment similar to a laboratory for experimental purposes, albeit ensuring it is as comfortable and convenient for subjects as possible. Hence, some control variables exist to guarantee a simple experimental procedure as well as providing practical and efficient results. More information about the actual procedure and experimental results of this survey will continue to be addressed in next chapter.
113
Chapter 6. Evaluation of Hardware Interfaces of Chatbots
The purpose of this research is to discover the alterations of human affective factors when users interact with various agent interfaces. In fact, the main issue of this research is to ascertain how the emotional expressions of ECAs affect user emotions. This chapter continues on from the previous chapter, though it focuses on the experiment, actual procedure and evaluation of the hardware interface survey.
This chapter comprises five sections as follows: 1) Procedure, 2) Subjects, 3) Analysis of Questionnaire, 4) Qualitative Results, and 5) Summary. The first section presents the details of the actual procedure, including the duration and location. According to the personal information that was provided by subjects in the questionnaires, the background of subjects was analysed in the second section. The third section analyses of the quantitative results based on the measurements of the coding schemes. The fourth section addresses the qualitative analysis of interview recordings and keyboard conversations, which provides further insights into the quantitative results. The fifth section discusses the causes and effects of affective influences of agent interfaces on users based on both the qualitative and quantitative results.
The other survey, the character interface survey, which studies the affective influences of different character classifications on users, will be undertaken in following two chapters, including the method, materials, procedure and analysis of both the qualitative and quantitative results.
6.1. Procedure
A one-factor-at-two-levels (computer interface - portable device interface) counterbalanced within-subjects design was employed in this experiment. This experiment was performed from the 20th of July 2009 and a week before the start of the experiment period recruitment posters were posted (see Appendices) on several bulletin boards on the Kensington campus of the University of New South Wales (UNSW). The experiment was conducted in a consultation room (G02) of Computer Science Building (K17) of UNSW, and subjects could make appointments for the experiment via e-mail or in person.
115 Firstly, subjects received a briefing that included a short explanation of what embodied conversational agents (ECAs) are and how the experiment works upon attending the consultation room for participation in the experiment. The briefing typically commenced by introducing and explaining ECAs, prior to progressing to address issues, such as computer generated artificial intelligent agents with character forms. Subjects were also informed that ECAs are able to communicate with humans both textually and by spoken language. In this experiment, ECAs were developed as talkative agents, chatbots, which had the potential to be applied to computers and portable devices. Subjects were informed that they would be communicating via keyboard with one chatbot called Gary on laptop computers, and with the other chatbot, named Gavin, on a portable device (iPod Touch). Subjects were also told that they could chat with both chatbots just like they would to a new friend.
During keyboard conversations, the devices were designed to inform subjects that they have completed the phase of experiment after accumulating 3,000 words while conversing with the chatbots. Furthermore, subjects were obliged to complete a questionnaire in relation to both chatbots. After this introduction, subjects were asked to sign consent forms and the actual experiment began.
The goal of this survey was to test the variations in user affective responses towards ECAs on computers and portable devices. Subjects were told that chatbots possessed some emotional facial expressions that react to subjects’ input entries, and they could attempt to observe how many facial expressions they could make both chatbots display. Additionally, chatbots possessed a wide degree of knowledge about giraffes. Thus, subjects could chat with them about personal information of chatbots and basic knowledge of giraffes since two AIML-Bots were specially developed for this survey. The keyboard conversations between subjects and both chatbots were automatically recorded for analysis.
After demonstrating how to chat with Gary on computers, subjects started their keyboard conversations with the first chatbot and were asked to chat with Gary in the consultation room. The average time to accumulate 3000 words during the user-Gary conversations generally took subjects approximately 10-15 minutes to complete. Afterwards, subjects were asked to complete one part of questionnaire about the interactions between themselves and Gary when subjects finished the keyboard conversations with Gary. Figure 28 shows the interaction between one subject and Gary.
116
Figure 28. The interaction between one subject and Gary on a computer.
Similar to the keyboard conversations with Gary, after demonstrating how to chat with Gavin on an iPod Touch, subjects started their keyboard conversations with the second chatbot. The average time to accumulate 3000 words of user-Gavin conversations required longer than the user-Gary conversations, taking roughly 10-20 minutes to complete. Subsequently, subjects were asked to complete another part of questionnaire about the interactions between subjects and Gavin when subjects finished the keyboard conversations with Gavin. Figure 29 shows the interactions between subjects and Gavin. The subjects were permitted to move around the consultation room while chatting with Gavin, but were not authorised to leave the building. The interaction environments for subjects to chat with both chatbots were designed to resemble typical conditions for device usage, such as sitting in front of desk to interact with computers and carrying portable devices around.
Figure 29. The interactions between subjects and Gavin on portable devices.
After subjects finished the keyboard conversations with both ECAs, subjects were asked to answer the other part of questionnaire, in which a few questions concerned the interactions between themselves and both chatbots. Later, subjects partook in an interview, involving 11 semi-structured questions, which were audio recorded. The entire procedure endured for about 60-90 minutes to complete including the time for 117 questionnaires and the interview. When subjects had totally completed the experiment, they were graciously thanked and rewarded with a 2G USB flash drive.
6.2. Subjects
The experiment involved 30 subjects (N=30, aged 15-35), recruited from the Kensington Campus of UNSW, with an experimental environment similar to a lab experiment, and all subjects were required to complete the experimental procedure. According to the subject information provided on the questionnaire, 12 (40%) of the 30 subjects were female and 18 (60%) were male, as shown in Table 6. Ten (33.3%) subjects were aged between 15 and 25 years of age and the other 20 (66.7%) were between 26 and 35 years of age, as illustrated in Table 7.
Table 6. Gender of subjects in the hardware interface survey. Gender Number of Subjects Percentage of Total Sample (%) Male 18 60.0 Female 12 40.0 Total 30 100.0
Table 7. Age groups of subjects in the hardware interface survey. Age groups Number of Subjects Percentage of Total Sample (%) 15-25 10 33.3 26-35 20 66.7 Total 30 100.0
In terms of the usage experience of computers and portable devices, more subjects had more than ten years usage experience of computers (11 subjects, 36.7%) than on portable devices (two subjects, 6.7%). Additionally, 26 subjects (86.7%) had in excess of six years computer usage experience and 19 out of 30 (63.3%) had greater than six years portable device usage experience. The experience of using computer and portable devices among subjects is illustrated in Table 8. The results signify that the majority of subjects in this experiment had more computer usage experience than portable device experience. This information also indicates that subjects would be comfortable to utilise these two devices to process keyboard conversations with chatbots, since all of them had more or less usage experience on computers and portable devices.
118 Table 8. Subjects’ usage experience of computers and portable devices in the hardware interface survey. Number of Percentage of Number of Percentage of Duration Subjects using Total Sample Subjects using Total Sample Computer (%) Portable device (%) 0-2 years 0 0.0 2 6.7 2-6 years 4 13.3 9 30.0 6-10 years 15 50.0 17 56.6 More than 10 years 11 36.7 2 6.7 Total 30 100.0 30 100.0
The results also showed the time dedicated to computer portable device usage daily. In fact, 22 subjects (73.3%) spent two to six hours per day on computers and six subjects (20%) spent six to ten hours daily. On the other hand, there were nine users (30%) who spent two to six hours per day on portable devices and five subjects (17.7%) who spent six to ten hours with portable devices. Ten subjects (33.3%) who spent more than ten hours per day with portable devices might have reasoned that this derives from the convenience and portability of portable devices. However, none of the sample group spent in excess of ten hours working on computers per day. The hours of computer usage and carrying portable devices per day is demonstrated in Table 9 below.
Table 9. The number of hours of computer usage and carrying portable devices per day in the hardware interface survey. Computer Percentage of Percentage of Hours carrying Duration usage Total Sample Total Sample (%) portable device hours (%) 0-2 hours 2 6.7 6 20.0 2-6 hours 22 73.3 9 30.0 6-10 hours 6 20.0 5 17.7 More than 10 hours 0 0.0 10 33.3 Total 30 100.0 30 100.0
All of the 30 subjects had used both computers and portable devices to send text-based messages, for example messages sent to friends via MSN Messenger on computers or text messages via Short Message Service (SMS) on a mobile phone. This information shows that all the subjects had the necessary experience to type messages via these two devices, and they would be likely to have a general idea to undertake keyboard conversations with chatbots via these two devices. Approximately half of them (53%) chat with others through these devices for less than two hours per day, only four people chat with others for more than six hours per day. This information was used to determine whether these subjects are talkative through these devices since they would be asked to chitchat with two strange chatbots via computers and portable devices. In addition, 11 subjects (36.7%) self-reported that they possess average knowledge about AI, ten (33.3%) have a little understanding, and seven (23.3%) have more than average understanding of AI. This 119 information is used to determine whether these subjects are interested in conversing with chatbots. Moreover, subjects were informed that they would talk to computational devices represented by social actors.
Overall, all subjects seemed to have a relatively good usage experience of both computers and portable devices. More subjects had long-term experience (defined as more than ten years) of computers than on portable device, and as a result these subjects should not be inconvenienced to have keyboard conversations with chatbots because they are familiar with these two devices. Furthermore, on average, subjects spend more time on portable devices than on computers every day. Subjects also seemed cognisant of the differences between these two devices in relation to their characteristics of convenience and portability. Additionally, all subjects have used both devices to send message to friends, and this might determine whether subjects were comfortable with communicating with ECAs on computers or portable devices.
6.3. Analysis of Questionnaire
This survey aims to ascertain the answer to the question concerning whether different hardware interfaces of ECAs have distinct affective influences (emotional influences and user-agent relationships) on users. In this experiment, three separate types of data were obtained. The first one was gained via the questionnaire that contained subjects self- reporting their interactions with ECAs on both computers and portable devices. Subjects completed questionnaires as soon as they finished their conversations with both ECAs. The data collected from the questionnaire was utilised to deliver meaningful quantitative experimental result. Other data was gathered from the keyboard conversations between subjects and ECAs along with the information that had been recorded and stored as text files. The final form of data was collected through in-person interviews by means of audio recordings. The latter two types of data were analysed as the qualitative results of this experiment in order to identify interactions between humans and both of the chatbots.
The questionnaire consisted of four parts. Firstly, the personal information of the subjects, involving ten questions about subjects’ basic personal profile, such as gender, age, and experience of computer and portable device usage, was requested. This part of information regarding subject-related background has been documented in the previous section. In its entirety, the second part comprised 20 questions. Ten questions concerned emotional responses of users towards the computer chatbot and the other ten questions
120 addressed the relationships between subjects and the computer chatbot. The third part also contained 20 questions. Ten questions referred to the emotional influences of the portable device chatbot on the subject and the other ten questions were in relation to the relationship of the subject towards the portable device chatbot. In the final part of the questionnaire, ten questions were used to indicate the subject’s impressions of chatbots and to describe the subject’s personality.
6.3.1. Emotional Influences towards Chatbots
The second part of the questionnaire intends to investigate how different hardware interfaces of ECAs elicit varied user emotions. In the emotional influences section of the questionnaire, ten questions were divided into four specific categories of emotions, namely emotional appearance, sentiment, attention and assessment. The measurements of these categories have been described in the Measures subheading of Chapter 5.
The result of the four categories were analysed from the emotional influences part of both chatbots and indicated as Table 10.
Table 10. Wilcoxon signed-rank of the emotional influences of chatbots on subjects. C PD Item Z M SD M SD Emotional influences 3.49 0.675 3.19 0.910 -1.729 C>PD Emotional appearance 3.47 0.823 3.34 1.022 -0.231 C>PD Sentiment 3.34 0.814 3.00 1.127 -2.121* C>PD Attention 3.73 1.172 3.23 1.305 -1.840 C>PD Assessment 3.42 0.788 3.20 0.756 -1.678 C>PD *: p<.05, C=Computer chatbot, PD=Portable device chatbot
The emotional appearance category is the prerequisite of this part. The findings indicated that no significant difference exists between the computer chatbot and the portable device chatbot in this category. According to the mean results of both chatbots, both of them got the mean quantities over the average amount, three. This indicates that subjects sensed the emotions from these two chatbots. While the emotional influences part involved ten questions, only three questions belonged to the sentiment category. Additionally, among these four categories, only sentiment yielded a significant difference (p<.05). This demonstrates that subjects felt more sentiment towards the computer chatbot than the portable device chatbot. Other results in the emotional influences part failed to reflect significant differences. This indicates that subjects did not feel much difference of attention and assessment between these two chatbots. 121 Even though no significant difference was yielded in attention and assessment, there were slight differences between the computer chatbot and the portable device chatbot based on the mean results of these two categories. The computer chatbot gained the higher quantities of the means in these two categories than the portable device chatbot.
The results in this section indicate that subjects feel significantly greater sentiment and slightly more attention and assessment from computer chatbots compared to portable device chatbots. This result also substantiates the above hypothesis that computer chatbots have greater emotional influences on users.
6.3.2. Relationships between Subjects and Chatbots
The third part of the questionnaire aims to investigate how different hardware interfaces of ECAs build distinct levels of relationships with users. In the relationships part of questionnaire, a further twenty questions referred to three categories of relationships between subjects and both of the chatbots in terms of closeness, likeableness and trustworthiness, with ten questions for each chatbot. The above measurements of these three categories have been described in the Measures section of Chapter 5.
The results of the three categories of relationships have been analysed to identify the relationships between subjects and both chatbots and this is depicted in Table 11.
Table 11. Wilcoxon signed-rank of the relationships between subjects and chatbots. C PD Item Z M SD M SD Relationships 3.03 0.620 2.92 0.687 -0.945 C>PD Closeness 2.95 0.619 2.97 0.700 -0.391 C
Among the three categories in the relationships component, only the likeableness category yielded a significant difference (p<.05). This also exhibits that subjects like the computer chatbot more than the portable device chatbot. The other two categories, closeness and trustworthiness, did not indicate noteworthy differences between these two chatbots. This demonstrates that subjects did not experience considerable difference in closeness and trustworthiness towards these two chatbots.
122 However, according to the singular item analysis, in the closeness category, ‘I feel that this ECA is familiar’ and ‘The longer that I am in contact with this ECA, the better our relationship’ yielded significant differences (p<.05). Both of these results signify that the portable device chatbot recorded higher scales than the computer chatbot. In terms of the closeness category, the size of the mean indicates that subjects felt closer with the portable device chatbot than the computer chatbot.
In the trustworthiness category, scores demonstrated that the computer chatbot attained better scores in the size of its means than the portable device chatbot. However, the means of both chatbots were below the average figure, which were three. This indicates that subjects did not particularly trust either chatbots.
Regarding the results in this section, only the closeness category has partially confirmed the hypothesis that portable device chatbots build better relationship with users. However, on average, computer chatbots achieve significantly more likeableness and slightly more trustworthiness among users, and this indicates that computer chatbots build deeper relationships with users than portable device chatbots. Even this result shows that the hypothesis is partially invalid, and the qualitative results are likely to offer some additional insights about the underlying reasons.
6.3.3. Multiple Analyses
In addition to emotional influences and the relationships of both chatbots, gender, age and other factors that might impact affective influences of chatbots on humans were also analysed.
Between different genders and age groups of subjects, some results indicated that they had different affective influences towards both chatbots, however, those results did not achieve any significant differences. This illustrates that these two chatbots did not majorly influence different genders and age groups of subjects. Although, according to the means of all measurements, the computer chatbot accomplished higher scores than the portable device chatbot on average. Additionally, it proved that the computer chatbot has more affective influences on users than the portable device chatbot, regardless of the genders and ages of subjects.
123 In fact, most statistics proved that the computer chatbot has slightly deeper emotional influences on humans and developed better relationships with humans than the portable device chatbot according to the comprehensive survey of other factors, irrespective of the experience of device usage, personality of subjects, and the authenticity and intelligence of chatbots. Even the quantitative results of this experiment have confirmed that the hypothesis that the portable device chatbot is likely to develop better relationships with users than the computer chatbot is incorrect. The analysis of following qualitative data might provide more insights to resolve these dilemmas.
6.4. Qualitative Results
In this experiment, not only questionnaires were collected as the quantitative data, but also keyboard conversations and interview recordings were gathered as the qualitative materials.
6.4.1. Interview Recordings
After subjects interacted with both chatbots and completed their self-reporting questionnaire, they were briefly interviewed about their experiences. The interview encompassed eleven semi-structured questions of which five questions were related to the emotional influences of chatbots on the subject, four concerned the relationships between both of the chatbots and the subject, and the other two questions were regarding further suggestions for the future development of chatbots on different devices.
Emotional Influences of Chatbots on Humans
The first five questions addressed the emotional influences of chatbots on humans. According to the quantitative results in relation to emotional appearance, subjects sensed the emotions of both chatbots during HCIs. The result of the quantitative data was used to correlate with the conclusions of the interview recordings.
In the first three questions, subjects were asked whether they felt chatbots had emotions or not. In fact, a precise and defined relationship exists between facial expressions and
124 emotions (Hanger & Ekman, 1983). If subjects notice the emotional expressions of these ECAs, they might believe that these ECAs possessed emotions. In response to this question, 25 subjects answered that they were able to recognise different facial expressions that appeared on both chatbots. In the fourth question, 17 subjects responded that they felt greater emotions from the computer chatbot because of the larger screens. In truth, subjects stated that the larger screens indicated clearer emotional faces, and as a result they felt greater emotions. This question intends to ascertain which ECA hardware interface had more emotional impact, and more subjects responded that the computer chatbot was the one that elicited this outcome. However, seven subjects felt greater emotions from the portable device chatbot because of some emotional words used by this chatbot. This finding might account for the ECA’s small facial expressions on the portable devices, leading subjects to pay greater attention to the textual responses. The remaining six subjects failed to give much attention to facial expressions for the reason that they focused solely on textual conversations.
In terms of the fifth question, 23 subjects agreed that facial expressions of chatbots were likely to affect human emotions. This information demonstrates that the majority of subjects agreed that the chatbot emotions have an influence on human emotions, regardless of whether the chatbots appear on computers, portable devices or even other devices. When chatbots manifested negative faces, such as anger and sadness, subjects typically would attempt to change the topic of conversation or returned to be assured of the wellbeing of the chatbots and perhaps offer some comfort. Furthermore, three subjects answered that because facial expressions did not always match their answers, or because facial expressions were not centrally located, they only felt limited emotions from the chatbots. The remaining four subjects replied that they did not feel any emotions from the chatbots whatsoever. In this case, the main reasons given were that those emotions were oversimplified and that chatbots did not answer questions correctly. Subjects also suggested that exaggerating gestures to match with facial expressions might enhance the emotional influences of chatbots on humans.
The results of this section can be summarised as reflecting that subjects experienced greater emotions from computer chatbots because the larger screens of computers indicated clearer emotional faces. This verifies the hypothesis that computer chatbots leads to greater emotional influence on users because of their larger screens.
ġ
125 Relationships between Humans and both Chatbots
21 of 30 subjects noticed that the computer chatbot and the portable device chatbot exhibited the same characteristics. This result proves that subjects paid attention to recognise these two chatbots from the character representations, emotional expressions and conversational content. In reality, both chatbots use the same database, albeit displayed slightly different graphic representations. In this regard, 18 subjects preferred the computer chatbot because computers are easy to use and the larger screens have the ability to illustrate facial expressions more clearly. On the contrary, eight subjects stated they preferred the portable device chatbot for the reason they experienced a stronger connection when they chatted with it. This result indicates that the input and output devices of computers as well as the probabilities of portable devices influenced the character preferences.
Moreover, 19 subjects stated that they would like to install chatbots on their computers because computers are easy to input via keyboard and equipped with larger screens. This was a practical question employed to ascertain if ECAs are suitable to install in which device according to users’ opinion. Alternatively, 10 subjects wanted to install chatbots on their portable devices, because portable devices were more transportable for a longer period of time. Despite these findings, at present, the majority of interesting software relating to ECAs is designed for computers rather than for portable devices. Comparing the relationships between subjects and both chatbots, the used device was a major factor that affected the feelings and thoughts of subjects.
In terms of the relationships between subjects and chatbots, 19 subjects deemed that they were friends, though seven defined their relationships with the chatbots as resembling that of strangers. The remaining four subjects judged that their relationships with the chatbots were best described as acquaintances, neither close nor stranger. Eight of the 19 subjects who had friendly relationships with the chatbots believed that they had superior relationships with the computer chatbot than with the portable device chatbot. This information indicates that subjects more or less built relationships with both chatbots.
The results of this section can be summarised as more subjects indicating that computer hardware interfaces were more convenient to build two-way interactions (such as larger screens are able to output clearer chatbot expressions and keyboards could prove the easier way for subjects to input), and for that reason they build better relationships with computer chatbots than portable device chatbots. This result also proves that the hypothesis that postulates that portable device chatbots are able to build better 126 relationships with users is inaccurate. Nevertheless, it is noteworthy that this result is considerably impacted by the device features of modern tools. In fact, portable device chatbots might be able to build closer relationships with users depending on advanced device features, which may exist in the future. For the time being, computer chatbots that optimally utilise the advantages of device features are more likely to be able to have enhanced relationships with users.
Suggestions for Chatbot Developments on Different Devices
Two more questions were asked in order to generate further suggestions for ECA future developments. Firstly, subjects were asked to provide some recommendations for further development of chatbots on different devices in order to better express emotions. This question is used to determine the optimal ECA emotional expressions on various devices. 19 subjects thought chatbots on different devices should offer various ways to express their emotions. For the chatbot on computers, adding sound effects and gestures is likely to make the computer chatbot appear more emotional. Furthermore, exaggerating facial expressions of the computer chatbot is another possible method to enhance emotional expressions. In relation to the chatbot on portable devices, subjects suggested sound effects, gestures and facial expressions may improve the emotional expressions of the portable device chatbot. Therefore, most subjects recommended that vibration would probably be the most effective way for the portable device chatbot to enhance expressions.
When asked for advice on chatbots offering various functions and characters on different devices, 19 subjects suggested that distinct chatbots should have various characters and functions. For example, chatbots on portable devices may only offer simple functions and appear as some warm-hearted characters, such as pets. In the case of the computer chatbots, computers are equipped with superior hardware and additional software, meaning that computer chatbots may develop more powerful functions for users, like an all-purpose assistant. Furthermore, some subjects judged that the functions and character designs of chatbots should depend on the demands of their users. This result indicates that ECAs should convey various functions and designs when they are applied to different devices.
127 6.4.2. Small Talk with Chatbots
In this experiment, subjects were asked to undertake a short conversation with both chatbots on computers and portable devices for about 10 to 20 minutes. They were allowed to ask the chatbots questions about the chatbots themselves and some information in relation to giraffes. Subsequently, the keyboard conversations between subjects and both chatbots were recorded. This data was used to investigate the activity levels of the user-agent conversations, and ascertain further aspects for researchers to consider. This is an example of an exchange, which occurred when one subject chatted with the chatbot on computer:
You say: How old are you
Giraffe: Do you really want to know? Okay, I can tell you this big secret. I am one and a half giraffe years old. About half of your age.
You say: What is your gender
Giraffe: What an embarrassing question! My mommy told me I am a boy.
You say: where are you from
Giraffe: I live in “Long neck wonderland” at i.Zoo.
You say: do u have any friends
Giraffe: I have a bunch of friends. I usually talk to them online. I really enjoy it. You should come chat with me more.
You say: what’s your name
Giraffe: My name is Gary, and I have a twin brother, Gavin, who appears on portable devices. How nice to meet you!
Generally, subjects started the conversations with a greeting, as subjects had been told to attempt to interact with the computational chatbots on the basis of being new friends. Subjects typically followed with some questions about the chatbots themselves, for example their name, age, gender and address, similar to social actions performed in person. This process supports the analyses of Kopp, Gesellensetter, Krämer and Wachsmuth (2005) that majority users would formally greet ECAs and then query ECAs about their implied human-likeness, such as ‘Do you have a girlfriend?’. Subsequent to this, subjects normally attempted to further acquaint themselves with these chatbots, and asked questions about the chatbots themselves as well as interacting with them based on the reactions of the chatbots. In truth, several features can be derived from the small talk between subjects and chatbots.
128
Feeling Compassion for Chatbots
During the experiment, each chatbot’s emotional expression appeared at least twice during every keyboard conversation with subjects. Subjects seemed extremely concerned about the feelings of chatbots when they first began conversing with the chatbots. When chatbots displayed happy smiling faces, some subjects praised them for their lovely appearance. On the other hand, when chatbots displayed sad faces, subjects felt sympathy for them. This situation was especially valid when chatbots mentioned the poor survival rate of giraffes and wore a crying face, and as a result most subjects showed their compassion and attempted to comfort the chatbots:
Giraffe: It is sad to know that only 25 to 50% of giraffe babies reach adulthood.
You say: Oh that is so sad
Giraffe: Yeah, that’s sad.
You say: May be we can find a way to rescue them
Giraffe: It is sad to know that only 25 to 50% of giraffe babies reach adulthood.
You say: what a pity you suffer such lives!
Chatbots in this experiment were not designed to be sufficiently intelligent to answer all the questions correctly. In this case, only two simple AIML-Bots were developed for this experiment, and they were designed to encourage subjects to pay more attentions to the chatbot’s facial expressions regardless of the intelligence level of the chatbots. In fact, it is common practice to design chatbots to behave in a juvenile manner to encourage likeability and display sad faces to gain compassion. Indeed, some subjects showed sympathy with chatbots when chatbots asked for consideration and wore a sad face.
Giraffe: You know I like you, do not make things difficult for me. I can run extremely fast, up to 55 km/h.
You say: Ok, I would say something easily.
The above responses demonstrated that subjects interacted with chatbots in a similar fashion as in interactions with real social actors as proved by Reeves and Nass (1996).
129 Benevolent Responses
During the interaction between chatbots and subjects, subjects not only showed compassion for the chatbots but also gave positive feedback to the chatbots when chatbots indicated benign intentions to them. When chatbots told subjects that they enjoyed their company and were glad to talk to them, subjects generally replied with positive responses.
Giraffe: You are welcome. I really enjoy talking to you.
You say: Im glad to talk to you
You say: What’s your birthday?
Giraffe: How nice you are! You want to give me a birthday present! My birthday is on 01/02/03 (i.zoo calendar), actually it is today. You can give the gift to the researchers at the scene. Love you!
You say: Love you too
In real social environments, people typically provide polite feedbacks when someone displays his/her benevolence. Hoffmann, Krämer, Lam-chi and Kopp (2009) found out that users were more polite to give a better evaluation when they were questioned by the ECA itself compared to a paper-and-pencil questionnaire. Likewise, even subjects that were not satisfied with the chatbots’ answers still replied positively. This process supports the result of Kopp et al. (2005) that users employ polite and well-mannered conversational strategies towards ECAs during small talk.
Giraffe: You know I like you, do not make things difficult for me. I can run extremely fast, up to 55 km/h.
You say: I like you very much but could you answer my question.
As mentioned above, chatbots were designed to be juvenile in manner and not always provide appropriate responses. In most circumstances, subjects remained polite and replied respectfully even if the chatbots gave atypical responses.
Other Observations
Subjects seemed to be curious when they interacted with both chatbots, and most of them were chatting with AI robots for the first time. Subjects typically patiently chatted with both chatbots in the beginning, although they often got bored with their asinine answers.
130 However, this survey aims to investigate the affective influences of different agent interfaces on humans.
In addition to the data used to investigate the active level of HCIs, some other interesting reactions from subjects towards ECAs are worth considering. These reactions also demonstrated that humans treated both chatbots like conventional social actors. Additionally, subjects treated chatbots as friends in order to share their worries. The following script is one conversation between an international student and the computer chatbot.
Giraffe: You know I like you, do not make things difficult for me. I can run extremely fast, up to 55 km/h.
You say: can i be your friends, you know i have a lots of friends in my own country, but know i am lonely
Sometimes, when chatbots revealed their excellence, subjects flattered chatbots in a similar manner as if conducted in a real social setting.
Giraffe: I have no answer for that. But, it is great to know that newborn giraffes are about 1.8 metres tall.
You say: you make me feel like so wanting to be a giraffe
The subject-chatbot interactions were similar to normal interpersonal interactions, given that individuals generally attempt to be polite to others when they are not familiar with them. In reality, the majority of subjects were also courteous to chatbots in the beginning. However, not all the subjects agreed with chatbots all the time during the conversation. After subjects conversed with chatbots for a period of time, subjects felt chatbots were not as strange as when they first met and they began to make some sarcastic remarks to the chatbots.
Giraffe: I hate lions. They always hunt us. But I can defend myself with a powerful kick. It is sad to know that only 25 to 50% of giraffe babies reach adulthood.
You say: Yes it’s really sad but you know lions have no choice. They want to survive too.
Generally, subjects chatted with chatbots as they interacted with other humans, given people normally comfort others as well as make jokes about others during interpersonal interactions.
131
6.5. Summary
The hardware interface experiment has been conducted to ascertain whether different hardware interfaces of ECAs elicit varying levels of user affective influences. This precise result has been discovered, as this research hypothesised that different hardware interfaces of ECAs do influence users’ affective states differently, and this result has been mostly impacted by the actual features of hardware interfaces. In other words, the actual input and output devices utilised in the study had a major impact on users’ affective states.ġ
According to the analysed results, when chatbots showed emotional faces, subjects did experience more or less emotional feelings towards the chatbots. Chatbots in this experiment were only equipped with four emotional expressions and one neutral face. Moreover, subjects not only sensed the emotions of chatbots, but also agreed that the chatbot emotions affect human feelings during HCIs. While it is true that humans experienced various emotional influences from chatbots on different devices, this mainly derived from the device’s features, especially the screen sizes. Subjects supposed that the larger screens allowed chatbots a greater range of emotional responses and expressions, and subjects experienced greater emotional influence towards the computer chatbot employed in this study. In addition, the chatbots’ intelligence occasionally negatively affected the interactions between subjects and the chatbots. Subjects also felt annoyed when chatbots gave incorrect answers, but soon they forgave them soon because of their positive emotional faces.
Regarding the relationships between subjects and the chatbots, most subjects felt closer to the portable device chatbot because of the portability of portable device. However, users judged that it was easier to use computers to communicate with the chatbot than portable devices. In fact, subjects did develop different relationships with both chatbots, as on average, subjects developed better relationships with the computer chatbot than the portable device chatbot. The first stated reason is that they had different feelings regarding the interactions with chatbots, for example some subjects experienced positive feelings when the computer chatbot manifested a wide smile face for no apparent reason. The second reason is the nature of chatbots, such as intelligence and lifelikeness of the character. Nevertheless, subjects intermittently felt annoyed when chatbots performed incorrectly. The final, albeitthemost important, reason is likely to be the features of the devices. Chatbots on different devices brought varied feelings to subjects and developed distinct relationships with subjects, mostly because of the screen sizes and the portability 132 of the unit. In mitigation of some of the results, this experiment was conducted in 2009, and the experimental findings only reflected the device features of modern tools. In truth, developments of computational devices continue to progress, and as a consequence the experimental results that were impacted by the device features might change depending on such advancements. In any event, the ECA interfaces on various devices do form bonds with users with the changing times.
In terms of the future development of chatbots, the majority of subjects suggested that ECAs on different devices should be equipped with various systematic functions and visual designs based on the device features and user needs. Moreover, other findings, such as subjects suggesting a greater number of characters and more exaggerated facial expressions for the ECAs, as well as the social interactions between users and ECAs, are suitable for language learning and these aspects will be applied to later ECA developments.
This chapter follows the former chapter to address the details of the hardware interface survey. In this chapter, the details, including the duration, location and actual procedure of the experiment were offered. Moreover, the subjects were analysed with regard to their ages, genders, and usages of computers and portable devices. Additionally, the quantitative and qualitative data collected during the experiment was examined and the results of the survey were provided in order to prove that agent hardware interfaces have affective influences on users. Nonetheless, this result creates another dilemma in agent interface research, namely whether the character classifications of agents have similar effects on user affective feeling alterations. The following two chapters will describe another experiment, the character interface survey, in order to ascertain the answer to the above dilemma, including experimental design, measurements and results.
133
Chapter 7. Method of Character Interfaces of CompanionBots
This research aims to study the affective impact of agent interfaces on user feelings during HCIs. Having presented the method and evaluated the hardware interface survey in Chapters 5 and 6, this section commences the introduction of the character interface survey in Chapters 7 and 8. The character interface survey intends to explore how learners respond differently towards distinct character classifications of ECAs. One experiment has been conducted to ascertain the answers to the research questions concerning when beginner English learners participate in language practice tasks, do they generate different levels of affective factors (character preference, user engagement and user-agent relationships) towards distinct character classifications of ECAs (humans, animals and creative creatures). Moreover, it intends to establish the particular relationship among these different affective factors during HCIs. This chapter investigates the affective factors of humans toward ECAs presented by various classifications of characters and outlines the methodology of the experiment.
This chapter comprises four sections as follows: 1) Method, 2) Measures, 3) Materials, and 4) Design and Classifications of CompanionBots. The first section presents the methodology and experimental design of the character interface survey, including the conversational user interface design, chat flow and character design. The second section illustrates the measurements that are adopted to analyse affective responses of users, especially user engagement, character preference and user-agent relationships. The third section concerns the International English Language Testing System (IELTS) speaking AIML database, IELTS-Bot, built specifically for the character interface survey, and adopts the emotion expression evaluation of Chapter 5 to design facial expressions for the CompanionBots. The major issue this survey intends to investigate is the character classifications of agent interfaces. The fourth section undertakes a literature review in relation to the classifications of agent characters and provides some guidelines for design companion agents.
This chapter principally addresses experimental design and CompanionBot developments. The evaluation of the character interface survey including the actual procedure and result analyses will be addressed in the next chapter.
135 7.1. Method
The survey design of the character interface survey is closely analogous to the hardware interface survey. Additionally, experimentation is also adopted in this survey as a means of investigating which character classification of ECA interface has the ability to improve user affective factors in learning tasks as well as the underlying associations between character preference, user engagement and user-agent relationships. In this case, the independent variable is the character classification, and its attributions are human, animal and creative creature. This survey intends to compare the affective influences between different ECA character classifications. In order to achieve this object, subjects were asked to interact with different character classifications of CompanionBots and compare them. A one-factor-at-three-levels (a human form – an animal form – a creative creature form) counterbalanced by within-subjects design is employed in this study. In reality, the dependent variables encompassed the degree of user engagement, user-agent relationships, character preference, learning attitudes and empathy, given that human affective factors are the variables this survey attempts to study. In addition, control variables include the emotional expression amount of CompanionBots, conversational user interfaces, the conversational database and adopting keyboard conversation as the communication manner. In this survey, the main purpose is to ascertain the different influences among various character classifications of ECAs on users. Language practice is a domain designed as the conversational content to attract subjects to have conversations with the CompanionBots. Subsequently, subjects were asked to type conversations in order to practice English with three learning companion agents (called CompanionBots in this research) in Table 12. The three CompanionBot are as follows, one agent named Andrew was designed with a human form, another named Ben was conceived with an animal form, and the other known as Chris was devised with a creative creature form. In the course of the research, each CompanionBot accompanied subjects through online English conversations for at least five days during the character interface experiment.
Table 12. ECAs as learning companion agents (CompanionBots) in the character interface survey. Name Andrew Ben Chris Character interface
Character Human Animal Creative creature classification
136 As indicated in Figure 30, all conversational user interfaces employed the character interface experiment are exceptionally similar, with the differences between them limited to the actual characters and names. In truth, each character has the potential to exhibit facial expressions, and the interface includes an introduction, an output frame, an input frame and a submit button, which is similar to the conversational user interfaces utilised in the hardware interface survey. The conversational user interface entails subjects typing sentences in the input frame, and then receiving responses in the output frame while facial expressions are displayed in the top right-hand corner.
Figure 30. Conversational user interfaces of learning companion agents.
As illustrated in Figure 31, 18 characters, grouped into three categories, with each category containing six characters were designed for this experiment. In the beginning of the character interface experiment, subjects were asked to select five characters from the 18 available characters to be their ideal language learning companion agents and rank them in order of preference. Following this, subjects were asked to chat with each CompanionBot for at least five minutes per day for a period of five days. Afterwards, they were asked to complete one aspect of the questionnaire in relation to the interaction with the CompanionBot subsequent to completing chatting with each CompanionBot. Later, subjects participated in a brief interview to discuss their experiences during the experiment. This was concluded after subjects finished chatting with all three CompanionBots and had completed the corresponding questionnaires.
137
Figure 31. 18 characters in the character interface survey.
Besides the PI-Bot, which similar to the hardware interface survey includes personal information about ECAs, an IELTS-Bot was specially developed for the CompanionBots to encourage subjects to interact with them during the experimentation phase. The IELTS-Bot includes conversational patterns in relation to the speaking part of IELTS exams. Therefore, subjects could not only practice English conversations but also partake in the IELTS speaking part online with the CompanionBots. Six basic emotional expressions (happiness, anger, surprise, disgust, sadness and fear) as well as one neutral facial expression were designed for the CompanionBots to depict their emotions during the conversations.
The chat flow of CompanionBots is depicted in Figure 32. This aspect functions similar to the chat flow of chatbots, whereby in the textual response phase subjects input sentences in the conversational user interface. In fact, the IELTS-Bot will generate a response according to the sentence that subjects input if the sentence is related to the knowledge of IELTS speaking test. Alternatively, PI-Bot will answer the question if the sentence concerns personal information about the CompanionBots. In all other circumstances, the AAA-Bots will answer other general questions or generate a random response. In truth, the PI-Bot and IELTS-Bot employed in the character interface survey 138 are analogous to the two AIML-Bots, namely PI-Bot and Giraffe-Bot, utilised in the hardware interface survey. Similarly, the PI-Bot administers all personal information about ECAs, and the Giraffe-Bot and IELTS-Bot are responsible for the specific domain knowledge. In addition, the AAA-Bots in the character interface survey is a revised version of the free ALICE source (AAA-Bots will be addressed more in Chapter 9) that has been adopted to generate a textual response excluding personal information and IELTS specific knowledge. Subsequent to generating a textual response, the ‘Emotional Keyword Filter’ selects one facial expression according to the textual response. Finally, both the textual response and facial expression are displayed in the conversational user interface in order to elicit further responses from users.
Figure 32. The chat flow of the CompanionBots.
7.2. Measures
As discussed in Chapter 4, self-reporting is the most widely adopted form of measurement applied in agent interface related research. This experiment has also adopted this manner to design the questionnaire and semi-structured interview for investigating human affective feeling alterations. Therefore, this experiment is a hybrid method, and both quantitative and qualitative analyses are embraced within this survey.
139 This character interface experiment aims to investigate which character classification of agent interface has the potential to improve affective factors. In addition, this survey intends to explore the relationship between character preference, user engagement and user-agent relationships. Therefore, user engagement, character preference and user- agents relationships are one of the core evaluations in the character interface survey.
A questionnaire is deployed to quantify character preference, user engagement, user- agent relationships, learning attitude and empathy for characters. In reality, user engagement measurements from the I-PEFiC framework, adjusted by van Vugt and his colleagues (van Vugt et al., 2004; van Vugt et al., 2006) from the PEFiC framework (Konijn & Hoorn, 2005), have been implemented in this research. Additionally, user engagement is an index composed of seven items that were grouped into two tendencies, namely involvement and distance, where the involvement tendency comprises three items – including ‘this CompanionBot appealed to me’, ‘this CompanionBot gave me a good feeling’, and ‘this CompanionBot attracted me’. This aspect is measured by a single item on the five-point Likert scale. Furthermore, evaluation of the distance tendency consists of four items – ‘this CompanionBot left me with cold feelings’, ‘I felt negative toward this CompanionBot’, ‘I felt distance between this CompanionBot with me’, and ‘I disliked this CompanionBot’. These aspects are measured in broadly similar manner.
In terms of character preference, two types of character preferences were measured, one prior to subjects interacting with agents and the other after subjects had chatted with the agents for at least five days. In fact, Koda and Maes (1996) also investigated character preferences before and after interaction in order to ascertain which type of character is more popular before interaction as well as which classification of character maintains the character preferences of users. In the course of this experiment, subjects were asked to choose five characters from 18 characters and rank them in order of preference prior to subject interaction with the CompanionBots. In this way, the character preferences of each character classification and CompanionBot were measured by the total score amount. For example, the most preferred character selected by a subject gains five points for itself and its category, the second most preferred is granted four points for itself and its category, and so on, with the fifth preference receiving one point. In truth, character preferences before interaction were measured on the basis of the total amount of points. However, the character preference material obtained after interaction was measured by an index composed of two items on the five-point Likert scales – ‘I like this CompanionBot’ and ‘I would like to talk with this CompanionBot more’.
140 This study has adopted the relationship assessment scale (RAS) of Hendrick (1988) to measure the relationships between agents and users. Additionally, evaluation of the relationship between user and agent is determined by an index composed of seven items on five-point Likert scales. The items on the RAS encompass several relationship dimensions, such as love, problems and expectations. As a result, the RAS has been used to assess the relationships of married couples, dating couples, and even with minor changes in the case of platonic friendships. In this experiment, the RAS has been adopted with minimal changes in order to measure the generic relationships between users and agents.
The character interface experiment is primarily a language-learning task. Although language learning is only a cause to attract users to interact with agents, the learning attitudes of users are also measured during this experiment. In fact, learning attitudes of users may provide evidence to explain the influences of agents on users. Moreover, the evaluations of learning attitudes – including positively and negatively worded items, integrative and instrumental orientation and English anxiety – have been adopted partly from ‘The Attitude/Motivation Test Battery Manual (AMTB)’ (Gardner, 2004) and measured by single items on five-point Likert scales.
Empathy is an index, which is composed of two items, namely sorrow and anger on the part of the agent. In this domain, Hall et al. (2004) used sorrow and anger expressions to measure the empathic responses of users to characters. Therefore, the feeling of sorrow and anger by the character is measured as a single item on various five-point Likert scales.
Table 13 provides the summary definitions of the psychological constructs that adopted in this experiment.
Table 13. Summary definitions of the psychological constructs of the character survey Item Definition User User engagement involves two tendencies namely involvement and engagement distance. Involvement referred to positive feelings of users that toward to the ECAs during the interactions, and distance referred to negative feelings. Character Character preference prior to interactions is measured by the ranking preference of likeableness.
Character preference after interactions referred to the likeableness of ECAs and the intention of users to interact more.
141 Relationship Relationship encompasses several dimensions like love, problem and expectations. Learning The evaluations of learning attitudes include positively and negatively attitudes worded items, integrative and instrumental orientation as well as language anxiety. Empathy Empathy composes sorrow and anger on the part of the agent.
Additionally, a semi-structured interview approach is employed to investigate user feelings toward their preferred agent characters and user expectations concerning their ideal learning companion agents by means of pertinent questions. Furthermore, data gleaned from auto-recorded keyboard conversations between subjects was used to observe the interactions between users and agents. These two types of qualitative data were used to ascertain clear the insights into the results.
7.3. Materials
AIML and emotional design of agents are presented here as materials similar to the materials employed in the hardware interface experiment.
7.3.1. AIML
Two AIML-Bots have been especially developed for the purposes of the character interface experiment and conversation database, namely IELTS-Bot and PI-Bot, and AAA-Bots (AAA Files) respectively. In fact, the IELTS-Bot has been specifically designed for the subjects of the character interface experiment to practice the speaking section of IELTS. More details of IELTS-Bot will be listed in the Appendices. Here follows a simplified example of the IELTS-Bot:
The PI-Bot of the character interface experiment contains certain personal information about Andrew, Ben and Chris similar to the PI-Bot used in the hardware interface experiment. In addition, AAA-Bots is the ALICE AIML Bots that is adopted as the conversational materials. In this way, when subjects input any question that is not included in the content of the PI-Bot or IELTS-Bot, AAA-Bots will probably be able to answer the question. This makes the CompanionBots more successful in their ability to converse, and allow subjects to effectively practice English by means of online communication.
7.3.2. Emotional Expressions of CompanionBots
As mentioned in Chapter 5, a significant number of studies have applied Ekman, Friesen and Ellsworth’s (1972) six basic emotions to develop their emotional agents. The six basic emotions (anger, happiness, fear, surprise, disgust and sadness) can be communicated efficiently and recognised across a number of different cultures (Ekman, 1992). In addition, agents who present appropriate expressions provide students with encouragement in undertaking their learning tasks (Elliott, Rickel, & Lester, 1997). This character interface survey utilises the same categories as a point of departure to take advantage of this well-established theoretical framework.
The facial expression design of CompanionBots has adopted these guidelines, as alluded to in Chapter 5. Table 14 indicates the facial expressions of CompanionBots, the human classification (Andrew), the animal classification (Ben) and the creative creature classification (Chris) in the character interface study. Each CompanionBots possesses seven different facial expressions, including one neutral face and six basic emotional faces exhibiting anger, happiness, fear, surprise, disgust and sadness.
143 Table 14. Emotional expressions of the CompanionBots in the character interface survey. Neutral Anger Happiness Fear Surprise Disgust Sadness
Andrew (Human)
Ben (Animal)
Chris (Creative creature)
In truth, emotional expressions of ECAs form a crucial aspect of this research. In the hardware interface survey, ECA expressions are utilised to investigate the immediate emotional influences after the short-term interactions. Alternatively, in the character interface survey, ECA expressions are employed to assist to ascertain the affective relation developments after the long-term interactions.
7.4. Design and Classification of CompanionBots
Designers typically are likely to consider the domain of the character in terms of applying the character design as well as aspects including gender, ethnicity, and style for each character. In the area of agent character design, several studies exist relating to the aesthetics (van Vugt et al., 2006), anthropomorphism (Nowak & Rauh, 2005), gender, ethnicity and realism (Baylor, 2005) of characters. However, a paucity of studies has explored issues concerning character preferences for different classifications of agent characters. This survey intends to investigate user engagement, character preferences and user-agent relationships for various classifications of agent characters. Moreover, this endeavour may make it possible to design more highly preferred agent interfaces that may improve user engagement and encourage a stronger user-agent relationship in order to generate an optimal experience during HCIs.
Among varying character classifications, ‘human’ and ‘animal’ are two of the most common distinctions recognised by multifarious researchers (Erickson, 1997; Szarowicz, Francik, Mittmann, & Remagnino, 2005; van Vugt et al., 2006; van Vugt et al., 2007). However, alternative classifications include ‘object’ (Nowak & Rauh, 2005; Baylor, 2005; Koda et al., 2009), ‘robot’ (Piza, Zuniga, & Ramos, 2004), ‘fantasy creature’ (van Vugt et al., 2004; Beun et al., 2003), ‘plant’ (Koda et al., 2009) and ‘other’ (Zimmerman 144 et al., 2005). In this survey, agent characters have been grouped into three particular classifications, namely ‘human’, ‘animal’ and ‘creative creature’. Any variation of a human-like character is classified as human, including characters depicted as children, Caucasian persons and ladies. In reality, all animal-like characters, whether avian, mammalian, aquatic or reptilian, have been grouped into the animal classification. Other characters, which resemble plants, robots, aliens or others descriptions are grouped into the creative creature classification.
The character preference investigation involved 18 specially designed characters prior to any interactions. These 18 characters have been categorised into three classifications with each one containing six characters. In fact, various types of characters were designed in each classification in order to better diversify the characters. As shown in Figure 33, in the human classification, distinct age categories and genders of characters are employed, such as a boy, a girl, a gentleman, a lady, an older gentleman and an older lady. Moreover, a giraffe, a monkey, a rabbit, a rooster, a penguin and a turtle are included in the animal classification (as illustrated in Figure 34). In the creative creature classification, a plant- like character, a robot, an angle, a devil, a solar-image character and a monster are presented as Figure 35.
Figure 33. The six characters belonging to the human classification.
145
Figure 34. The six characters belonging to the animal classification.
Figure 35. The six characters belonging to the creative creature classification.
At the beginning of the experiment, subjects had to choose their five favourite characters from these 18 characters and rank them in order of preference prior to any further interaction with the CompanionBots. Subsequently, during the actual experiment, subjects were obliged to have keyboard conversations with three selected characters. Therefore, three characters were selected from three classifications (as shown in Table 15). A little boy called Andrew represents the human classification, a giraffe known as Ben signifies the animal classification, and a plant-like character named Chris denotes the creative creature classification.
Table 15. Representations of the three selected characters from three character classifications.
Character
Name Andrew Ben Chris Description A boy A giraffe A plant-like character Classification Human Animal Creative creature
146 In addition to the classifications of characters, several aspects should be considered in order to design the characters for this experiment. In this experiment, the ECAs are learning companion agents that accompany inexperienced English learners to practice conversations online. Castellano et al. (2009, p. 1) suggested that “robot companions must be able to display social, affective behaviour”. In their study, robot companions were not only able to express affective expressions, but were also endowed with affect recognition abilities. In reality, social and affective behaviours confer greater humanity on companion agents, and as a result users tend to treat the companion agents as real human beings during such social interactions. In addition, Baylor and Kim (2009) suggested that a pedagogical agent that employs non-verbal communicative behaviour is likely to be the most appropriate for a positive learning outcome. Agent non-verbal communicative behaviours typically involve deictic gestures and facial expressions. Baylor and Kim (2008) propounded a study, which has indicated that students learned more when agents offer facial expressions, but deictic gestures in attitudinal learning. However, students learned more when agents with deictic gestures, but facial expressions in procedural learning. In the character interface experiment, the learning task is more productive than attitudinal learning. Additionally, the use of gestures may sometimes be counterproductive in recognising facial expressions of agents (Koda, 2007). Therefore, only facial expressions were applied to the CompanionBots as a means of social behaviour.
Nevertheless, problems remain about how to design the facial expressions of agents. Some theories in relation to the various categories of agent emotional expressions were discussed in Chapter 5, and these include exaggeration and animation as the significant methods to express vivid emotions of agents. In fact, exaggerated expressions of characters are necessary to elicit powerful emotional responses from audiences (Gard, 2000). In general, emotional expressions of agents in the character interface survey were designed with exaggerated features. For instance, Andrew has been designed with six exaggerated expressions in order to aid interaction with subjects (see Figure 36). These emotions moving from left to right are as follows: anger, surprise, sadness, happiness, fear and disgust.
147
Figure 36. Exaggerated expressions of Andrew.
In addition to the exaggerated facial expressions of agents, Baylor and Ryu (2003) discovered that animation is beneficial for pedagogical agents. Lee and Nass (2002) demonstrated that animated agents are more attractive and trustworthy than stick images and text boxes. For this reason, character expressions are normally designed for simple animations with Graphics Interchange Format (GIF) files. As shown in Figure 37, five static images have been assembled into the fear expression animation of Andrew. Additional static images of facial expression animations are also available to observe in the Appendices.
Figure 37. Static images of fear animation of Andrew.
This chapter presents the method design of the second survey for this research, namely the character interface survey, which differs from the hardware interface survey in terms of investigating the long-term interactions. Firstly, the details of the experimental method employed in this study are introduced, such as the design of characters, experiment and conversational user interface. Secondly, the measurement framework of each category of the questionnaire is addressed through theoretical references. Thirdly, a brief description of AIML applications and character expression design are elucidated. Finally, a literature review of the classifications of character design is undertaken. This chapter principally describes the experimental design of the character interface survey. The next chapter will expand upon the content of this chapter as well as addressing the actual procedure and offering more comprehensive analyses of this experiment.
148 Chapter 8. Evaluation of Character Interfaces of CompanionBots
The intention of this research is to explore the affective interactions that occur between users and assorted agent interfaces. The character interface survey, conducted in this research, is specially designed to discover the affective influences of distinct ECA character classifications on users. This chapter is a continuation of the previous chapter through focusing on the actual procedure and evaluation of the character interface survey.
This chapter comprises five sections: 1) Procedure, 2) Subjects, 3) Analysis of Questionnaire, 4) Qualitative Results, and 5) Summary. The first section presents the entire process of the character interface experiment, including the location and duration. The second section encompasses the personal information of subjects, such as the gender, age and usage experience of computers, based on the background part in the questionnaire. In the third section, character preference before interaction and the other seven affective responses of users towards CompanionBots are offered quantitatively. The fourth section covers the qualitative results, which are analysed from the collected interview recordings, thereby providing an insight into the quantitative results. The fifth section examines the causes and effects from both the quantitative and qualitative data and this leads to several results being reached from this survey.
This chapter finalises the experimental part of the study. The practical part follows and this is contained in Chapters 9 and 10, where the practical use of adopting existing and experimental results on a multi-agent website is discussed and other contributions and conclusions of this research are granted.
8.1. Procedure
A one-factor-at-three-levels (a human classification – an animal classification – a creative creature classification) counterbalancing within-subjects design was employed in this experiment. This experiment commenced on the 17th of April 2010. As the target focus group centred on inexperienced English learners, this experiment was conducted in an IELTS training school, Global Language School, located at 201, 683 - 689 George Street, in Haymarket, New South Wales, Australia. Most students at the Global Language School are learning English as their second language and intend to participate in more
149 IELTS programs in the near future for migration or further education purposes. For this study, subjects were randomly recruited from this school.
Throughout the recruitment procedure, the experiment was briefly introduced to possible subjects in order to allow them to decide as to whether they wished to partake in this experiment or not. Once subjects had confirmed their participation in this experiment, they were asked not to discuss the content during the experiment duration to avoid affecting other subjects. The experiment was likely to continue for at least three weeks to be entirely completed. Prior to the commencement of the actual experiment, subjects were clearly introduced to the experiment process and content.
In addition, subjects were also introduced to the concept of ECAs as a type of computer generated AI with character forms. Moreover, subjects were informed that ECAs had the capacity to communicate with humans via both text and spoken language. Learning companion agents (called CompanionBots in this experiment) are one form of ECAs and were designed for users in order to assist English learning. Furthermore, subjects have the ability to chat with CompanionBots anytime and anywhere online. In this experiment, CompanionBots were developed with an IELTS speaking database specifically to assist subjects practice for their IELTS speaking test component. The study obliged subjects to chat with three CompanionBots (Andrew with the human form, Ben with the animal form and Chris with the creative creature form) with whom they had the opportunity to practice English conversations. Subjects had to communicate with each CompanionBot textually online for at least five days. Finally, after subjects had become accustomed to the workings of this experiment, they were asked to sign the consent forms and the actual experiment began.
Questionnaire design involved the creation of four separate parts. The first part asked subjects about their background and character preferences prior to the interaction, and the remaining questions concerned the interactions between subjects and the three CompanionBots in each part. In the first part, subjects were invited to add some basic personal information, such as age, gender, and previous usage experiences of computers and portable devices. This experiment intends to investigate the character preferences of users towards three classifications of agent characters. Therefore, subjects chose five preferred options for characters to act as their English learning CompanionBots (from 18 characters), and then ranked them before any interactions occurred with the CompanionBots. During the experimental period, subjects were instructed to have keyboard conversations with three CompanionBots online in any location. Additionally,
150 subjects were obliged to chat with each of them for at least five minutes per day over a period of five days. The order of conversations with the CompanionBots was randomly selected, for example subject A chatted with Ben in the first week, Andrew in the second week and then Chris in the third week, however subject B chatted with Chris first followed by Ben and Andrew.
The procedure of keyboard conversations undertaken with the three CompanionBots was as follows: subjects received an email with an URL to the first CompanionBot on the day after recruitment. Subsequently, subjects began to text-chat with the first CompanionBot, and after five days of chatting with the first CompanionBot, subjects received one part of the questionnaire in relation to the first CompanionBot along with another URL to facilitate connection to the second CompanionBot. Similarly, after another five days of chatting with the second CompanionBot, subjects received another email containing another part of the questionnaire and the pertinent URL. Finally, after chatting with the third CompanionBot for five days, subjects received the concluding part of the questionnaire and the date and location for a five to ten minute face-to-face interview. In truth, the entire experiment endured for a minimum of three weeks before completion.
Subjects were informed that each CompanionBot possessed some emotional facial expressions that reacted to the subject’s input, and they had the possibility to attempt to observe how many facial expressions they could make them display. Moreover, CompanionBots were designed with an IELTS speaking database included and subjects had the ability to practice the speaking component of the IELTS exam with the CompanionBots. Additionally, the keyboard conversations between subjects and the three CompanionBots were recorded automatically for analytical examination. After completing the keyboard conversations with the three CompanionBots, subjects were asked to partake in a face-to-face interview. In fact, the interview entailed six semi- structured questions and an audio recording of the interview was taken. When subjects had completed the entire experiment, they were thanked for their contribution and rewarded with a 2 GB USB flash drive.
This experiment involved recording three types of data. Firstly, the keyboard conversations between subjects and CompanionBots was recorded automatically during the chatting sessions and stored as a computer text file. Secondly, the subject interviews was recorded and stored as a digital audio file. Finally, the questionnaire data was gathered for analysis in order to identify the activity level during HCIs.
151 8.2. Subjects
42 subjects partook in the actual experiment, of which 11 subjects failed to complete the experiment procedure (seven did not finish the keyboard conversations with three CompanionBots and four did not attend the interview), and their responses were discarded from the analysis, which was performed on the remaining 31 cases. According to the background information provided in the first part of the questionnaire, all 31 subjects were learning English as a second language and were recruited on the basis of this to partake in the experiment. Additionally, the complete subject group spoke Chinese as their first language (including Mandarin and Cantonese).
Other personal information gathered in the first part of the questionnaire is as follows: 11 (35.5%) of the 31 subjects were male, and 20 (64.5%) were female, as indicated in Table 16. 14 subjects (45.1%) were aged from 15 to 25 years and the other 17 (54.9%) were aged between 26 and 35 as depicted in Table 17.
Table 16. Gender of subjects in the character interface survey. Percentage of Total Sample Gender Number of Subjects (%) Male 11 35.5 Female 20 64.5 Total 31 100.0
Table 17. Age groups of subjects in the character interface survey. Percentage of Total Sample Age Group Number of Subjects (%) 15-20 years old 3 9.6 21-25 years old 11 35.5 26-30 years old 11 35.5 More than 31 years old 6 19.4 Total 31 100.0
In terms of the experience of English learning, ten (32.3%) had less than six years English learning experience and two subjects had less than two years experience. Furthermore, 21 (67.7%) subjects possessed more than six years English learning experience and eight of them in excess of ten years as shown in Table 18. The vast majority, 29 subjects (93.5%), had been learning English for more than 2 years. The utilisation of language practice as the content for experimental interactions underlines the language learning background of subjects. This is critical, given that the majority of subjects had been studying and becoming familiarised to English for a considerable period of time. Additionally, eight subjects (25.8%) had from two to six years of 152 computer usage experience, 11 (35.5%) possessed from six to ten years, and 12 (38.7%) had gained more than ten years experience as indicated in Table 19. In this survey, subjects were asked to have keyboard conversations with CompanionBots via computers. This information provides a general understanding of subjects in terms of their computer skills, thereby ensuring that computers were unlikely to become barriers during the conversations. Moreover, as depicted in Table 20, only two subjects (6.5%) used computers less than two hours per day on average, 16 (51.5%) used computers for from two to six hours, 11 (35.5%) employed computer for between six and ten hours and two (6.5%) utilised computers for more than ten hours a day. Accordingly, it is patently clear that the majority of subjects have substantial experience in using computers.
Table 18. English learning experience of subjects in the character interface survey. Percentage of Total Sample English Learning Duration Number of Subjects (%) 0-2 years 2 6.5 2-6 years 8 25.8 6-10 years 13 41.9 More than 10 years 8 25.8 Total 31 100.0
Table 19. Computer usage experience of subjects in the character interface survey. Percentage of Total Sample Computer Usage Experience Number of Subjects (%) 0-2 years 0 0.0 2-6 years 8 25.8 6-10 years 11 35.5 More than 10 years 12 38.7 Total 31 100.0
Table 20. Computer usage per day of subjects in the character interface survey. Percentage of Total Sample Computer Usage/Day Number of Subjects (%) 0-2 hours 2 6.5 2-6 hours 16 51.5 6-10 hours 11 35.5 More than 10 hours 2 6.5 Total 31 100.0
Furthermore, the entire subject group had experience chatting with people online, such as chatting with friends via MSN Messenger or QQ Messenger. This information is beneficial to indicate that subjects are unlikely to be troubled to converse with CompanionBots online, given that having keyboard conversations with CompanionBots is similar to chatting with others online. When it comes to the average time that subjects
153 spent chatting with others in their primary language compared with conversations conducted with others in English (as indicated in Table 21), 17 subjects (54.8%) chatted with others less than two hours a day, 12 (38.7%) between two and six hours, and only two (6.5%) spend more than ten hours daily chatting with others through various computer programs. Thus, this data was deployed to determine whether the subjects were talkative since they would have to converse with three strange computational agents online. In terms of chatting with others in English (as demonstrated in Table 22), 28 (90.3%) chatted with others in English less than two hours daily, only three (9.7%) between two and six hours, and no one spends more than six hours to chat with others in English. As already alluded to above, the mother language of all subjects is not English, and they were learning English in a country where it is the official language. However, compared to the talkative data, subjects seemed not to be using English to chat with each other to a major extent.
Table 21. The average time subjects spent chatting with others per day. Percentage of Total Sample Chatting with Others/Day Number of Subjects (%) 0-2 hours 17 54.8 2-6 hours 12 38.7 6-10 hours 0 0.0 More than 10 hours 2 6.5 Total 31 100.0
Table 22. The average time subjects spent chatting with others in English per day. Percentage of Total Sample Chatting with Others in English/Day Number of Subjects (%) 0-2 hours 28 90.3 2-6 hours 3 9.7 6-10 hours 0 0.0 More than 10 hours 0 0.0 Total 31 100.0
Overall, the entire subject group seemed to have relatively developed usage experience of computers, thereby meaning that it should not be a problem to have keyboard conversations with CompanionBots via computers. In terms of factors of learning English as second language, 67.7% of subjects possessed more than six years of English learning experience, though it seemed that those subjects had spent some time learning English. On the other hand, 45.2% of subjects chatted with others for more than two hours per day, however, only 21.4% of them chatted with others in English for in excess of two hours daily. In summary, the entire subject group were learning English as a second language when they partook in this experiment. This experiment provided them with the
154 opportunity to practice English conversations online. As a result, these factors might influence subject’s willingness to undertake keyboard conversations with CompanionBots online.
8.3. Questionnaire Analysis
This survey intends to ascertain the answers to the questions whether beginner English learners have different affective responses towards distinct character classifications of ECAs. Furthermore, it aims to establish the relationship between character preference, user engagement and user-agent relationships during HCIs. This experiment was analogous to the hardware interface survey in that three types of data were obtained and two of these were analysed. One form is the questionnaire concerning subjects’ interactions and the three character classifications of CompanionBots. Subjects completed the questionnaires when they had finished the keyboard conversation component of the experiment with each individual CompanionBot. This data was collected from the questionnaire and was utilised to provide meaningful qualitative results in this experiment. Other data was obtained from the keyboard conversations between the subjects and the CompanionBots, and this information has been recorded and stored as text files. These files were employed to determine the validity of the data based on the level of activeness of the conversations, as subjects were invited to chat with CompanionBots for at least five minutes per day during the experimental duration. The third type of data was garnered from the audio recordings of the in-person interviews. This form of data was the qualitative component of the experimental results, and was analysed to identify interactions between subjects and the three classifications of CompanionBots.
The questionnaire entailed four parts. The first one was concerned with the personal information of the subjects, involving ten questions addressing individual’s basic personal data, such as gender, age, experience of computer usage and familiarity with English learning. There were also questions surrounding the subjects’ selection of their five favourite CompanionBots selected from the list of 18. In this case, subjects were instructed to rank their five chosen CompanionBots in order of preference. The second, third and fourth parts were questions in relation to the interactions between subjects and the three respective CompanionBots. Each part contained 27 questions surrounding aspects, including their character preference, user engagement, user-character relationships and empathy formation.
155
8.3.1. Character Preferences before Experiment
This part of the questionnaire intends to investigate the character preference of users towards CompanionBots prior to any interactions with particular CompanionBots. Before subjects partook in the actual experiment, they were asked to choose five characters (from 18 characters) as their preferred language learning CompanionBots and ranked them according to their personal inclination. This part intends to ascertain which type of character is popular as language companion agents before having interactions with users. The first preference of each subject was awarded five points, number two received four points, number three was granted three points, number four earned two points and number five acquired one point. The scores gained by each character and classification are indicated in Table 23, and the ranks of each character of the three classifications are depicted in Figure 38.
Table 23. The scores of each character. Human Total
Scores 47 18 9 56 12 13 155 Animal Total
Scores 63 23 11 25 39 7 168 Creative Total creature
Scores 40 14 22 12 24 30 142
70 60 50 40 Human 30 Ainmal 20 Creative creature 10 0 No. 1 No. 2 No. 3 No. 4 No. 5 No. 6
Figure 38. The ranks of the three character classifications.
From the above table and graph, it is evident that the highest score was attained by the monkey (63 points), which belongs to the animal classification, the second and the third
156 ones are the girl and the boy (56 and 47 points respectively), which derive from the human classification, the fourth most popular was the plant-like character (40 points), which belongs to the creative creature classification, and the fifth one is the giraffe (39 points), which also belongs to the animal classification. Among the top five characters, two derive from the human classification, two belong to the animal classification, and one from the creative creature classification. On the other hand, the five least preferred characters were: the turtle (7 points; animal), the lady (9 points; human), the rooster (11 points; animal), the old lady (12 points; human) and the monster (12 points; creative creature) in that particular order. This least favoured category also entailed two preferences arising from the human classification, two from the animal classification, and one from the creative creature classification. In fact, the most and least popular character both belonged to the animal classification, namely the monkey and the turtle. As a result, it is problematic to determine the character preferences of users prior to interactions. Additionally, the total scores of the three classifications were ranked from highest to lowest as animal, human and creative creature respectively.
8.3.2. Item analysis
In terms of the reliability of this experiment, this experiment adopted the laboratory similar environment as the experiment environment. This experiment asked subjects to went through the controlled experimental process, interact with ECAs during the controlled period, and conducted the face-to-face interviews under the controlled environment. These factors are all improve the stability and consistency of this survey. In addition, the wordings of the questions of this experiment were simple and easy to understand, as well as provided Chinese translations of the questionnaire questions. This factor aid subjects to focus on the questions and simplify the answering procedure. Referred to the validity of this experiment, questionnaire questions were adopted from previous research theories that listed in the session of Measures in Chapter 7. Moreover, these questions are designed to answer the research questions and compass the research hypotheses. This experiment also analyzed these 27 items of the research questionnaire to ameliorate the reliability and validity of this survey.
The analysis entailed 27 questions grouped into seven categories in the questionnaire, which was used to compare the three CompanionBots. As indicated in Table 24, the item 10 ‘Learning English is a waste of time’ was removed from the scales after item analysis
157 (r=.153). The remainder of the items reached significant differences, and as a result were included in the analysis.
Table 24. Item analysis about the questionnaire of the three CompanionBots. Scale Item CR r 1. I like this CompanionBot 9.479 .795** Preference 2. I would like to talk to this CompanionBot more 12.283 .783** 3. I felt sorry for this CompanionBot when it was crying 5.125 .499** Empathy 4. I felt this CompanionBot was angry when it wore an 2.612 .317** angry face. 5. I felt relaxed when I chatted with this CompanionBot 8.802 .710** Enjoyable 6. I enjoyed chatting with this CompanionBot 10.366 .794** 7. I felt anxious when I chatted with this CompanionBot in 4.869 .549** English. Anxious 8. I did not feel good when I chatted with this 4.613 .535** CompanionBot in English 9. I plan to learn as much English as possible 2.220 .241** 10. Learning English is a waste of time 1.750 .153 11. Studying English can be important for me because I will be able to participate more freely in the activities of 3.752 .323** Learning other cultural groups Attitude 12. Studying English can be important for me because it 3.772 .402** will make me a more knowledgeable person 13. I am afraid the other people will laugh at me when I 1.252 .212** speak English 14. How well did this CompanionBot meet your needs? 9.625 .776** 15. In general, how satisfied were you with your 10.111 .766** relationship with the CompanionBot? 16. How good was your relationship with the CompanionBot compared to your best human face-to-face 9.581 .760** relationships? Relationship 17. How often did you wish you had not got into this 5.612 .585** relationship with the CompanionBot? 18. To what extent has your relationship with the 7.992 .726** CompanionBot met your original expectations? 19. How much do you like this CompanionBot? 8.241 .750** 20. How many problems were there in your relationship? 5.789 .604** 21. This CompanionBot appealed to me 6.956 .641** Engagement - 22. This CompanionBot gave me a good feeling 6.206 .646** Involvement 23. This CompanionBot attracted me 7.698 .671** 24. This CompanionBot left me with cold feelings 6.319 .575** Engagement - 25. I felt negative toward this CompanionBot 6.909 .677** Distance 26. I felt distance between this CompanionBot with me 7.932 .685** 27. I disliked this CompanionBot 9.909 .727** *: p<.05, **: p<.01
8.3.3. Affective Responses of Users toward CompanionBots
This segment of the questionnaire aims to determine the answers to the research questions that ask whether learners have different levels of character preference, user engagement and user-agent relationships towards distinct character classifications of ECAs. In total, seven categories of user affective responses were measured in the questionnaire. Those
158 affective responses of users were analysed by means of a one-way ANOVA test. Significant differences were found in ‘Engagement (F=11.210, p<.05)’and its two factors, ‘Involvement (F=7.901, p<.05)’ and ‘Distance (F=11.610, p<.05)’, among the three CompanionBots. Table 25 below includes the details of the sum of squares, degrees of freedom and mean of the squares. Among the three classifications that exhibited significant differences, they all indicated that C>B>A. This demonstrates that subjects experienced a stronger engagement in the interactions with Chris (creative creature), followed by Ben (animal), and weakest with Andrew (human). In fact, no significant differences were determined between ‘Preference’, ‘Empathy’, ‘Enjoyable’, ‘Anxious’, ‘Learning Attitude’ and ‘Relationship’. This indicates that the subjects experienced a similar level of these affective responses towards the three CompanionBots.
Table 25įġOne-way ANOVA of seven affective responses of subjects toward CompanionBots. Category Source SS DF MS F P Scheffé Factor 1.274 2 .637 Preference Error 69.806 90 .776 .821 .443 Total 71.081 92 Factor .339 2 .169 Empathy Error 58.677 90 .652 .260 .772 Total 59.016 92 Factor .489 2 .245 Enjoyable Error 64.581 90 .718 .341 .712 Total 65.070 92 Factor 2.167 2 1.083 Anxious Error 67.565 90 .751 1.443 .242 Total 69.731 92 Factor .114 2 .057 Learning Error 34.735 90 .386 .148 .862 Attitude Total 34.850 92 Factor .316 2 .158 Relationship Error 63.221 90 .702 .225 .799 Total 63.537 92 Factor 9.957 2 4.978 Engagement - Error 56.710 90 .630 7.901* .001 C>B>A Involvement Total 66.667 92 Factor 12.052 2 6.026 Engagement - Error 46.714 90 .519 11.610* .000 C>B>A Distance Total 58.766 92 Factor 11.128 2 5.564 Engagement Error 44.673 90 .496 11.210* .000 C>B>A Total 55.801 92
159 *: p<.05, A: Andrew, B: Ben, C: Chris
The findings demonstrated that the mean scores in the categories of ‘Preference’, ‘Enjoyable’, ‘Anxious’, and ‘Relationship’ did not reveal a significant difference, according to the slight differences of the quantities of the means of these categories. In other words, Chris continued to receive the highest score, followed by Ben and the lowest figures for Andrew. This indicates that Chris had more affective influences on users, including encouraging greater ‘Preference’, ‘Enjoyable’ and ‘Relationship’ among users as well as causing a increased amount for ‘Anxious’ of users during HCIs. On the other hand, Andrew had a less affective impact on users in terms of these four affective factors. In the case of ‘Empathy’, Andrew received the highest score, Ben was placed second and Chris was ranked lowest in this category. This demonstrates that subjects sensed more empathy towards the human-like character, Andrew, followed by Ben, the animal-like character, and lastly the creative creature, Chris. Overall, Chris (the creative creature character) received the most affective response from the subjects, followed by Ben (the animal character), and least affective responses being towards Andrew (the human character).
The results in this segment have indicated that subjects were significantly more engaged in interactions with Chris (the creative creature character) than either Ben (the animal character) or Andrew (the human character). Additionally, the creative creature classification character had marginally more influence on human affective factors (‘Preference’, ‘Enjoyable’, ‘Anxious’ and ‘Relationship’) compared with the human and animal classification characters. On the contrary, the human classification character (Andrew) gained more empathy from subjects than either Ben or Chris. On average, Chris earned more affective responses from subjects, and this result verifies the research hypothesis that character classification of ECAs influence human affective factors during HCIs.
8.3.4. Relations between Preference, Relationship and Engagement
This section of the questionnaire intends to establish the answers to the research questions concerning whether relationships exist among character preference, user engagement and user-agent relationships. According to numerous studies already mentioned in Chapter 4, affective factors occur to be closely linked to others under the most circumstances. This study hypothesises that the more the user prefers the agent character, the greater the
160 amount of engagement in interactions with the agent character, resulting in a better relationship between the user and the agent character. The relationship between character preference, user-agent relationships and user engagement were analysed using the Pearson’s product-moment correlation coefficient.
As indicated in Table 26, significant correlations exist between preference, relationship and engagement, thereby indicating the close connections that pertain between them. The strongest correlation determined was between preference and relationship (r=0.800, p<.01), and the second being between preference and engagement (r=0.540, p<.01), although a weaker correlation was established between relationship and engagement (r=0.486, p<.01). The entire range of relationships between the three affective responses achieved statistically significant correlations. In truth, the relationship proved that the affective factors are linked closely to other factors like those mentioned in previous studies noted above. Additionally, these responses corroborate the research hypothesis that correlations exist among character preferences, user engagement and user-agent relationships. The level of user preference for an agent character directly impacts the degree of user engagement in interactions with the agent character, and the degree of user engagement further influences the development of relationships between the user and the agent character.
Table 26. Pearson’s product-moment correlation coefficient of preference, relationship and engagement. Affective Factors Preference Relationship Engagement Preference 1 Relationship .800** 1 Engagement .540** .486** 1 *: p<.05, **: p<.01
In addition, Figure 39 illustrates the overall correlation between character preference, user-agent relationships and user engagement that found in the character interface survey.
Figure 39. The correlation between preference, relationship and engagement 161 8.3.5. Other Analyses
The collected data from the questionnaires was also employed to compare the various affective responses between gender and age. These various affective responses between males and female were analysed utilising the T-test. In reality, no significant differences (p>.05) existed between males and females in terms of ‘Preference’, ‘Empathy’, ‘Enjoyable’, ‘Anxious’, ‘Learning Attitude’, ‘Relationship’ and ‘Engagement’. As a result, this indicates that males and females have a similar level of affective responses towards the CompanionBots. Moreover, the various affective responses among the four particular age groups were analysed via the one-way ANOVA test. In fact, no significant differences (p>.05) existed between these four age groups in the seven affective response categories. This also confirms that different aged subjects had similar affective responses toward the CompanionBots.
8.4. Qualitative Results
In the character interface experiment, the questionnaire was collected as quantitative data, but the semi-structured interview recordings were also utilised as qualitative materials. This qualitative data was gathered to gain better insights into the quantitative results.
In truth, subjects were briefly interviewed about their experience and interactions with the CompanionBots following the interactions with the three CompanionBots and completing their self-report questionnaires. The interview process involved six structured questions and some open-ended questions that considered the subjects’ answers to the previous six questions.
The first question concerned the English learning assistance offered by the CompanionBots. In fact, this question intends to confirm the utility of the mission the CompanionBots conveyed, language practice, and to verify that the utility is unlikely to become a barrier for subjects to converse with the CompanionBots. After undertaking a keyboard conversation with the CompanionBots, 23 subjects admitted that CompanionBots offered them minimal assistance with English learning. Most assistance was noted in relation to vocabulary, such as learning new words and phrases during the conversations as well as granting the opportunity to practice spelling. Others valued the conversations with the CompanionBots, as they provided opportunities for subjects to use formal English and learn various greetings from the CompanionBots. On the other hand,
162 nine subjects stated that the CompanionBots did not help them learn English, as the CompanionBots were not particularly smart and often gave incorrect answers. This is important, given that most subjects were inexperienced English learners. Nevertheless, the keyboard conversations with the CompanionBots offer opportunities for the subjects to practice English in privacy in their free time. Subjects involved in chatting with the CompanionBots sensed that the experience aided them to think in an ‘English’ way, and increased the prospect of employing formal or academic style English. This result indicates that the CompanionBots in this experiment did adequately perform their function with their utilities to assist subjects with their English learning.
This section was followed by three questions, which invited the subjects to select which of the three CompanionBots they most liked to chat with, which one helped them to learn the most, and which one they would want to spend more time with. These series of questions intended to confirm the character preferences of subjects towards these CompanionBots. In this study, seven subjects declared that they preferred to chat with Andrew, and the main reasons given were that he was human-like and humorous. In addition, nine subjects who most liked to chat with Ben did so because they enjoyed his use of expressions. The other 15 subjects choose Chris as the CompanionBot that they preferred. According to the 15 subjects, most of them felt that Chris was lovelier than Andrew and Ben, and they felt more comfortable when they chatted with Chris. It is obvious that character preference affect the users’ willingness to chat with the CompanionBots. This underscores that subjects are more likely to chat with the CompanionBot if they feel a connection with them and one for whom they have a preference. This result shows that majority preferred the creative creature character, Chris, followed by Ben, and the least preferred, Andrew.
Another question directed towards the subjects concerned which CompanionBot was most supportive to augment English learning. Among the three CompanionBots with the same conversational database and interfaces, albeit assorted classifications of characters, this question was employed to confirm whether the character classifications affect the utilities of the CompanionBots. Although the three CompanionBots shared the same conversational database, only eight subjects stated that the three CompanionBots offered the same level of assistance with English learning. In fact, three subjects deemed that Andrew was the most helpful, as they chatted with him more. In contrast, seven subjects selected Ben as the CompanionBot that helped them most with their English learning, for the same reason that these subjects chatted with Ben more than with the other CompanionBots. The remaining 13 subjects judged that Chris was the most
163 accommodating of the CompanionBots, as they sensed a stronger connection towards Chris, and therefore spent more time conversing with this particular CompanionBot. In truth, this information demonstrates that even those CompanionBots were equipped with the same conversational database, albeit with various character classification representations of them might have caused the outcomes to be dismissed. The creative creature, Chris, was judged to offer better assistance in terms of language practice, followed by the animal character, Ben, and the least amount being derived from the human character, Andrew.
When subjects were asked which CompanionBot they wanted to spend more time with, six subjects selected Andrew because they thought that Andrew was more human-like. On the contrary, eight subjects selected Ben as the CompanionBot that they wanted to spend more time with. Additionally, some subjects suggested that they felt more familiar and comfortable with Ben because he was depicted as an animal character. Moreover, 15 subjects believed that Chris was preferable to the others, as it he had some funny expressions, and as a result they wanted to spend more time with Chris. One subject selected both Ben and Chris since he thought they were cute and he liked them more. Another subject stated that the three CompanionBots were similar and equipped with the same conversational database, thus he/she intended to spend a broadly similar time with each. In truth, this question assessed the subjects’ willingness to spend time with the agents and to continue the interactions that assist user engagement. Overall, more subjects intended to spend more time with Chris, followed by Ben, and finally Andrew, as well as subjects seemed more engaged with Chris than either Ben or Andrew.
The above three questions all invited subjects to select one from the three presented CompanionBots, and 14 of the 31 subjects chose the same CompanionBot for each of these three questions. In reality, subjects declared that the more they liked the CompanionBot, the more time they wanted to spend time with it in order to facilitate the CompanionBot to assist them with their English learning. In this case, nine of the 14 subjects selected Chris, four chose Ben and only one subject preferred Andrew. These findings illustrate that Chris was the most popular among the three CompanionBots, followed by Ben and the least popular being Andrew. In addition, several subjects assumed that character preferences related to the level of CompanionBots’ assistance with English learning. This reiterates that the affective factors may closely connect to other factors, such as preference related to utility and engagement in the subjects’ view.
164 The fifth question invited subjects to estimate which scenario is likely to make them feel most relaxed during English learning activities in terms of interacting with a CompanionBot, with an English native speaker or with a classmate who was learning English as a second language. Since CompanionBots were developed as language companion agents, this question intends to ascertain whether ECAs are suitable to be developed as language companions. Eight of the 31 subjects selected English native speakers, mostly based on the reason that speaking with native speakers would improve their oral English communication by the greatest amount or it may imbue subjects with additional fluency and correct English. On the other hand, subjects deemed that conversing with a language-learning classmate was unlikely to improve their English by a significant degree, as these classmates possessed a similar level of English abilities as the subjects themselves. In addition, subjects did not equate a conversation with a CompanionBot compared with a face-to-face conversation, given they identified the CompanionBots as computer programs and not adequately intelligent to respond correctly to all the questions being asked.
11 subjects selected classmates as their preferred means to develop their English skills. According to their responses, they experienced some stress when they chatted with English native speakers, due to their poor English abilities. Moreover, they were afraid to make mistakes when they chatted with native speakers and also felt that the native speakers may become impatient with them. Additionally, subjects did not always understand the answers that CompanionBots provided. On occasions, subjects felt frustrated when they communicated with CompanionBots. On the other hand, most subjects who suggested classmates made them feel the most relaxed to practice English with, stated that classmates had a similar level of English ability, and thus only had the ability to extend them to a limited degree. In fact, they also deemed that classmates had similar problems with English to them and that together they may be capable to resolve these issues. Based on the similar level of English abilities among subjects and their classmates, subjects assumed that their classmates were more likely to understand what they were talking about and express more patience for their situation.
The other 12 subjects chose CompanionBots as their preferred and most relaxed mode of English conversation. This group noted similar situations as above as disadvantages to chat with native speakers and with classmates. In truth, some subjects experienced stress when conversing with native English speakers and were afraid to learn incorrect English from their classmates. Conversely, subjects felt more relaxed while chatting with CompanionBots. Firstly, they judged that CompanionBots used correct English even
165 when they gave incorrect answers. Secondly, CompanionBots were constantly available. Thirdly, subjects were not afraid to hurt CompanionBots feelings. Fourthly, CompanionBots were unlikely to display impatience or negative emotions towards the subjects personally. Finally, subjects typically were not afraid of making mistakes when they chatted with the CompanionBots. This may derive from subjects’ ability to take their time to check their dictionary or consider their questions. Therefore, if subjects did make mistakes in their questions and/or CompanionBots did not fully understand their questions, all they had to do was to attempt another question. According to these results in the subject background of the questionnaire, 12 subjects chose CompanionBots as their preferred language practice companions (more than chose classmates and native speakers), and 90.5% of subjects (who were learning English as a second language) spent less than two hours per day to practice English even they were in an country where it is the official language. Hence, it seems an effective idea to develop ECAs as language practice companions for inexperienced language learners for practice purposes.
The final structured question of the interview process asked subjects concerning the appearance and functions of their ideal CompanionBot. Five of the 31 subjects stated that the CompanionBots’ appearance had minimal impact on their usability or preference. The remaining 26 subjects depicted the appearances of their perfect CompanionBots with the majority stating that their ideal CompanionBot is likely to be lovely, cartoon-like and good-looking characters, such as babies, handsome gentlemen or beautiful ladies. Two subjects specifically displayed preference for existing and known virtual characters, such as Hello Kitty and Doraemon, and selected such figures as their ideal CompanionBots. In terms of the functions of CompanionBots, the majority of subjects suggested that CompanionBots should possess more intelligence, as existing CompanionBots often offered incorrect answers. Heylen, van Es, Nijholt and Dijk(2002) also proved that a cartoon-like character with human-like gaze is rated as having more usability, satisfaction, involvement and naturalness. Another function mentioned during these interviews was the ability of CompanionBots to correct their mistakes, whether in grammar, spelling or pronunciation. Other functions, such as telling joke and weather forecasting were mentioned while several subjects indicated that more expressions might improve the interactions between users and CompanionBots.
After the six structured questions, subjects were asked about their interactions with the CompanionBots and their opinions in relation to them. This question intends to gain further suggestions for the optimal design of future CompanionBot developments. In fact, variables of character classifications, backgrounds, gestures and expressions were
166 discussed in terms of visual interface. Subjects believed that assorted characters and backgrounds may generate varied conversational topics as well as suggesting that additional gestures and expressions may create more vivid CompanionBots. Subjects suggested that a significant difference existed between chatting with a human-like character rather than with an animal or creative creature. As a consequence, users hinted that they might feel more relaxed when chatting with an animal-like or a creative creature character.
Several subjects referred to the personalities of CompanionBots, and they suggested that different personalities of the CompanionBots were likely to elicit fresh feelings and new conversation topics. According to the study of Chen, Naveed and Porzel (2010) that the personalities of agents do influence user preference and behaviour, and users typically prefer extroverted agents to introverted ones. In addition, the visual appearance of the CompanionBots often affected how the user views the CompanionBots’ personalities. Furthermore, subjects also raised the intelligence level of CompanionBots as a major topic of concern, as the existing CompanionBots often offered incorrect answers. Accordingly, subjects suggested that a more intelligent CompanionBot may be more helpful to assist language learning.
8.5. Summary
The character interface experiment has been conducted to determine the answers to the questions concerning whether learners have different affective responses (character preference, user engagement and user-agent relationships) towards distinct character classifications of ECAs. Moreover, it aims to define the precise nature of relationship, if any, amongst character preference, user engagement and user-agent relationships. The result has been determined, as this research hypothesised that distinct character classifications of ECAs influence human affective factors differently, and significant correlations exist among character preference, user engagement and user-agent relationships during HCIs.
According to the analysed quantitative results, subjects were more engaged in interactions with Chris (the creative creature), followed by Ben (the animal) and least with Andrew (the human). While no significant difference was discovered between 'Preference', 'Empathy', 'Enjoyable', 'Anxious', 'Learning attitude' and 'Relationship', subjects had a similar level of affective responses towards the three CompanionBots. However, there
167 remains some minor differences of the quantities of means, which indicated that the creative creature, Chris, as the agent interface that had more affective influences on users on average. Moreover, subjects sensed greater empathy towards Andrew, the human-like character, followed by Ben, the animal-like character, and Chris, the creative creature. In fact, Chris did offer more affective influences on users according to the analysis of qualitative data, including preference, utility and engagement, with Ben offering less and Andrew the least.
All relationships among the three affective responses, namely character preference, user engagement and user-agent relationships, achieved statistically significant correlations. The qualitative data also proved that various agent interfaces have assorted levels of impact on affective factors, and connections existed among affective factors during HCIs, such as preference, utility and engagement. In addition, it is obvious that character preference affects the users’ willingness to chat with the CompanionBots. Moreover, subjects were more likely to chat with the CompanionBot for whom they felt a stronger attraction. In terms of appearance, the majority of subjects preferred the lovely, cartoon- like and good-looking characters, such as lovely babies, handsome gentlemen and beautiful ladies. In truth, subjects claimed that chatting with human-like CompanionBots is more comparable to human interaction. Alternatively, subjects suggested that it is different from chatting with animal-like CompanionBots and creative creature CompanionBots. Similarly, results from the study of Sträfling et al.(2010) indicated that their rabbit-like agent was preferred, and subjects rated it more positively and friendly than other agents. It is likely that users might feel more relaxed to chat with animal-like characters and creative creature characters. “Perhaps the non-humanoid agent is more likeable and the participants are thus more likely to spend time with it” (Sträfling et al., 2010, p. 80).
In terms of the future development of CompanionBots, subjects answered that chatting with the CompanionBots during the experimental period did assist them to think in an ‘English’ way, and increased the opportunities to use formal and academic style English. Moreover, more subjects agreed that the CompanionBots are the ideal language practice companions as opposed to classmates and native speakers, and the majority of subjects reported that the CompanionBots did improve their English in the domain of vocabulary and grammar. In addition to the affective factors explored in this survey, the ECAs have also been discovered to be suitable to be developed as language practice companions. Furthermore, other findings from this experiment and the hardware interface experiment will be applied to later ECA developments. Firstly, subjects experienced increased ease to
168 chat with the animal-like and creative creature characters, thus, more animal-like and creative creature characters will be applied to the multi-agent website. Secondly, the character preferences of subjects impacted their willingness to chat with ECAs, therefore, applying the highly preferred characters to CompanionBots will attract users to spend more time with them and learn more effectively from them. Thirdly, subjects gave greater attention to the actual background suitable for a particular conversation topic, consequently, designing suitable background should be considered when creating the language learning environment. Fourthly, users might need tips on the interfaces in order to input questions to trigger specific functions of the CompanionBots, such as sample questions to generate character expressions exchange. Finally, subjects do not always grant sufficient attention to the exchanges of ECA facial expressions since the faces might be distant from the input frame, hence, ECA faces should be placed in a more obvious position.
This chapter succeeds the former chapter to continue to address the details of the character interface survey. In this chapter, firstly each phase of the actual procedure was presented. Secondly, the subjects were analysed according to their factors, including ages, genders, English learning experience and the average time of chatting with others. Thirdly, the quantitative and qualitative data collected was analysed during the experiment and the results of the study were provided. This chapter finalises the experimental part. The following part contains two chapters, which will present the development of a multi-agent platform as well as possible contributions and conclusions of this research.
169
Part III. Adopting Results to Build a Multi-agent Platform
Following providing a detailed introduction and addressing the theoretical and experimental parts of this research study, the practical part will now be covered. In the previous sections, literature reviews have been undertaken in terms of technological, design and psychological perspectives as well as describing the methods, procedures and results of these two surveys. This section will encompass the possible practical applications as well as the results and conclusions of this research.
The precise design guidelines have been explored in the related literature review and the results garnered from these two surveys are adopted to develop a multi-agent website. This website intends to implement these design guidelines into practical use in order to develop better agent interfaces. Chapter 9 involves the development of the website, including the requirements and systems architecture. In fact, this research has conceived some original developments. Finally, Chapter 10 addresses the main contributions of this research, along with some practical, sociological and research-related implications, and several concluding remarks.
171
Chapter 9. Development of a Multi-agent Website with Affective Interfaces
This research intends to design better agent interfaces for users to enable superior interaction during language practice. Subsequent to conducting two experiments to investigate the affective feelings of users towards different agent interfaces, the outcomes that resulted from the two experiments have been adopted in conjunction with some existing guidelines in order to develop a multi-agent website. This chapter elucidates the main details in developing this website
This chapter comprises four sections as follows: 1) Introduction, 2) Requirements and Specifications, 3) Systems Architecture and Interface Design, and 4) Building the Website. The first section clarifies the purpose, target users, contents and functions of the website as well as the agent character selection. The detailed requirements of file hosting, such as the storage of website data and hardware requirements for users are analysed in the second section. In the third section, the website structure, a detailed description of the content of each webpage and the various facial expressions designed for the CompanionBots are presented. Finally, the actual procedure of website development and the relevant tools that were utilised to develop this website are addressed in the fourth section.
The development of this multi-agent website for inexperienced English learners has been developed with respect to the research results of the previous two experiments. These results include the preferred agent interface design for language learners and the facial expressions of agents that attract users to engage in the learning tasks. Therefore, this is a website designed with preferred agent interfaces that is likely to attract users to further their engagement in the language practice tasks. The website also incorporates ten characters equipped with six basic expressions as well as one neutral face, and these were created as CompanionBots.
Moreover, this website provides some significant design, computer science and psychological challenges in terms of agent character design (user impression in a learning task), agent emotional expression design (evoking users’ affective responses), the Emotional Keyword Filter (emotional keywords lead to agent expression exchanges), device compatibility (suitable use on computers and smart mobile devices) and the IELTS-Bot (designed for IELTS rehearsal use). In truth, the website has been devices via
173 certain computer science techniques and psychological theories. This chapter presents the design of this multi-agent website that attempts to address many of these challenges.
9.1. Introduction
This research investigates the affective interfaces of ECAs, and this website aims to provide a platform for improved design-agent interfaces and enable improved interactions between users and the ECAs. The previous two surveys established that ECAs interfaces could alter users’ affective states through hardware interfaces and character classifications. Furthermore, users experience greater emotional influence from computer agents, and build better relationships with them because of the device features of hardware interfaces. Hence, this website will utilise computers as the primary platform to present CompanionBots in order to develop greater affective influences on users. Additionally, the character preferences of users towards CompanionBot impact users’ willingness to spend more time with CompanionBots, and further impact the learning assistance judgement evaluated by users. This website adopted existing guidelines and outcomes from these two surveys to design better interfaces for users to enhance engagement during HCIs, such as more preferred characters for CompanionBots, vivid and exaggerated facial expressions of CompanionBots, themed backgrounds for language practice, tips for trigger specific functions and rearranged positions for more obvious display of character faces. In addition to applying these aspects to designing improved interfaces, this website also adjusted some functions from the previous two surveys in order to ameliorate assistance to inexperienced English learners during language practice.
Although verisimilitude models of human-like companion agents should be equipped with a high level of AI and presented by high-end graphic workstations, the CompanionBots applied in this website are unlikely to require users to interact with the CompanionBots in a laboratory setting. Aside from simply attracting inexperienced English learners with a more positive affective state in a comfortable setting, practice tasks were developed and represented by various well-designed characters. Subsequently, ten different characters of CompanionBots were created for the multi-agent website in order to offer users additional choice options to chat with their favoured characters and evoke more positive affective states in the learning tasks.
This website adopted some of the research results from the previous two experiments as well as considering some existing design guidelines to develop better agent interfaces.
174 According to the results of the hardware interface survey, users experienced more emotional influences from the computer agent than the portable device agent because computers are equipped with larger screens. As a consequence, this portrays a larger and more visually comprehensible computer agent, thereby allowing agents to be better able to exhibit facial and emotional expressions, and assist user-agent affinity and engagement. Furthermore, users developed better relationships with the computer agent because the use of keyboard input was considerably easier to navigate than the smaller keys on a portable device. Thus, this increased usability allowed subjects to interact in a more comfortable and familiar manner. However, a trend for users to utilise portable devices and access the Internet in a variety of locations has increasingly developed. Figure 40 displays how users now regard these mobile devices as a necessity, rather than an accessory. In reality, this phenomenon has been manifested by a considerable increase recorded from 2006 to 2009, with usage ratings recorded in 2009 exceeding the values for 2006 by nine percentage points (Phillips, 2010). In addition, Alex Spektor, Senior Analyst at Strategy Analytics, said: “Global handset shipments grew 16 percent annually to reach 400 million units in Q4 2010. Apple was a star performer during the quarter” (Mawston, 2011). Therefore, the compatibility of employing computers and portable devices were both considered when developing the website. Although this website was primarily designed for computer usage, development avoided the use of Flash in order to ensure that this website was also compatible for mobile devices, such as the iPhone, iPod Touch and iPad.
Figure 40. US Internet users who believe a mobile phone is a necessity (Phillips, 2010).
According to the results of the character interface survey, users prefer lovely, cartoon-like and novel characters. In the human classification, the boy (47 points) and the girl (56 175 points) were more popular than the others (which recorded preference lower than 20 points). As a result of these user preferences, both the boy, named Andrew, and the girl, named Calla, were both adopted for application in this website. In the animal classification, the rooster (11 points) and the turtle (7 points) were unpopular and thus discarded, while the other more popular characters (the monkey named Jacob, the giraffe named Gary, the rabbit named Annie and the penguin named Pax) were also harnessed for implementation in the website. In terms of the creative creature classification, the monster (12 points) and the robot (14 points) were rejected due to their lower subject preference. Nevertheless, the other four characters, the angel named Angel, the devil named Evan, the sun-like character named Sunny and the plant-like creature named Chris, were executed as CompanionBots for the website. The full list of these ten characters is depicted as Table 27.
Table 27. Ten characters applied to the multi-agent website.
The human classification
Name Andrew Calla
The animal classification
Name Annie Gary Jacob Pax
The creative creature classification
Name Angel Chris Evan Sunny
Additionally, three AIML-Bots have been applied in this website. Similarly, AAA-Bots containing 59 AIML files have been realised as a conversational database. Moreover, the PI-Bot and IELTS-Bot developed for the character interface survey were also adopted for this website. This means that users who intended to partake in the IELTS exam had the potential to practice the IELTS speaking test with the CompanionBots. Therefore, inexperienced English learners had the ability to practice English conversations with ten CompanionBots as well as the opportunity to revise sample IELTS speaking tests online.
176 The primary target users of this site are inexperienced learners of the English language. In truth, CompanionBots are only able to chat with users with a basic level of English because of the limitations of the AI of CompanionBots. At this stage, CompanionBots do not possess sufficient expertise in the English language to allow users to have extended conversations. However, in the case of individuals who are learning English as a second language, CompanionBots provide notable opportunities for them to practice the basics of English. In fact, inexperienced learners studying English as a second language differ significantly from expert English users who possess the ability to use English fluently. Moreover, inexperienced English learners are prone to make some mistakes in grammar, spelling and pronunciation, possibly resulting in a reluctance to converse with expert or native English speakers due to a fear of making mistakes. According to subjects in the character interface experiment, chatting with CompanionBots assisted them with their spelling and vocabulary.
In reality, the results of the character interface experiment indicated that most subjects judged that CompanionBots were helpful in their language learning tasks, especially in domain of spelling and vocabulary. More than forty thousand patterns exist on the conversational database, covering a number of discussion topics, and this is more than adequate for inexperienced English learners to practice basic conversations. Unfortunately, the conversational database of the CompanionBots is likely to prove insufficient for advanced English conversation practice. However, this research mainly focuses on the agent interface design and human affective responses.
In this study, users can visit this website and select any CompanionBot to engage in a keyboard conversation. As mentioned above, this website is also compatible for most portable devices with access to the Internet. Therefore, users also have the ability access this website via their portable device. This website aims to provide a platform for inexperienced English learners to practice conversations and prepare for the IELTS speaking test online. In this case, the CompanionBot characters were represented with emotional expressions designed to attract users’ engagement in the learning tasks and evoke positive affective responses during HCIs, such as higher character preferences and better user-agent relationships.
177 9.2. Requirements and Specifications
The requirements and specifications in this section have been categorised into the user and the client-server architecture. The client-server architecture is classified as the client (the front-end of the website), designed to be as simple, reliable and light-weight as possible (regarding to user system resources), and the server (the back-end of the website), which is used to store the source code and perform dialogue handlings. Additionally, the observable content (e.g. page layouts and agent interfaces) is referred to as the front-end, which works on the Windows Server 2003 operating system with Microsoft MSSQL Server 2005 Express Edition and .Net Framework 2.0. In fact, the back-end comprises three AIML-Bot format files that process the textual output to the front-end operations under the Windows Server 2003 operating system.
The website facilitates user to browse via their choice of web browsers, including Internet Explorer, Firefox, Safari and Chrome by means of computers or smart mobile devices. The optimal screen resolution available for the website is 1280x800, albeit it remains compatible for other screen resolutions. For higher screen resolution (such as computers equipped with larger screens), extendable backgrounds ensure that the observable content remains in the centre of the screens. In the case of poor screen resolution (for example portable devices equipped with smaller screens), devices are able to adjust the best browser resolution through a zoom-in or zoom-out function. Thus, users have the capacity to adjust their browser resolutions to fit better recognisable presence when they utilise portable devices to browse this website.
9.3. System Architecture and Interface Design
In the back-end, the conversational database contains three AIML-Bots, namely the IELTS-Bot, the PI-Bot and the AAA-Bots. In fact, the IELTS-Bot was developed for the character interface survey to allow subjects to prepare for the IELTS speaking test online. This website also adopted the IELTS-Bots as the conversational database for users to practice sample IELTS speaking tests. Additionally, the PI-Bot was designed for the CompanionBots in the character interface experiment to respond to some questions in relation to the personal information of the CompanionBots. The Annotated ALICE AIML Files (AAA Files, named AAA-Bots in this research) is a revised version of the free ALICE source. In reality, the AAA Files contain 59 AIML that are mostly compatible with all AIML 1.01 compliant software (as shown in the Appendices). The AAA is
178 specifically reorganised to facilitate BotMasters to clone ALICE’s brain and create customised bot personalities, without having to expend much effort in editing the original ALICE content (Wallace, 2007).
The content of the website has been segmented into three distinct categories, namely chatting with CompanionBots, Emotional agents and Experiments. Moreover, the structure of the front-end of the website is depicted in Figure 41 as well as the layout and content of this website as listed in Table 28. In the Chatting with CompanionBots category, three character classification pages exist, namely the human classification page, the animal classification page and the creative creature classification page. In reality, each character classification page contains two to four CompanionBots for users to select for augmented language practice. Additionally, when users click any CompanionBot in the character classification page, the webpage links to one chatting page and users may have more keyboard conversations with the CompanionBots on the chatting page. Furthermore, in the Emotional agents category, all the animated facial expressions of the CompanionBots are listed in the Emotions of CompanionBots page while evaluation of agent facial expressions from the OCC mode’s 22 emotions to the six basic emotions are provided in the Evaluation of agent emotions page. Similarly, all the emotional keywords that initiate a change of expression for the CompanionBots are shown on the Emotional keywords page. In the Experiments category, the aims, variables, procedures and results of both the hardware and character interface experiments are addressed in the Hardware interface experiment and Character interface experiment pages.
179 Chatting page The human classification (X2)
Chatting page Chatting with CompanionBots The animal classification (X4)
Chatting page The creative creature classification (X4)
Emotions of CompanionBots
Emotional agents Evaluation of agent emotions
Homepage Emotional keywords
Hardware interface experiment Experiments Character interface experiment About this site
Sitemap
Figure 41. Structure of the front-end of website.
Table 28. The layout and content of web pages. Title Layout Content Homepage The website title is Affective Interfaces of Embodied Conversational Agents. Users have the ability to chat with any CompanionBots by selecting a character in the three character classification pages and browse other information in the Emotional agents and Experiments categories. About this This page gives an overview of this site website. Firstly, it presents the position of this site in the entire research and the content of this site. Secondly, it addresses the features of target users. Finally, it introduces how to use this site.
Sitemap This page gives an overall structure of the Affective Interfaces of Embodied Conversational Agents web site. It also depicts all the page titles, and users are able link to any page by clicking the webpage title.
180 The human There are two characters in the human classification classification, a boy named Andrew and page a girl named Calla. Users may select anyone of them and go to the chatting page to have keyboard conversations with them.
The animal There are four characters in the animal classification classification, a rabbit named Annie, a page giraffe known as Gary, a penguin known as Pax, and a monkey called Jacob. Users may select any of them and go to the chatting page to have keyboard conversations with them.
The creative There are four characters in the creative creature creature classification, an angel named classification Angel, a devil called Evan, a plant-like page character named Chris, and a sun-like character named Sunny. Users may select any of them and go to the chatting page to have keyboard conversations with them. Chatting Each character has one chatting page. pages Users might have keyboard conversations with any character on the chatting pages. For example, users input sentences in the input frame and they will get textual responses in the output frame and one emotional expression next to the textual response. Emotional There are three pages in this category: agents pages Emotions of CompanionBots, Evaluation of agent emotions and Emotional keywords. Information in relation to agent emotions in this research is illustrated in this category.
Experiments There are two pages in this category as pages follows: the Hardware interface experiment and Character interface experiment. The aims, variables, procedures and results of the hardware interface experiment and character interface experiment are addressed in this category.
This website was designed for inexperienced English learners to practice language online, and the interactions between users and agents formed a vital component tested through the learning tasks. Therefore, the website interface was designed to stimulate a study environment, such as the inclusion of a book to present learning content, vocabulary 181 cards by the side, and a cup of coffee on the table. In terms of the interface structure of this website (see Figure 42), every webpage featured the website title, which allows users to navigate to the homepage of the site. This feature is situated at the top of the page, while a copyright announcement is located at the bottom of each webpage. Furthermore, three tags are positioned on the right fringe of the book. In fact, users have the ability to rapidly link to the three character classification pages via these three tags. Similarly, the Home tag, on the left fringe of the book, provides users with a link to the homepage by clicking the Home tag.
Figure 42. Interface structure of the multi-agent website.
Figure 43 indicates the conversational user interface page of this website. In truth, this interface is an enhanced version of the conversational user interface of the character interface survey. Moving the input frame next to the characters’ facial expressions, and users are likely to pay more attention to the exchange of emotional expressions. Additionally, adding textual input tips, including the IELTS-Bot sample question in the input frame and the emotional exchange tips in the left-bottom corner, in the interface to remind users to operate these utilities effectively.
182
Figure 43. The conversational user interface page.
In fact, agent emotional expressions on this multi-agent website are extended from the character interface survey, with each CompanionBot designed exhibiting six basic emotions, namely anger, happiness, disgust, fear, sadness and surprise as well as a neutral face. The facial expression design also adopts exaggeration and animation as the design guidelines (addressed in the Chapter 7). The entire ranges of the facial expressions of the CompanionBots are indicated in Table 29 and static images of the CompanionBot emotional animations are shown in the Appendices.
Table 29. Facial expressions of the CompanionBots on the multi-agent website. Name Anger Disgust Fear Happiness Surprise Sadness Neutral
Andrew
Calla
Annie
Gary
Jacob
Pax
183 Name Anger Disgust Fear Happiness Surprise Sadness Neutral
Angel
Chris
Evan
Sunny
Similar to the previous two surveys, the Emotional Keyword Filter has also been applied in this website to generate the exchange of the CompanionBot expressions. These ten CompanionBot alter their emotional expressions based on emotional keywords (see the Appendices). The emotional keyword filter program has been developed for this multi- agent website in order to alter facial expressions of the CompanionBots during the keyboard conversations. In truth, the emotional keyword filter program detects the textual responses of CompanionBots and then modifies the emotional expressions of the CompanionBots.
9.4. Building the Website
The development of the website is facilitated by most conversational databases used for the CompanionBots being free resources as well as most components having already been developed for the previous two experiments. Additionally, this website continues to employ and improve these components, and the procedure of this multi-agent website had been classified into two phases, namely an interface design phase and a website development phase.
In the interface design phase, Adobe Photoshop CS4 was employed to design the webpage layout, which was conceived with 1280x800 widescreen resolution. In fact, an extendable background has been adopted in the interface design, considering some users’ computers are equipped with better screen resolutions. Five webpage layouts were designed at first as follows: Homepage, Emotional agents, Experiments, Character
184 classifications and Chatting. However, more than five pages exist on the multi-agent website while these five layouts have been produced as templates that are similarly applied to the other webpages. Furthermore, these webpage layouts have been split into smaller images and saved as JPEG or GIF format files presented on the multi-agent website.
Not only were the interface layouts designed in the interface design phase, but also the character representations and facial expressions for the CompanionBots. In truth, these characters and emotional expressions were designed in Adobe Illustrator CS4. Ten characters of the CompanionBots have been designed with seven expressions each (six basic emotions and a neutral face). Moreover, each face has been produced with several static images into an animation and saved as an animated GIF format in Adobe Photoshop CS4.
In the website development phase, the multi-agent website was developed on Adobe Dreamweaver CS4. Webpages, such as homepages, character classification pages, emotional agent pages and other pages have been created as Hyper Text Markup Language (HTML) format files. Additionally, the chatting pages that exchange data with the server in order to make keyboard conversation operable were developed as Active Server Pages (ASP) format files. The multi-agent website can be viewed at the following URL: http://163.23.151.228/proj/index.html, or by means of the disc, which has been placed in an envelope glued to the inside back cover of this thesis.
This chapter provides the development of the multi-agent website, including introducing the content and purpose of the site, describing the software and hardware requirements for the site, presenting the website structure and interface design, and addressing the process of building the site. Some additional information, such as emotional keywords and static images of animations, are presented in the Appendices. The next chapter will present the significant contributions and implications related to this research.
185
Chapter 10. Conclusions and Future Work
The discussion in this thesis has been addressed from psychological, technological and design perspectives of affective interfaces of ECAs in HCIs with respect to various character representations on various devices. This chapter offers a summary of the conclusions and future work derived from this research. This chapter consists of five sections as follows: 1) Summary of Significant Contributions, 2) Implications, 3) Discussions, 4) Future Work, and 5) Concluding Remarks. The first section examines the four main contributions of the entire research as well as several minor contributions in each chapter. The detailed implications for practice, sociology and research are mentioned in the second section. In the third section, issues, such as dialogue planning, ethical considerations and learning applications are addressed. The fourth section considers technological, design and psychological directions in relation to future work of this research. Some conclusions and results are addressed in the fifth section.
10.1. Summary of Significant Contributions
Affective interfaces of ECAs have formed a central aspect of this thesis. The main contributions of this research are that: (1) The hardware interfaces of ECAs may be related to affective responses of users, such as Sentiment and Likeableness. Due to the actual hardware interface, users felt more sentiment from the computer ECA than the portable device ECA, and liked the computer ECA more than the portable device ECA as well. (2) Character classifications may be related to human affective factors, for example User engagement. During learning tasks, learners experienced a stronger engagement in the interactions with the creative creature character, followed by the animal character, and the weakest with the human character. (3) There are significant relationships between character preference, user engagement and user-agent relationships. The strongest correction determined was between character preference and use-agent relationships, and the second being between character preference and user engagement, although a weaker correction was established between user-agent relationships and user engagement. Additionally, (4) implementing those results along with existing design guidelines into practical use on a multi-agent website.
The intention of this research is bettering ECA interfaces to improve interactions between users and agents, such as emotional influences, character preference, user engagement
187 and user-agent rrelationships. Although the specific knowledge domains, such as the giraffe conversation knowledge in the hardware interface survey and the language learning tasks in the character interface survey, are adopted in this research as the contents/tools to attract subjects to interact with ECAs and further experience distinct ECA interfaces. Above research findings did prove that there are different affective influences of ECAs when they are represented by different interfaces. Besides the interfaces of ECAs, the content of specific knowledge and intelligence of ECAs might be the next steps to improve HCIs.ġ
The preceding chapters have addressed the motivation for the development of affective interfaces of ECAs and a series of explorations in the design, development and evaluation of these ECAs. Chapter 1 tackled the inspirations and the importance of the three core components of this research, namely agent interfaces, affective factors and ECAs as well as a short introduction of this research.
Theories have been presented in Chapters 2 to 4 to illustrate ways to develop theoretical models of affective interfaces of ECAs. In this research, the developments of ECA affective interfaces were based on the theories of AI and social psychology as well as serving as a bridge between the above fields, interface design and HCIs. Studies presented in Chapter 2 have indicated how computational algorithms could be implemented in an intelligent agent to interact with users affectively. Although intelligent agents can be developed without visual representations and social psychology, several studies have pointed out that agents equipped with positive affective factors are beneficial in a number of ways, such as affective computing improve user engagement, satisfaction and cognition. The interface design of ECAs presented in Chapter 3 is novel within the fields of interface design and agent development, in that it considers the visual impact of agent interface on users, rather than solely focusing on usability design. The affective factors presented in Chapter 4 demonstrate how agent interfaces influence human behaviours and minds as well as human affective factors that are closely linked to other factors.
The two surveys presented in Chapter 5 and Chapter 7 reveal that ECAs developed in this research incorporated many firsts. The hardware interface survey is the first designed for comparing affective influences of ECAs on users through different devices, such as computers and portable devices. The character interface survey is the first learning companion agent represented by different character classifications in order to compare various levels of affective responses from users. Furthermore, ECAs in the character
188 interface survey are the first ECAs designed for users to revise IELTS speaking tests online. Both studies demonstrated that users would readily engage in keyboard conversations with ECAs using different devices.
Chapter 6 and Chapter 8 revealed some statistics and qualitative data that can be applied to the ECA affective interfaces, and determined that the ECAs may be used to establish some strategies to maintain and create positive affective influences on users in various devices with different character classification. This data also established some design guidelines for how these strategies can be implemented in language learning tasks.
Chapter 9 presented a multi-agent website that applied numerous design guidelines from the existing interface design and some results from previous experiments. This website combined AI techniques, affective factor design for language learning and affective interface design to develop learning companion agents for novice English learners in order to practice their language skills online.
Finally, perhaps the most significant contribution of this thesis is proof that the ECA interfaces can alter users’ affective states through hardware interfaces and character classifications. In the hardware interface survey, it was ascertained that users had different levels of affective interactions when ECAs were presented in computers and portable devices. In the character interface survey, affective factors were proven to be closely linked to others and were influenced by different character classifications of ECA interfaces.
10.2. Implications
In fact, several lessons can be learned from this research that are applicable to the practical, sociological and research dimensions of building affective interfaces of ECAs.
10.2.1. Practical Implications
Practical implications include implications for human-computer interface design and experimental design. One of the most intriguing findings from these experiments concerns the implications for interface design, which discovered that following just a brief introduction all subjects managed to utilise the agent interfaces without any
189 problems, possibly due to the popularity of instant messaging (IM)7 and intuitive usage of natural language. Another important lesson is that, while realism and aesthetics are highly prized in the interface design of ECAs, character classification representations and emotional expressions of ECAs are similarly important for keeping users engaged with their tasks.
The experimental protocols employed in these two studies are likely to facilitate a new sphere of longitudinal designs. Subjects’ perceptions of agent interfaces were evaluated with a short-term or a long-term experience. In the hardware interface survey, subjects were evaluated after they interacted with ECAs for around 10 minutes as the short-term experiences. However, in the character interface survey, subjects were evaluated after they had keyboard conversations with each ECA for five-days as part of the long-term experiences and the long-term interactions involved are generally more complex (Gong, 2008).
10.2.2. Sociological Implications
In this research, methods that the ECA affective interfaces of ECAs can enhance the positive affective factors during HCIs have been comprehensively addressed. Additionally, this study has shown that ECAs with affective interfaces may provide a positive role in terms of socialisation. In fact, ECAs with affective interfaces have the potential to provide virtual social support when users lack real social support, such as experienced when in a foreign country. Moreover, ECAs are valuable for learning tasks and may assist socialisation by providing specific data sharing. In the character interface survey, it has been confirmed that ECAs are suitable for developing as language learning companions. Furthermore, they can be applied to the learning domain via data sharing. In the case of a learning companion agent sharing learning information with classmates, friends and teachers, and tells them how learning processes are progressing. This might result in competitive or cooperative actions between various subjects as well as allowing instructors to offer appropriate advice based on the information agents provide. In truth,
7 Instant messaging (IM): IM is a text form of real-time communication that happens like a conversation between users over the Internet. Two or more users use IM software (such as MSN Messenger and Yahoo Messenger) via computers or other devices to chat with others.
190 ECAs may also provide private learning demands for users. Some subjects in the character interface survey chose CompanionBots to practice language rather than classmates and native speakers because they intend to practice their second language independently and in privacy. Similarly, some subjects who have left school have claimed they would like to pursue further education by means of private learning with companion agents rather than publicly with classmates or friends.
Some positive impacts of ECAs exist in socialisation just like real humans attempting to capture someone’s attention (Dehn & van Mulken, 2000), and according to the old maxim: ‘better a little fire to warm us than a great one to burn us’. ECAs with affective interfaces might be able to satisfy users with psychological support. However, other sociological concerns are that if users rely excessively on these agents for the virtual social support and learning company, this may decrease individuals’ social skills. This underscores the need to ensure that ECAs with affective interfaces are as rich as possible, and used carefully (Baylor, 2009).
10.2.3. Research Implications
Implications arising from this research involve technological, psychological and design perspectives. With regards to the technological aspects, this thesis has shown how ECAs’ affective interfaces connect affective factors and computational programs. In the psychological area, ECA affective interfaces have shown that applying psychology to technology in order to create computers that appear more human. In terms of the design aspect, this research demonstrates the original intention of most designers, designing better products for users, also known as user-centred design, remains critical.
10.3. Discussion
A reasonable amount of work has been conducted over the past few years on models of agent interfaces and affective factors (discussed in Chapters 2 to 4). However, minimal studies have been applied to the understanding and modelling of the affective interfaces of ECAs between hardware interfaces and character classifications. As a consequence, this thesis represents an underrepresented field of study addressing the affective
191 influences of ECAs when ECAs are represented by various hardware interfaces or character classifications.
In terms of AI, complex real-time dialogue is not necessary for small talk in the hardware interface survey, although it is crucial for conversation practice in the character interface survey. In the hardware interface survey, a limited conversational database of general giraffe knowledge and personal information related to the ECAs was sufficient for subjects to achieve affective small talk with the ECAs. On the other hand, the ECAs in the character interface survey were developed in order for subjects to practice English conversations. More than 50 dialogue-planning files seemed insufficient for English experts to process English conversations. In truth, the complexity of dialogue planning should be designed based on the content and purpose that the ECAs intend to perform.
In relation to learning, one feature of ECAs requiring consideration is the application of correct information for successfully completing the learning tasks. Another dilemma involves the provision of sufficient information based on the content and purpose of the learning tasks. In this field, users typically interact with agents in the learning tasks in order to absorb specific knowledge, thus, the information that agents provide should be beneficial for the users as well as being correct and current. Furthermore, sufficient information related to learning topics is essential for agents to effectively interact with users during the learning processes and attract users to engage in the tasks.
In terms of ethical concerns, the affective interfaces of ECAs, as with any technology, have the potential to be abused. ECAs that are represented by affective interfaces may earn users’ trust during the interactions, which may lead to some prospective ethical problems. One major worry is that personal confidential information may be released to these ECAs. This is a vital issue, given that after a period of interactions, users generally believe agents because agents are computers that, unlike humans, resist gossip with others. However, if agents release any of the users’ personal data for improper or commercial purposes, this is likely to cause a series of ethical conundrums. In addition, the trust towards agents might be misused by scammers or marketers. For example, when users ask for product suggestions, agents might be programmed to provide biased information, resulting in reduced user trust towards agents. It is exceptionally important for agents to earn users’ trust and any type of abuse, such as user trust, is unethical.
A final issue emanating from this study is the judicious employment of the affective influences of agent interfaces. The affective interfaces of ECAs are designed to gain positive affective influences on users, and it is against the purpose of affective interfaces 192 of agents if the agents intentionally alter the affective states of users and further damage their learning outcomes. In fact, the aim of agents in the learning tasks is to assist users. That said, the appropriate use of affective influences of agents on users should be carefully considered.
10.4. Future Work
Numerous propitious future directions in which this research can develop may eventuate in the future. These potential research fields exist in the field of technology, design and psychology.
In terms of technological aspects, innumerable studies have been concerned with the development of advanced intelligent agents that interact with humans with greater efficiency, intelligence and practicality. The computational part of ECA affective interfaces, described in this research, represents a minor point in the larger space of ECA development potential. In truth, many other domains could be further applied to ECAs, such as a well-planned dialogue database, speech systems and emotional cognition. The results of the hardware interface survey were considerably influenced by the device features, thus, the results may only apply to contemporary situations. It is likely that continuing advances of digital devices will facilitate further investigation among various types of devices. Furthermore, the existing dialogue database is not appropriate for those designers who are unfamiliar with program coding. Therefore, some dialogue databases should be developed for BotMaster designers while specific topics of dialogue planning could also be designed for various conversational domains.
Considering the inconvenience and imprecision of speech cognition of existing agents, this research has only adopted keyboard conversation as the means of communication. As a result, real-time speech conversation with agents may be developed for various purposes and devices in the future. Furthermore, the function of agents to recognise human emotions may result in the development of more human-like agents. It is likely that those agents may reply to users with more appropriate emotional expressions, dialogue content and behaviours by means of detecting users’ current affective states. Additionally, agents equipped with more advanced technologies that closely approach affective humans might make interactions between humans and computers more natural and intuitive.
193 In terms of the psychological aspect, this research has only applied self-reporting, questionnaires and interviews to measure human affective states. This underlines the necessity to apply additional sensitive measures to detect human affective states in future studies, such as human behaviour and expression capture, and galvanic skin response. In terms of the psychological facet of agent development, the personalities of agents may involve an interesting issue, given that agents furnished with both emotional expressions and gestures might have remarkable influences in a number of domains. Moreover, agents that build various types of relationships or other affective interactions with users may be another issue worthy of further investigation.
In the design phase, the design of agent interfaces should consider the purpose and environment of agents. In the hardware interface survey, subjects suggested that applying a vibrator function and sound effects to ECAs on mobile phones are likely to make ECAs more attractive to users. Alternatively, in the character interface survey, character classification was not the only point that concerned subjects, as subjects also suggested that the background should also be well designed based on various missions and topics that agents conveyed. Finally, the design of ECA affective interfaces should not only address graphic agents that are displayed on screens interfaces, but also pay attention to the agents that are represented by robots.
In this research, ECAs were designed as learning companions, albeit ECAs in learning tasks may also be suited as all-purpose learning assistants and intelligent personal tutors. Furthermore, researchers and designers might be interested in some issues in relation to learning interactions and outcomes, which were affected by different mission-carried agents as well as some subjects of further affective alterations of humans during HCIs in the course of undertaking learning tasks.
10.5. Concluding Remarks
This body of work represents one of the first steps in the development of technologies for providing inexperienced English learners with ECA affective interfaces for language practice and IELTS speaking section revision, and compares the affective influences of ECAs on users when ECAs are applied to various hardware interfaces. This research specifically addresses agent interfaces, not only exploring existing design guidelines amongst technological, psychological and design aspects, but has also verified that ECA interfaces can alter users’ affective feelings through hardware interfaces and character
194 classifications from two surveys. ECAs, in different hardware interfaces, have determined that they possessed distinct levels of affective influence on users, such as users experiencing increasing emotional influence from computer agents and developing better relationships with computer agents. In addition, creative creature classification characters had more affective influence on users compared to human and animal classification characters. Moreover, relationships existed amongst the three affective factors, namely character preference, user engagement and user-relationships, and these elements achieved statistically significant corrections during HCIs.
In the case of the hardware interface survey, in which different hardware interfaces of ECAs impacted users’ affective states was investigated, this work ascertained that larger screens allowed computer agents a greater range of emotional responses and expressions. It also determined that larger screens had greater emotional influence on users as well as successfully equipping keyboard for easier subject input and developing better relationships. However, most subjects felt greater proximity and convenience using portable device agents because of the portability of portable device. Additionally, the primary objective of this work was not entertainment, and it provided five facial expressions for each agent that required the exchange of facial expressions by responding to specific emotional keywords. On occasion, this sudden change of emotion or the response on the part of the agent may lead to some unexpected entertainment.
In the character interface survey, among three character classifications, the creative creature classification character increased subjects’ engagement in interactions. Moreover, subjects experienced similar levels of other affective responses towards the three character classifications. The human classification character was verified by gaining more empathy from subjects and the creative creature classification character earned more affective responses, such as preference, enjoyment, anxiety, learning attitude and relationship from subjects. In addition, this work also proved that the level of character preference of a subject towards an agent character impacted the degree of user engagement in interactions with agent characters, and the degree of user engagement further influenced the development of relations between the subject and the agent character. Furthermore, in this work, inviting users to actively learn is not the primary design. In fact, the design of the experiment executed some dialogue conversational database for subjects to practice English conversations. This allowed subjects not only to develop their vocabulary and grammar from conversational sentences, but also gain new knowledge from the content of conversations by verifying and searching for the information on the Internet.
195 It is likely that ECAs with affective interfaces have the potential to manage the positive effects on users during HCIs while undertaking learning tasks. The interfaces of the agents are also a crucial component for them in successfully promoting motivations (Baylor, 2009). Those ECAs also have a positive effect on society by furthering interactions with users, such as building social bonds with people.
This thesis has defined the affective interfaces of ECAs, demonstrated them by developing the ECAs suitable for different devices (computers and portable devices) and with various character classifications (human, animal and creative creature classifications), demonstrated that they are capable of significant bonding with users in ways that significantly impacts users’ affective states, and further indicated that their affective influences in a language learning task. This thesis raised the main question as to whether there are any variations in human affective states responding to ECA interfaces that are represented in different hardware interfaces and with various character classifications during HCIs. It also hypothesised that ECAs, which are represented by distinct affective interfaces, should influence human affective states differently. It also proved the hypothesis that ECA interfaces can alter users’ affective feelings through hardware interfaces and character classifications. Similarly, it determined that affective factors (such as character preference, user engagement and user-agent relationships) achieve significant correlations during HCIs. That said, it should be pointed out that the results established in this research might be impacted by the device features of modern tools. This underlines that this field is constantly evolving and these theories are not immutable. However, some phenomena that have been raised in this research continue the necessity to consider the development of the interactions between users and ECAs.
196 References
AbuShawar, B., & Atwell, E. (2003). Machine Learning from dialogue corpora to generate chatbots. Expert Update Journal, 6 (3), 25-29. Adam, A. (1998). Artificial knowing: gender and the thinking machine. New York, NY: Routledge. Adolphs, R. (2005). Could a robot have emotions? Theoretical perspectives from social cognitive neuroscience. In J-M. Fellous, & M. Arbib (Eds.), Who needs emotions? The brain meets the robot (pp. 9-25). New York, NY: Oxford University Press. Amazon. (2011, August 31). Amazon.com. Retrieved March 17, 2010, from Amazon.com: http://www.amazon.com/ Andy. (2005, November 18). Microsoft BOB Review [Discussion forum message]. Retrieved March 17, 2010, from BentUser: http://www.bentuser.com/article.aspx?ID=327&AspxAutoDetectCookieSupport=1 Apple. (2007a, January 9). Apple - Press Info - Apple Reinvents the Phone with iPhone [Press Release]. Retrieved March 17, 2010, from Apple: http://www.apple.com/pr/library/2007/01/09Apple-Reinvents-the-Phone-with- iPhone.html Apple. (2007b, December 18). Mac 101: Exposé [Newsgroup Message]. Retrieved March 17, 2010, from Apple Inc.: http://docs.info.apple.com/article.html?artnum=304786 Apple. (2011). Apple - iTunes - Learn about the features of iTunes 10 [Fact Sheet]. Retrieved March 17, 2010, from Apple: http://www.apple.com/au/itunes/features/ Artificial Linguistic Internet Computer Entity (ALICE) Article Intelligence (AI) Foundation. (2008). A. L. I. C. E. The Artificial Linguistic Internet Computer Entity. Retrieved March 17, 2010, from ALICE. Artificial Intelligence Foundation: http://www.pandorabots.com/pandora/talk?botid=f5d922d97e345aa1 Artificial Linguistic Internet Computer Entity (ALICE) Article Intelligence (AI) Foundation. (2010). AIML: Artificial Intelligence Markup Language. Retrieved March 17, 2010, from A.L.I.C.E. Artificial Intelligence Foundation: http://www.alicebot.org/aiml.html Ashby, F. G., Isen, A. M., & Turken, A. U. (1999). A neuropsychological theory of positive affect and its influence on cognition. Psychological Review, 106(3), 529- 550. doi: 10.1037/0033-295X.106.3.529 Atkinson, R. (2002). Optimizing learning for examples using animated pedagogical agents. Journal of Educational Psychology, 94(2), 416-427. doi: 10.1037/0022- 0663.94.2.416 Bailenson, J., Blascovich, J., & Guadagno, R. (2008). Self-Representations in Immersive Virtual Environments. Journal of Applied Social Psychology, 38 (11), 2673–2690. doi: 10.1111/j.1559-1816.2008.00409.x Ball, G., & Breese, J. (2000). Emotion and personality in a conversational agent. In J. Cassell, J. Sullivan, S. Prevost, & E. Churchill (Eds.), Embodied conversational agents (pp. 189-219). Cambridge, MA: MIT Press. Barker, P., & King, T. (1993). Evaluating interactive multimedia courseware—A methodology. Computers & Education, 21 (4), 307-319. doi: 10.1016/0360- 1315(93)90034-G Barnathan, M., Columbus, C., Katz, G. (Producers), & Columbus, C. (Director). (1999). Bicentennial Man [Motion Picture]. US: 1492 Pictures. Barnes, S. B. (2000). Bridging the Differences between Social Theory and Technological Invention in Human-Computer Interface Design. New Media & Society, 2(3), 353- 372. doi: 10.1177/14614440022225850 Bartlett, M., Hager, J., Ekman, P., & Sejnowski, T. (1999). Measuring facial expressions by computer image analysis. Psychophysiology, 36 (2), 253-263. doi: 10.1017/S0048577299971664 Bartneck, C. (2003, June 23-26).Interacting with an embodied emotional character. Proceedings of the 2003 International Conference on Designing Pleasurable Products and Interfaces (DPPI) (pp. 55-60). Pittsburgh, PA: ACM. doi: 10.1145/782896.782911 Bartneck, C., van der Hoek, M., Mubin, O., & Al Mahmud, A. (2007). "Daisy, Daisy, give me your answer do!": switching off a robot. Proceedings of the Second ACM/IEEE International Conference on Human-Robot Interaction (pp. 217-222). Washington DC: ACM. doi: 10.1145/1228716.1228746 Bates, J. (1994). The role of emotion in believable agents. Communications of the ACM, 37(7), 1-9. Batson, C., Shaw, L., & Oleson, K. (1992). Differentiating affect, mood, and emotion: Toward functionally based conceptual distinctions. Emotion. In M. Clark (Ed.), Emotion, Review of personality and social psychology (pp. 294-326). Thousand Oaks, CA: Sage Publications. Baylor, A. L. (2005). The Impact of Pedagogical Agent Image on Affective Outcomes. Proceedings of Workshop on Affective Interactions: Computers in the Affective Loop, International Conference on Intelligent User Interfaces. San Diego, CA. Baylor, A. (2009). Promoting motivation with virtual agents and avatars: role of visual presence and appearance. Philosophical Transactions of the Royal Society B, 364(1535), 3559-3565. doi: 10.1098/rstb.2009.0148 Baylor, A. L., & Kim, Y. (2004). Pedagogical agent design: the impact of agent realism, gender, ethnicity and instructional role. Lecture Notes in Computer Science, 3220/2004, 268-270.doi: 10.1007/978-3-540-30139-4_56 Baylor, A., & Kim, Y. (2005). Simulating Instructional Roles through Pedagogical Agents. International Journal of Artificial Intelligence in Education, 15(2), 95-115. Baylor, A. L., & Kim, S. (2008). The Effects of Agent Nonverbal Communication on Procedural and Attitudinal Learning Outcomes. Lecture Notes in Computer Science, 5208/2008, 208-214. doi: 10.1007/978-3-540-85483-8_21 Baylor, A. L., & Kim, S. (2009, March). Designing nonverbal communication for pedagogical agents: When less is more. Computers in Human Behavior, 25 (2), 450- 457. doi: 10.1016/j.chb.2008.10.008 Baylor, A., & Rosenberg-Kima, R. (2006). Interface agents to alleviate online frustration. Proceedings of the Seventh International Conference on Learning Sciences (ICLS) (pp. 30-36). Bloomington, IN: ISLS. Baylor, A. L., & Ryu, J. (2003). Does the presence of image and animation enhance pedagogical agent persona? Journal of Educational Computing Research, 28(4), 373-395. BBC. (2010, July 9). Over 5 billion mobile phone connections worldwide. Retrieved
198 March 17, 2010, from BBC News: http://www.bbc.co.uk/news/10569081 Bensaid, N., & Mathieu, P. (1999, December 14-17). A Virtual Shop Navigation System Based on the Multi-Agent Technology. Proceedings of the First Asia-Pacific Conference on IAT (pp. 498-502). Hong Kong Baptist University, Hong Kong. Berry, D., Butler, L., & de Rosis, F. (2005). Evaluating a realistic agent in an advice- giving task. International Journal of Human-Computer Studies, 63 (3), 304-327. doi: 10.1016/j.ijhcs.2005.03.006 Bettman, J. (1982). A Functional Analysis of the Role of Overall Evaluation of Alternatives in the Choice Process. In A. Mitchell (Ed.), Advances in Consumer Research (pp. 87-93). Ann Arbor, MI: Association for Consumer Research. Beun, R. J., de Vos, E., & Witteman, C. (2003). Embodied Conversational Agents: Effects on Memory Performance and Anthropomorphisation. Proceedings of the Fourth International Workshop: Intelligent Virtual Agents (pp. 315-319). Heidelberg, Germany: Springer-Verlag. doi: 10.1007/978-3-540-39396-2_52 Bickmore, T. (2003). Relational Agents: Effecting Change through Human–Computer Relationships (Doctoral dissertation, Massachusetts Institute of Technology). Retrieved March 17, 2010, from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.7.1474&rep=rep1&type= pdf Bickmore, T., & Cassell, J. (2001). Relational agents: a model and implementation of building user trust. Proceedings of the 2001 SIGCHI Conference on Human Factors in Computing Systems (pp. 396-403). New York, NY: ACM. doi: 10.1145/365024.365304 Blair-Early, A., & Zender, M. (2008). User Interface Design Principles for Interaction Design. Design Issues, 24(3), 85-107. doi: 10.1162/desi.2008.24.3.85 Booth, P. A. (1989). An Introduction to Human-Computer Interaction. Hove, East Sussex, UK: Lawrence Erlbaum Associates. Brave, S., & Nass, C. (2003). Emotion in Human-Computer Interaction. In J. A. Jacko, & A. Sears (Eds.), The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications (pp. 81-96). Mahwah, NJ: Lawrence Erlbaum Associates. Brave, S., Nass, C., & Hutchinson, K. (2005, February). Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent. International Journal of Human-Computer Studies, 62 (2), 161-178. doi: 10.1016/j.ijhcs.2004.11.002 Breazeal, C. (2003). Emotion and sociable humanoid robots. International Journal of Human-Computer Studies, 59 (1-2), 119-155. doi: 10.1016/S1071-5819(03)00018-1 Breckler, S. (1984). Empirical validation of affect, behavior, and cognition as distinct components of attitude. Journal of Personality and Social Psychology, 47(6), 1191- 1205. doi: 10.1037/0022-3514.47.6.1191 Brenner, W., Zarnekow, R., & Wittig, B. (1998). Intelligent Software Agents: Foundations and Apllications. Berlin: Springer. Buchanan, B. G. (2005). A (very) brief history of artificial intelligence. AI Magazine, 26(4), 53-60. Bush, V. (1945, July). As We May Think. The Atlantic Monthly, 101-108. Cassell, J. (2000, April). Embodied Conversational Interface Agents. Communications of the ACM, 43(4), 70-78. doi: 10.1145/332051.332075
199 Cassell, J. (2011). Research [Fact Sheet]. Retrieved July 12, 2011, from the Justine Cassell web site: http://www.justinecassell.com/jc_research.htm Cassell, J., & Thorisson, K. (1999, June 1). The power of a nod and a glance: Envelope vs. emotional feedback in animated conversational agents. Applied Artificial Intelligence: An International Journal, 13(4-5), 519-538. doi: 10.1080/088395199117360 Cassell, J., & Vilhjálmsson, H. (1999). Fully Embodied Conversational Avatars: Making Communicative Behaviors Autonomous. Autonomous Agents and Multi-Agent Systems, 2, 45-64. doi: 10.1023/A:1010027123541 Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjálmsson, H., & Yan, H. (1999). Embodiment in conversational interfaces: Rea. Proceedings of the 1999 SIGCHI Conference on Human Factors in Computing Systems: The CHI is the Limit (pp. 520-527). New York, NY: ACM. doi: 10.1145/302979.303150 Cassell, J., Bickmore, T., Campbell, L., Vilhjalrnsson, H., & Yan, H. (2000). Human Conversation as System Framework: Designing Embodied Conversational Agents. In J. Cassell, J. Sullivan, S. Prevost, & E. Churchill (Eds.), Embodied Conversational Agents (pp. 28-63). Cambridge, MA: MIT Press. Cassell, J., Bickmore, T., Campbell, L., Vilhjálmsson, H., & Yan, H. (2001, March). More Than Just a Pretty Face: Conversational Protocols and the Affordances of Embodiment. Knowledge-Based Systems, 14(1-2), 55-64. doi: 10.1016/S0950- 7051(00)00102-7 Castellano, G., Leite, I., Pereira, A., Martinho, C., Paiva, A., & McOwan, P. W. (2009, September 10-12). It's all in the game: Towards an affect sensitive and context aware game companion. Proceedings of the Third International Conference on Affective Computing and Intelligent Interaction and Workshops, (pp. 1-8). Amsterdam, The Netherlands. doi: 10.1109/ACII.2009.5349558 Chan, T-W., & Baskin, A. (1990). Learning companion systems. In C. Frasson, & G. Gauthier (Eds.), Intelligent Tutoring Systems: At the Crossroads of Artificial Intelligence and Education (pp.6-33). Norwood, NJ: Ablex. Chapman, P. (1997). Models of engagement: Intrinsically motivated interaction with multimedia learning software (Unpublished master's thesis). University of Waterloo, Waterloo, ON, Canada. Chapman, P., Selvarajah, S., & Webster, J. (1999, January 5-8). Engagement in multimedia training systems. Thirty-Second Annual Hawaii International Conference on System Sciences. 1(1), 1-9. Aston Wailea Resort, Wailea Maui, HI: IEEE Computer Society. doi: 10.1109/HICSS.1999.772808 Chen, Y., Naveed, A., & Porzel, R. (2010, November 8-12). Behavior and preference in minimal personality: a study on embodied conversational agents. Proceedings of 12th International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction (ICMI-MLMI). Beijing, China. doi: 10.1145/1891903.1891963 Chittaro, L., Buttussi, F., & Nadalutti, D. (2006, May). MAge-AniM: a system for visual modelling of embodied agent animations and their replay on mobile devices. Proceedings of the Eighth International Conference on Advanced Visual Interfaces (AVI) (pp. 344-351). Venezia, Italy. doi: 10.1145/1133265.1133337 Chou, C. Y., Chan, T. W., & Lin, C. J. (2003, April). Redefining the learning companion: the past, present, and future of educational agents. Computers & Education, 40 (3), 255-269. doi: 10.1016/S0360-1315(02)00130-6 Chou, D., & Chou, A. (2000). A Guide to the Internet Revolution in Banking. Information 200 Systems Management, 17(2), 51-57. doi: 10.1201/1078/43191.17.2.20000301/31227.6 Clarebout, G., Elen, J., Johnson, W., & Shaw, E. (2002). Animated pedagogical agents. An opportunity to be grasped? Journal of Educational Multimedia and Hypermedia, 11 (3), 267-286. Colburn, A. R., Cohen, M. F., & Drucker, S. M. (2000, July 31). The Role of Eye Gaze in Avatar Mediated Conversational Interfaces (Technical Report, No. MSR-TR-2000- 81). Seattle, WA: Microsoft Research, Retrieved March 17, 2010, from http://research.microsoft.com/apps/pubs/default.aspx?id=69800 Compiz. (2007, April 16). Compiz Screenshots. Retrieved March 17, 2010, from Compiz.org: http://www.compiz.org/Home/Screenshots Corsini, R. J. (1999). The Dictionary of Psychology. Philadelphia, PA: Brunner- Routledge. Cosi, P., Fusaro, A., & Tisato, G. (2003). LUCIA a New Italian Talking-Head Based on a Modified Cohen-Massaro’s Labial Coarticulation Model. Proceedings of Eurospeech 2003, (pp. 2269-2272). Geneva, Switzerland. Craig, J. J. (1986). Introduction to robotics: Mechanics and control (2nd ed.). Reading, MA: Addison-Wesley. Crawford, C. (1990). Lessons from Computer Game Design. In B. Laurel (Ed.), The Art of Human-Computer Interface Design (pp. 103-111). New York, NY: Addison- Wesley. Darwin, C. (1998). The expression of the emotions in man and animals (3rd ed.). New York, NY: The New York Academy of Sciences. Day, M. Y., Lu, C. H., Yang, J. T., Chiou, G. F., Ong, C. S., & Hsu, W. L. (2005, July 5- 8). Designing an Ontology-based Intelligent Tutoring Agent with Instant Messaging. Proceedings of the Fifth IEEE International Conference on Advanced Learning Technologies (ICALT) (pp.318-320). Kaoshiung, Taiwan. doi: 10.1109/ICALT.2005.108 DeCarlo, D., Stone, M., Revilla, C., & Venditti, J. (2004). Specifying and animating facial signals for discourse in embodied conversational agents. Computer Animation and Virtual Worlds, 15 (1), 27-38. doi: 10.1002/cav.5 Dehn, D. M., & van Mulken, S. (2000). The Impact of Animated Interface Agents: A Review of Empirical Research. International Journal of Human-Computer Studies, 52(1), 1-22. doi: 10.1006/ijhc.1999.0325 Detlor, B. (2004). Towards knowledge portals: from human issues to intelligent agents. Norwell, MA: Kluwer Academic Publishers. Dickinson College. (2007, February 12). Microsoft's Office Assistant. Retrieved August 3, 2011, from Dickinson College Library & Information Services: http://web.archive.org/web/20070212204645/http://lis.dickinson.edu/Technology/Tr aining/Tutorials/MsOffice/assistant.htm Diener, E., Larsen, R., Levine, S., & Emmons, R. (1985, May). Intensity and frequency: Dimensions underlying positive and negative affect. Journal of Personality and Social Psychology, 48(5), 1253-1265. doi: 10.1037/0022-3514.48.5.1253 Doyle, P. (1999, May). When is a communicative agent a good idea? Proceedings of the Third International Conference on Autonomous Agents. Seattle, WA. Durrenberger, S. (1999). Mad Genius Controversy. In S. Pritzker (Ed.), Encyclopedia of creativity (Vol. 2, pp. 169-178). San Diego, CA: Academic Press.
201 Ekman, P. (1992, July). Are There Basic Emotions? Psychological Review, 99(3), 550- 553. doi: 10.1037/0033-295X.99.3.550 Ekman, P. (1999a). Basic Emotions. In T. Dalgleish, & M. Power (Eds.), Handbook of Cognition and Emotion (pp. 45-60). Hoboken, NJ: John Wiley & Sons. Ekman, P. (1999b). Facial Expressions. In T. Dalgleish, & M. Power (Eds.), Handbook of Cognition and Emotion (pp. 301-320). Hoboken, NJ: John Wiley & Sons. Ekman, P., & Davidson, R. J. (1994). Affective Science: A Research Agenda. In P. Ekman, & R. J. Davidson (Eds.). The nature of emotion: Fundamental questions (pp. 411-434). New York, NY: Oxford University Press. Ekman, P., & Friesen, W. (1976). Measuring facial movement. Journal of Nonverbal Behavior, 1 (1), 56-75. doi: 10.1007/BF01115465 Ekman, P., & Friesen, W. (1977). Facial Action Coding System. Palo Alto, CA: Consulting Psychologist Press. Ekman, P., Friesen, W., & Ancoli, S. (1980). Facial signs of emotional experience. Journal of Personality and Social Psychology, 39(6), 1125-1134. doi: 10.1037/h0077722 Ekman, P., Friesen, W., & Ellsworth, P. (1972). Emotion and the Human Face: Guidelines for Research and an Integration of Findings. New York, NY: Pergamon Press. Elliott, C., Rickel, J., & Lester, J. (1997). Integrating Affective Computing Into Animated Tutoring Agents. Proceedings of the IJCAI Workshop on Animated Interface Agents: Making Them Intelligent (pp. 113-121). Nagoya, Japan. Erickson, T. D. (1990). Working with Interface Metaphors. In B. Laurel (Ed.), The Art of Human-Computer Interface Design (pp. 65-73). New York, NY: Addison-Wesley. Erickson, T. (1997). Designing agents as if people mattered. In J. M. Bradshaw (Ed.), Software Agents (pp. 79-96). Menlo Park, CA: AAAI Press. Estrada, C. A., Isen, A. M., & Young, M. J. (1997, October). Positive Affect Facilitates Integration of Information and Decreases Anchoring in Reasoning among Physicians. Organizational Behavior and Human Decision Processes, 72(1) 117- 135. doi: 10.1006/obhd.1997.2734 Evans, J. (1994, September 12-16). HelpMate: an autonomous mobile robot courier for hospitals. Proceedings of the IEEE/RSJ/GI International Conference on Intelligent Robots and Systems, (pp. 1695 - 1700). Munich, Germany. doi: 10.1109/IROS.1994.407629 Facebook. (2010). Statistics [Fact Sheet]. Retrieved March17, 2010, from Facebook: http://www.facebook.com/press/info.php?statistics Fenske, M., & Raymond, J. (2006, December). Affective Influences of Selective Attention. Psychological Science, 15 (6), 312-316. doi: 10.1111/j.1467- 8721.2006.00459.x Fong, S., Sun, A., & Wong, K. K. (2001). Price Watcher Agent for E-commerce. Proceedings of the Second Asia-Pacific Conference on Intelligent Agent Technology (IAT) (pp. 294-299). Maebashi, Japan. doi:10.1142/9789812811042_0040 Forgas, J. P. (2005, January). Affective influences on interpersonal behaviour: Towards understanding the role of affect in everyday interactions. In J.P Forgas (Chair), Hearts and minds: Affective influences on social cognition and behaviour. Symposium conducted at the Eighth Conference on Social Psychology. University of New South Wales, Sydney. Retrieved March 17, 2010, from Sydney Symposium:
202 http://www.sydneysymposium.unsw.edu.au/2005/Papers/2005ForgasAffIntBehNOR EFS.doc Forgas, J. P. (2002). Feeling and Doing: Affective Influences on Interpersonal Behavior. Psychological Inquiry, 13(1), 1-28. doi: 10.1207/S15327965PLI1301_01 Forlizzi, J., Zimmerman, J., Mancuso, V., & Kwak, S. (2007). How interface agents affect interaction between humans and computers. Proceedings of the 2007 Conference on Designing Pleasurable Products and Interfaces (DPPI) (pp. 209 - 221). Helsinki, Finland. doi: 10.1145/1314161.1314180 Fox, J., & Bailenson, J. (2009, January). Virtual Self-Modeling: The Effects of Vicarious Reinforcement and Identification on Exercise Behaviors. Media Psychology, 12 (1), 1-25. Fridlund, A., Ekman, P., & Oster, H. (1987). Facial expression of emotion. In A. W. Siegman, & S. Feldstein (Eds.), Nonverbal behavior and communication (pp. 143- 224). Hillsdale, NJ: Lawrence Erlbaum Associates. Fryer, D., & Marshall, J. (1979, April). The Motives of Jacques de Vaucanson. Technology and Culture, 20(2), 257-269. Gard, T. (2000, June 20). Building Character. Retrieved from Gamasutra: http://www.gamecareerguide.com/features/20000720/gard_02.htm Gardner, R. (2004). Attitude/Motivation Test Battery: International AMTB Research Project (Research Report). London, ON, Canada: University of Western Ontario. Retrieved March 17, 2010, from the University of Western Ontario: http://publish.uwo.ca/~gardner/docs/englishamtb.pdf Gardner, R., & MacIntyre, P. (1993). A student's contributions to second-language learning. Part II: Affective variables. Language Teaching, 26, 1-11. doi: 10.1017/S0261444800000045 Gentner, D., & Nielsen, J. (1996, August). The Anti-Mac interface. Communications of the ACM, 39(8), 70-82. doi: 10.1145/232014.232032 Gerard. (2008, August 20). Wolfgang Von Kempelen's Speaking Machine [Web blog post] Retrieved March 17, 2010, from The presurfer: http://presurfer.blogspot.com/2008/08/wolfgang-von-kempelens-speaking- machine.html Geven, A., Schrammel, J., & Tschelig, M. (2006, October 14-18). Interacting with Embodied Agents that can see: How Vision-Enabled Agents can assist in Spatial Tasks. Proceedings of the Fourth Nordic Conference on Human-Computer Interaction: Changing Roles (pp. 135-144). New York, NY. doi: 10.1145/1182475.1182490 Goh, O. S., Fung, C. C., Wong, K. W., & Depickere, A. (2006, December 18-22). An Embodied Conversational Agent for Intelligent Web Interaction on Pandemic Crisis Communication. In IEEE/WIC/ACM International Conferences on Web Intelligences and Intelligent Agent Technology (pp. 397-400). Hong Kong. doi: 10.1109/WI- IATW.2006.37 Gohring, N. (2009, October 1). IPhone, Android Boost Mobile Web Usage. Retrieved March 17, 2010, from PCWorld: http://www.pcworld.com/article/172932/iphone_android_boost_mobile_web_usage. html Gong, L. (2008). How social is social responses to computers? The function of the degree of anthropomorphism in computer representations. Computers in Human Behavior, 24 (4), 1494-1509. doi: 10.1016/j.chb.2007.05.007 203 Gong, L., & Lai, J. (2003). To Mix or Not to Mix Synthetic Speech and Human Speech? Contrasting Impact on Judge-Rated Task Performance versus Self-Rated Performance and Attitudinal Responses. International Journal of Speech Technology, 6, 123-131. doi: 10.1023/A:1022382413579 Gong, L., & Nass, C. (2007). When a Talking-Face Computer Agent is Half-Human and Half-Humanoid: Human Identity and Consistency Preference. Human Communication Research, 33, 163-193. doi: 10.1111/j.1468-2958.2007.00295.x Goodman, D., & Keene, R. (1997). Man versus Machine: Kasparov versus Deep Blue. Cambridge, UK: H3 Publications. Google. (2008, October 3). More Google products. Retrieved March 17, 2010, from Google: http://www.google.com.au/intl/en/options/ Grudin, J. (1990). The computer reaches out: the historical continuity of interface design. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Empowering People (pp. 261-268). Seattle, WA: ACM. doi: 10.1145/97243.97284 Gulz, A., & Haake, M. (2006). Design of animated pedagogical agents - A look at their look. International Journal of Human-Computer Studies, 64 (4), 322-339. doi: 10.1016/j.ijhcs.2005.08.006 Haake, M. (2009). Embodied Pedagogical Agents: From Visual Impact to Pedagogical Implications (Doctoral Thesis, Lund University, Scania, Sweden). Retrieved March 17, 2010, from Lund University: http://swepub.kb.se/bib/swepub:oai:lup.lub.lu.se:1389720?tab2=abs&language=en Hakansson, J. (1990). Lessons Learned from Kids: One Developer's Point of View. In B. Laurel (Ed.), The Art of Human-Computer Interface Design (pp. 123-130). New York, NY: Addison-Wesley. Hall, L., Woods, S., Dautenhahn, K., Sobral, D., Paiva, A., Wolke, D., & Newall, L. (2004). Designing Empathic Agents: Adults Versus Kids. Lecture Notes in Computer Science, 3220/2004, 175-196. doi: 10.1007/978-3-540-30139-4_57 Handke, J. (1995). The Structure of the Lexicon: Human versus Machine. Berlin: Mouton de Gruyter. Hanger, J., & Ekman, P. (1983). The Inner and Outer Meanings of Facial Expressions. In J. T. Cacioppo, & R. E. Petty (Eds.), Social Psychophysiology: A Sourcebook (pp. 287-305). New York, NY: Guilford Press. Hareli, S., & Rafaeli, A. (2008). Emotion cycles: On the social influence of emotion in organizations. Research in Organizational Behavior, 28, 35-59. Hartmann, B., Mancini, M., & Pelachaud, C. (2006, Feburary 15). Implementing Expressive Gesture Synthesis for Embodied Conversational Agents. Proceedings of the Sixth International Gesture Workshop: Gesture in Human-Computer Interaction and Simulation (pp. 188-199). Berder Island, France. doi: 10.1007/11678816_22 Haynes, R. (2003, July). From Alchemy to Artificial Intelligence: Stereotypes of the Scientist in Western Literature. Public Understanding of Science, 12(3), 243-253. doi: 10.1177/0963662503123003 Heylen, D., van Es, I., Nijholt, A., & Dijk, B. (2002, June 28-29). Experimenting with the Gaze of a Conversational Agent. In J. van Kuppevelt, L. Dybkjaer, & N. Bernsen (Eds.), Proceedings of the International CLASS Workshop on Natural, Intelligent and Effective Interaction with Multimodal Dialogue System (pp.93-100). Copenhagen, Denmark. Heinze, C., Smith, B., & Cross, M. (1998). Thinking quickly: Agents for modeling air warfare. Lecture Notes in Computer Science, 1502/1998, 47-58. doi: 204 10.1007/BFb0095040 Hendrick, S. (1988). A Generic Measure of Relationship Satisfaction. Journal of Marriage and the Family, 50 (1), 93-98. Hietala, P., & Niemirepo, T. (1998). The Competence of Learning Companion Agents. International Journal of Artificial Intelligence in Education (IJAIED), 9(2), 178- 192. Hilgard, E. (1980, April). The trilogy of mind: cognition, affection, and conation. Journal of the History of the Behavioral Sciences, 16(2),107-117. doi: 10.1002/1520- 6696(198004)16:2<107::AID-JHBS2300160202>3.0.CO;2-Y Hoffmann, L., Krämer, N., Lam-chi, A., & Kopp, S. (2009). Media Equation Revisited: Do Users Show Polite Reactions towards an Embodied Agent? Lecture Notes in Computer Science, 5773,159-165. doi:10.1007/978-3-642-04380-2_19 Hoorn, J., & Konijn, E. (2003). Perceiving and experiencing fictional characters: An integrative account. Japanese Psychological Research, 45 (4), 221-239. doi: 10.1111/1468-5884.00225 Horswill, I. (1993, July 11-15). Polly: A Vision-Based Artificial Agent. Proceedings of the Eleventh National Conference on Artificial Intelligence (pp. 824-829). Washington Convention Center, Washington DC. Incesoft. (2008). Xiaoi Robot - Number one intelligent web robot in China. (Incesoft Co.,Ltd.) Retrieved October 19, 2008 from Xiaoi Robot: http://www.xiaoi.com/ Inter-IKEA Systems B.V. (2011). Ask Anna. Retrieved March 17, 2010, from IKEA: http://193.108.42.79/ikea-au/cgi-bin/ikea-au.cgi Isen, A. M. (1993). Positive affect and decision making. In M. Lewis, & J. M. Haviland (Eds.), Handbook of Emotions (pp. 261-277). New York, NY: The Guilford Press. Iwatani, T., Funaki, S., Kai, T. (Designers), & Namco (Developer). (1980). Pac-Man [Arcade Game]. Jarnstrom, J., & Kiljander, H. (2003). User Interface Styles. In C. Lindholm, T. Keinonen, & H. Kiljander (Eds.), Mobile Usability: How Nokia Changed the Face of the Mobile Phone (pp. 15-46). New York, NY: McGraw-Hill. Jennings, N., & Wooldridge, M. (1998). Applications of Intelligent Agents. In N. Jennings, & M. Wooldridge, (Eds.). Agent technology: foundations, applications, and markets (pp. 3-28). Heidelberg: Springer-Verlag. Johnson, W., & Rickel, J. (1997). Steve: an animated pedagogical agent for procedural training in virtual environments. ACM SIGART Bulletin, 8 (1-4), 16-21. doi: 10.1145/272874.272877 Johnson, W., LaBore, C., & Chiu, Y. (2004). A pedagogical agent for psychosocial intervention on a handheld computer. Proceedings of the AAAI Fall Symposium on Health Dialogue System (pp. 64-70). Arlington, VA. Kadous, M., & Sammut, C. (2004). In CA: A Mobile Conversational Agent. In Proceedings of the Eighth Pacific Rim International Conference on Artificial Intelligence (PRICAI): Trends in Artificial Intelligence (pp. 644-653). Auckland, New Zealand: Springer. doi: 10.1007/978-3-540-28633-2_68 Kalick, S. (1988, November). Physical attractiveness as a status cue. Journal of Experimental Social Psychology, 24(6), 469-489. doi: 10.1016/0022- 1031(88)90047-9 Kaufmann, G., & Vosburg, S. K. (2002). The Effects of Mood on Early and Late Idea Production. Creativity Research Journal, 14 (3-4), 317-330. doi: 205 10.1207/S15326934CRJ1434_3 Kay, A. (1990). User Interface: A Personal View. In B. Laurel, The Art of Human- Computer Interface Design (pp. 191-207). New York, NY: Addison-Wesley. Kearsley, G. (1993). Intelligent agents and instructional systems: implications of a new paradigm. Journal of Artificial Intelligence in Education, 4(4), 295-304. Kelley, H. H. (1983). Epilogue: An Essential Science. In H. Kelley, E. Berscheid, A. Christensen, H. Harvey, T. L. Huston, G. Levinger, … D. Peterson, (Eds.), Close Relationships (pp. 486-503). New York, NY: Freeman. Kelly, V. (2009). A primer of affect psychology (Research Report). Retrieved March 17, 2010, from The Tomkins Institute: http://www.tomkins.org/uploads/Primer_of_Affect_Psychology.pdf Keltner, D., & Ekman, P. (2003). Introduction: Expression of Emotion. In R. J. Davidson, K. R. Scherer, & H. H. Goldsmith (Eds.), Handbook of Affective Sciences (pp. 411- 414). New York, NY: Oxford University Press. King, W., & Ohya, J. (1996, April 14-18). The representation of agents: anthropomorphism, agency, and intelligence. Proceedings of the Conference on Human Factors in Computing Systems: Common Ground (pp. 289-290). Vancouver, BC, Canada. Kleinginna, P., & Kleinginna, A. (1985, April). Cognition and affect: A reply to Lazarus and Zajonc. American Psychologist, 40(4), 470-471. doi: 10.1037/0003- 066X.40.4.470 Koda, T. (2007). Cross-Cultural Study of Avatars’ Facial Expressions and Design Considerations Within Asian Countries. Lecture Notes in Computer Science, 4568/2007, 207-220. doi: 10.1007/978-3-540-74000-1_16 Koda, T., Ishida, T., Rehm, M., & André, E. (2009). Avatar culture: cross-cultural evaluations of avatar facial expressions. AI & Society, 24 (3), 237-250. doi: 10.1007/s00146-009-0214-5 Koda, T., & Maes, P. (1996, November 11- 14). Agents with Faces: the Effect of Personification of Agents. Proceedings of the Fifth IEEE International Workshop: Robot and Human Communication (pp. 189-194). Tsukuba, Japan. doi: 10.1109/ROMAN.1996.568812 Konijn, E. A., & Hoorn, J. F. (2005). Some Like It Bad: Testing a Model for Perceiving and Experiencing Fictional Characters. Media Psychology, 7 (2), 107-144. Kopp, S., Gesellensetter, L., Krämer, N., & Wachsmuth, I. (2005). A Conversational Agent as Museum Guide – Design and Evaluation of a Real-World Application. Lecture Notes in Computer Science, 3661/2005, 329-343. doi: 10.1007/11550617_28 Kort, B., Reilly, R., & Picard, R. (2001, August 6-8). An Affective Model of Interplay between Emotions and Learning: Reengineering Educational Pedagogy-Building a Learning Companion. Conference Paper Presented at the International Conference of Advanced Learning Technologies. Madison, WI. doi: 10.1109/ICALT.2001.943850 Kosofsky-Sedgwick, E., Frank, A., & Alexander, I. E. (1995). Shame and Its Sisters: A Silvan Tomkins Reader. Durham, NC: Duke University Press. Krämer, N., & Bente, G. (2010). Personalizing e-Learning. The Social Effects of Pedagogical Agents. Educational Psychology Review, 22 (1), 71-87. Kring, A., & Moran, E. (2008). Emotional Response Deficits in Schizophrenia: Insights From Affective Science. Schizophrenic Bulletin, 34(5), 819-834. doi:
206 10.1093/schbul/sbn071 Kubrick, S., Clarke, A. (Writers), & Kubrick, S. (Director). (1968). 2001: A Space Odyssey [Motion Picture]. US: MGM. Kurtenbach, G., & Hulteen, E. A. (1990). Gestures in Human-Computer Communication. In B. Laurel, The Art of Human-Computer Interface Design (pp. 309-317). New York, NY: Addison-Wesley. Lager, M. (2008, August). Vendors and Users Both Now Have an Unprecedented Degree of Control Over What a CRM Application can Look Like. How Should that Power be Used?. Customer Relationship Management, 28-33. Larsen, R., & Fredrickson, B. (1999). Measurement issues in emotion research. In D. Kahneman, E. Diener, & N. Schwarz, (Eds.), Well-being: The foundations of hedonic psychology (pp. 40-60). New York, NY: Russell Sage Foundation. Laurel, B. (1990). Interface Agent: Metaphors with Character. In B. Laurel (Ed.), The Art of Human-Computer Interface Design (pp. 355-365). New York, NY: Addison- Wesley. Laurel, B. (1991). Computers as theatre. Boston, MA: Addison-Wesley. Laurel, B., & Mountford, S. J. (1990). Introduction. In B. Laurel (Ed.), The Art of Human-Computer Interface Design (pp. xi-xvi). New York, NY: Addison-Wesley. Lazarus, R. (1982). Thoughts on the relations between emotion and cognition. American Psychologist, 37(10), 1019-1024. doi: 10.1037/0003-066X.37.9.1019 Lee, E-J., & Nass, C. (2002). Experimental tests of normative group influence and representation effects in computer-mediated communication: When interacting via computers differs from interacting with computers. Human Communication Research, 28(3), 349-381. doi: 10.1111/j.1468-2958.2002.tb00812.x Lee, K. M., & Nass, C. (2003, April 5-10). Designing Social Presence of Social Actors in Human Computer Interaction. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 289-296). 5 (1), Fort Lauderdale, FL. doi: 10.1145/642611.642662 Lester, J., Branting, K., & Mott, B. (2004). Conversational Agents. In M. P. Singh, Practical Handbook of Internet Computing (pp. 220-240). Boca Raton, FL: Chapman & Hall/CRC Press. Lester, J. C., Converse, S. A., Kahler, S. E., Barlow, T., Stone, B. A., & Bhogal, R. S. (1997a). The persona effect: affective impact of animated pedagogical agents. Proceedings of the 1997 SIGCHI Conference on Human Factors in Computing Systems (pp. 359-366). Atlanta, GA: ACM. doi: 10.1145/258549.258797 Lester, J. C., & Stone, B. A. (1997). Increasing believability in animated pedagogical agents. Proceedings of the First International Conference on Autonomous Agents, (pp. 16-21). Marina del Rey, CA. doi: 10.1145/267658.269943 Lester, J., Stone, B., Converse, S., Kahler, S., & Barlow, S. (1997b). Animated Pedagogical Agents and Problem-Solving Effectiveness: A Large-Scale Empirical Evaluation. Proceedings of the Eighth World Conference on Artificial Intelligence in Education (pp. 23-30). Kobe, Japan: IOS Press. Lindholm, C., & Keinonen, T. (2003). Managing the Design of User Interface. In C. Lindholm, T. Keinonen, & H. Kiljander (Eds.), Mobile Usability: How Nokia Changed the Face of the Mobile Phone (pp. 139-156). New York, NY: McGraw- Hill. Louwerse, M. M., Graesser, A. C., Lu, S., & Mitchell, H. H. (2005, September). Social
207 cues in animated conversational agents. Applied Cognitive Psychology, 19(6), 693- 704. doi: 10.1002/acp.1117 Lu, C-H., Chiou, G-F., Day, M-Y., Ong, C-S., & Hsu, W-L. (2006). Using Instant Messaging to Provide an Intelligent Learning Environment. Intelligent Tutoring Systems, Lecture Notes in Computer Science, 4053/2006, 575-583. doi: 10.1007/11774303_57 Lucas, G. (Writer), & Lucas, G. (Director). (1977). Star Wars Episode IV: A New Hope [Motion Picture]. US: Lucasfilm. Luger, G. F., & Stubblefield, W. A. (1998). Artificial Intelligence: structures and strategies for complex problem solving (3rd ed.). Boston, MA: Addison-Wesley. Maes, P. (1994). Agents That Reduce Work and Information Overload. Communications of the ACM, 37 (7), 30-40. doi: 10.1145/176789.176792 Magee, J. R. (2008, July 16). Screen Works. Retrieved August 13, 2011, from Institutional Investor: http://www.institutionalinvestor.com/Article/1974065/Screen- Works.html?ArticleId=1974065 Mahoney, M. (1988). The History of Computing in the History of Technology. IEEE Annals of the History of Computing, 10, 113-125. Mandel, T. (1997). The Elements of User Interface Design. New York, NY: John Wiley & Sons. Manning, C. D., & Schütze, H. (2000). Foundations of Statistical Natural Language Processing. Cambridge, MA: MIT Press. Mark, L., Davis, L., Dow, T., Godfrey, W. (Producers), & Proyas, A. (Director). (2004). I, Robot [Motion Picture]. US: Davis Entertainment. Martinho, C., Machado, I., & Paiva, A. (2000). A Cognitive Approach to Affective User Modeling. Lecture Notes in Computer Science, 1814/2000, 64-75. doi: 10.1007/10720296_6 Mawston, N. (2011, January 28). Global Handset Shipments Reach 400 Million Units in Q4 2010 [Fact Sheet]. Retrieved August 13, 2011, from Strategy Analytics: http://www.strategyanalytics.com/default.aspx?mod=reportabstractviewer&a0=6052 McCorduck, P. (2004). Machines Who Think: A Personal Inquiry into the History and Prospects of Artificial Intelligence (2nd ed.). Natick, MA: A K Peters/CRC Press. McIntyre, G., & Göcke, R. (2008). The Composite Sensing of Affect. Lecture Notes in Computer Science, 4868/2008, 104-115. doi: 10.1007/978-3-540-85099-1_9 Microsoft. (2008). Windows Flip and Flip 3D [Brochure]. Retrieved March 17, 2010, from Microsoft: http://www.microsoft.com/windows/products/windowsvista/features/details/flip3D. mspx Microsoft. (2011). What happened to the Office Assistant? - Support - Office.com [Press Release]. Retrieved August 3, 2011, from Office.com: http://office.microsoft.com/en-us/help/what-happened-to-the-office-assistant- HA010047975.aspx Minsky, M. (1985). The Society of Mind. New York, NY: Simon and Schuster. Mishira, P. (2006, January). Affective Feedback from Computers and its Effect on Perceived Ability and Affect: A Test of the Computers as Social Actor Hypothesis. Educational Multimedia and Hypermedia, 15 (1), 107-131. Miura, A., & Matsumura, N. (2009). Social Intelligence Design: a Junction between
208 Engineering and Social Sciences. AI & Society, 23(2), 139-145. doi: 10.1007/s00146-007-0139-9 Molen, G. R., Curtis, B., Parkes, W. F., de Bont, J., Doven, M. (Producers), & Spielberg, S. (Director). (2002). Minority Report [Motion Picture]. US: Amblin Entertainment. Moore, J., & Swartout, W. (1991). A reactive approach to explanation: Taking the user’s feedback into account. In C. Paris, W. Swartout, & W. Mann, (Eds.), Natural language generation in artificial intelligence and computational linguistics (pp. 1- 48). Norwell, MA: Kluwer Academic Publishers. Moreno, R. (2001). Software agents in multimedia: An experimental study of their contributions to students' learning. Proceedings of the International Conference of Human-Computer Interaction (pp. 275-277). Tokyo, Japan. Mahwah, NJ: Lawrence Erlbaum Associates. Moundridou, M., & Virvou, M. (2002). Evaluating the persona effect of an interface agent in a tutoring system. Journal of Computer Assisted Learning, 18(3), 253-261. doi: 10.1046/j.0266-4909.2001.00237.x Mountford, S. J. (1990a). Technique and Technology: Introduction. In B. Laurel (Ed.), The Art of Human-Computer Interface Design (pp. 247-250). New York, NY: Addison-Wesley. Mountford, S. J. (1990b). Tools and Techniques for Creative Design. In B. Laurel (Ed.), The Art of Human-Computer Interface Design (pp. 17-30). New York, NY: Addison- Wesley. Mountford, S. J., & Gaver, W. W. (1990). Talking and Listening to Computers. In B. Laurel (Ed.), The Art of Human-Computer Interface Design (pp. 319-334). New York, NY: Addison-Wesley. Mumm, J., & Mutlu, B. (2011, September). Designing motivational agents: The role of praise, social comparison, and embodiment in computer feedback. Computer in Human Behaviour, 27(5), 1643-1650. doi: 10.1016/j.chb.2011.02.002 Nass, C., Isbister, K., & Lee, E-J. (2000). Truth is beauty: researching embodied conversational agents. In J. Cassell, J. Sullivan, S. Prevost, & E. Churchill (Eds.), Embodied Conversational Agents (pp. 374- 402). Cambridge, MA: MIT Press. Nass, C., & Moon, Y. (2000). Machines and mindlessness: Social responses to computers. Journal of Social Issues, 56(1), 81-103. Nass, C., Moon, Y., Morkes, J., Kim, E-Y., & Fogg, B. (1997). Computers Are Social Actors: A Review of Current Research. In B. Friedman (Ed.), Moral and Ethical Issues in Human-Computer Interaction (pp. 137-162). Stanford, CA: CSLI Press. Nass, C., Steuer, J., & Tauber, E. R. (1994). Computers are social actors. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating Interdependence (pp. 72-78). Boston, MA. doi: 10.1145/191666.191703 Nathanson, D. (2008). Prologue: affect imagery consciousness. Retrieved March 17, 2010, from The Tomkins Institute: www.tomkins.org/uploads/PROLOGUE.pdf Negnevitsky, M. (2002). Artificial Intelligence: A Guide to Intelligent Systems. Edinburgh Gate, Harlow, UK: Pearson Education. Negroponte, N. (1990). Hospital Corners. In B. Laurel (Ed.), The Art of Human- Computer Interface Design (pp. 347-353). New York, NY: Addison-Wesley. Negroponte, N. (1995). Being Digital. New York, NY: Alfred A. Knopf Publications. Neuman, W., Marcus, G., Grigler, A., & MacKuen, M. (2007). Theorizing Affect's Effects. In W. Neuman, G. Marcus, A. Grigler, & M. Mackuen (Eds.), The affect 209 effect: dynamics of emotion in political thinking and behavior (pp. 1-20). Chicago, IL: University of Chicago Press. Nilsson, N. J. (1998). Artificial Intelligence: A New Synthesis. San Francisco, CA: Morgan Kaufmann Publishers. Nishikado, T. (Designer), & Taito (Developer). (1978). Space Invaders [Arcade Video Game]. Norman, D. A. (1990). Why Interface Don't Work. In B. Laurel (Ed.), The Art of Human- Computer Interface Design (pp. 209-219). New York, NY: Addison-Wesley. Norman, D. (2002). Emotion & design: attractive things work better. Interactions, 9 (4), 36-42. doi: 10.1145/543434.543435 Nowak, K. L., & Rauh, C. (2005). The Influence of the Avatar on Online Perceptions of Anthropomorphism, Androgyny, Credibility, Homophily, and Attraction. Journal of Computer-Mediated Communication, 11 (1), 153-178. doi: 10.1111/j.1083- 6101.2006.tb00308.x Oatley, K. (2004). Emotions: a brief history. Malden, MA: Blackwell Publishing. O'Brien, H. L., & Toms, E. G. (2008). What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 59 (6), 938-955. doi: 10.1002/asi.20801 Opera Software. (2010, August). State of the Mobile Web, August 2010 [Fact Sheet]. Retrieved August 3, 2011, from Opera Software: http://www.opera.com/smw/2010/08/ Oren, T., Salomon, G., Kreitman, K., & Don, A. (1990). Guide: Characterizing the Interface. In B. Laurel, The Art of Human-Computer Interface Design (pp. 367-381). New York, NY: Addison-Wesley. Ortony, A. (2002). On making believable emotional agents believable. In R. Trappl, P. Petta, & S. Payr (Eds.), Emotions in Humans and Artifacts (pp.189-212). Cambridge, MA: MIT Press. Ortony, A., Clore, G., & Collins, A. (1990). The cognitive structure of emotions (Paperback ed.). Cambridge, UK: Cambridge University Press. Oviatt, S., & Adams, B. (2000). Designing and Evaluating Conversational Interfaces with Animated Characters. In J. Cassell, J. Sullivan, S. Prevost, & E. Churchill (Eds.), Embodied Conversational Agents (pp. 319-345). Cambridge, MA: MIT Press. Oxford University Press. (2011a). Definition of affect from Oxford Dictionaries Online. Retrieved August 3, 2011, from Oxford Dictionaries: http://oxforddictionaries.com/ Oxford University Press. (2011b). Definition of affective from Oxford Dictionaries Online. Retrieved August 3, 2011, from Oxford Dictionaries: http://oxforddictionaries.com/ Partala, T., Surakka, V., & Lahti, J. (2004). Affective Effects of Agent Proximity in Conversational Systems. Proceedings of the Third Nordic Conference on Human- Computer Interaction (pp. 353-356). Helsinki, Finland: ACM International Conference Proceeding Series. doi: 10.1145/1028014.1028070 Paul Ekman Group. (2011). About Ekman [Fact Sheet]. Retrieved August 3, 2011, from the Paul Ekman Group: http://www.paulekman.com/about-ekman/ Pelachaud, C. (2005). Multimodal Expressive Embodied Conversational Agents. Proceedings of the Thirteenth Annual ACM International Conference on Multimedia (pp. 683 - 689). Hilton Hotel, Singapore: ACM. doi: 10.1145/1101149.1101301
210 Pelachaud, C., & Poggi, I. (1998). Facial performative in a conversational system. Proceedings of the 1998 Workshop on Embodied Conversational Characters (WECC). Lake Tahoe, CA. Pelachaud, C., & Poggi, I. (2002, December 16). Subtleties of facial expressions in embodied agents. The Journal of Visualization and Computer Animation, 13(5), 301- 312. doi: 10.1002/vis.299 Peters, C., Asteriadis, S., Karpouzis, K., & de Sevin, E. (2008). Towards a real-time gaze- based shared attention for a virtual agent. Proceedings of the ACM International Conference on Multimodal Interfaces (ICMI): Workshop on Affective Interaction in Natural Environments (AFFINE). Chania, Crete, Greece. Petty, R. E., & Wegener, D. T. (1998). Attitude change: Multiple roles for persuasion variables. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.), Handbook of social psychology (4th ed., pp. 323-390). New York, NY: McGraw–Hill. Phan, K., Wager, T., Taylor, S., & Liberzon, I. (2002, June). Functional Neuroanatomy of Emotion: A Meta-Analysis of Emotion Activation Studies in PET and fMRI. NeuroImage, 16(2), 331-348. doi: 10.1006/nimg.2002.1087 Phillips, L. (2010). Boomers and Mobile Usage (Research Report). Retrieved March 17, 2010, from eMarketer: http://www.emarketer.com/ Picard, R. W. (1995). Affective Computing. Cambridge, MA: MIT Press. Picard, R. W. (1997). Affective Computing. Cambridge, MA: MIT Press. Picard, R. W. (2000). Affective Computing. Cambridge, MA: MIT Press. Picard, R. W. (2002). What does it mean for a computer to “have” emotions. In R. Trappl, P. Petta, & S. Pyar (Eds.), Emotions in Human and Artefacts (pp. 213-236). Cambridge, MA: MIT Press. Piza, H. I., Zuniga, F., & Ramos, F. F. (2004, November 18-20). A platform to design and run dynamic virtual environments. Proceedings of the 2004 International Conference on Cyberworlds (pp. 78-85). Tokyo, Japan: IEEE Computer Society. doi: 10.1109/CW.2004.9 Poggi, I. (2007). Mind, hands, face and body: A goal and belief view of multimodal communication. Berlin: Joachim Weidler. Poggi, I., & Pelachaud, C. (2000). Performative Facial Expressions in Animated Faces. In J. Cassell, J. Sullivan, S. Prevost, & E. Churchill (Eds.), Embodied Conversational Agents (pp. 155-188). Cambridge, MA: MIT Press. Rajan, S., Craig, S., Gholson, B., Person, N., Graesser, A. (2001). AutoTutor: Incorporating backchannel feedback and other human-like conversational behaviours into an intelligent tutoring system. International Journal of Speech Technology, 4(2), 117-126. doi: 10.1023/A:1017319110294 Raskin, J. (2000). The Human Interface: New Directions for Designing Interacting Systems. New York, NY: Addison-Wesley. Raymond, E. S., & Landley, R. W. (2004, April 18). The first GUIs (Research Report). Retrieved March 17, 2010, from The Art of Unix Usability: http://www.catb.org/~esr/writings/taouu/html/ch02s05.html Reeves, B., & Nass, C. (1996). The Media Equation: How People Treat Computer, Television, and New Media Like Real People and Places. Cambridge, MA: Cambridge University Press. Reilly, W. (1996). Believable Social and Emotional Agents (Doctoral dissertation, Carnegie Mellon University, Pittsburgh, PA). Retrieved March 17, 2010, from: 211 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.52.4599&rep=rep1&type =pdf Reimer, J. (2005, May 5). A History of the GUI (Research Report). Retrieved March 17, 2010, from Ars Technica: http://arstechnica.com/articles/paedia/gui.ars/2 Reynolds, C., & Picard, R. W. (2001, August). Designing for Affective Interactions. Paper Presented at the Ninth International Conference on Human-Computer Interaction. New Orleans, LA. Rickenberg, R., & Reeves, B. (2000). The effects of animated characters on anxiety, task performance, and evaluations of user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 49-56). New York, NY: ACM. doi: 10.1145/332040.332406 Rosenberg, E., & Ekman, P. (1994). Coherence between expressive and experiential system in emotion. Cognition and Emotion, 8(3), 201-229. doi: 10.1080/02699939408408938 Rosenberg, E., & Ekman, P. (1999). Facial Expression and Emotion. Neuroscience year: Supplement 3 to the Encyclopedia of Neuroscience, 51-52. Rosenberg-Kima, R., Baylor, A., Ashby-Plant, E., & Doerr, C. (2008, September 17). Interface agents as social models for female students: The effects of agent visual presence and appearance on female students’ attitudes and beliefs. Computers in Human Behavior, 24 (6), 2741-2756. doi: 10.1016/j.chb.2008.03.017 Russell, J. (1980, December). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161-1178. doi: 10.1037/h0077714 Russell, J. (2003, January). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145-172. doi: 10.1037/0033-295X.110.1.145 Russell, S. J., & Norvig, P. (2003). Artificial intelligence: A modern approach (2nd ed.). Upper Saddle River, NJ: Prentice Hall. Ryokai, K., Vaucelle, C., & Cassell, J. (2003, June). Virtual peers as partners in storytelling and literacy learning. Journal of Computer Assisted Learning, 19(2), 195-208. doi: 10.1046/j.0266-4909.2003.00020.x Sakagami, Y., Watanabe, R., Aoyama, C., Matsunaga, S., Higaki, N., & Fujimura, K. (2002). The intelligent ASIMO: system overview and integration. Paper Presented at the International Conference on Intelligent Robots and Systems, 3, 2478-2483. doi: 10.1109/IRDS.2002.1041641 Sarrafzadeh, A., Alexander, S., Dadgostar, F., Fan, C., & Bigdeli, A. (2008, July). “How do you know that I don’t understand?” A look at the future of intelligent tutoring systems. Computers in Human Behavior, 24(4), 1342-1363. doi: 10.1016/j.chb.2007.07.008 Sasaki, H., Kuroda, T., Manabe, Y., & Chihara, K. (1999). HIT-Wear : A menu system superimposing on a human hand for wearable computers. Proceedings of the Ninth International Conference on Artificial Reality and Telexistence (ICAT), (pp. 146- 153). Tokyo, Japan. Schaumburg, H. (2001). Computer as Tools or as Social Actors? The Users' Perspective on Anthropomorphic Agents. International Journal on Intelligent Cooperative Information System, 10(1-2), 217-234. doi:10.1142/S0218843001000321 Schilbach, L., Wohlschlaeger, A., Kraemer, N., Newen, A., Shah, J., Fink R., & Vogeley, K. (2006, September 19). Being with virtual others: Neural correlates of social interaction. Neuropsychologia, 44(5), 718-730. doi: 10.1016/j.neuropsychologia.2005.07.017 212 Schumann, J. (1975, December). Affective Factors and the Problem of Age in Second Language Acquisition. Language Learning, 25(2), 209-235. doi: 10.1111/j.1467- 1770.1975.tb00242.x Schwarz, N., & Clore, G. L. (1983, September). Mood, misattribution, and judgments of well-being: Informative and directive functions of affective states. Journal of Personality and Social Psychology, 45(3), 513-523. doi: 10.1037/0022- 3514.45.3.513 Scott. (2010, March 17). Guidelines for Adapting Web Site Designs for Mobile Delivery. Retrieved March 17, 2010, from Appnovation Technologies: http://www.appnovation.com/guidelines-adapting-designs-mobile-delivery Sculley, J. (1989, September). The relationship between business and higher education: A perspective on the 21st century. Communications of the ACM, 32(9), 1056-1061. Sharp, H. (2007). Interaction Design - beyond human-computer interaction (2nd ed.). Hoboken, NJ: John Wiley & Sons. Shneiderman, B., & Maes, P. (1997). Direct manipulation vs. interface agents. Interactions, 4(6), 42-61. doi: 10.1145/267505.267514 Shneiderman, B., & Plaisant, C. (2005). Designing the User Interface: Strategies for Effective Human-Computer Interaction (4th ed.). Upper Saddle River, NJ: Pearson Education. Silver, J. (Producer), Wachowski, A., & Wachowski, L. (Directors). (1999). The Matrix [Motion Picture]. Australia, US: Village Roadshow Pictures. Sing, G., Wong, K., Fung, C., & Depickere, A. (2006, September 20-24). Towards a more natural and intelligent interface with embodied conversation agent. Proceedings of the CyberGames International Conference on Game Research and Development (pp. 177-183). Perth: Murdoch University Press. Spielberg, S., Kubrick, S., Harlan, J., Kennedy, K., Parkes, W. F., Curtis, B., et al. (Producers), & Spielberg, S. (Director). (2001). Artificial Intelligence: AI [Motion Picture]. US: Dreamworks/Amblin Entertainment. Sproull, L., Subramani, M., Kiesler, S., Walker, J., & Waters, K. (1996). When the interface is a face. Human-Computer Interaction, 11(2), 97-124. doi: 10.1207/s15327051hci1102_1 Ståhl, O., Gambäck, B., Turunen, M., & Hakulinen, J. (2009). A mobile health and fitness companion demonstrator. Proceedings of the Twelfth Conference of the European Chapter of the Association for Computational Linguistics: Demonstrations Session (pp. 65-68). Athens, Greece: European Chapter Meeting of the ACL. Stern, A. (2002). Creating emotional relationships with virtual characters. In R. Trappl, P. Petta, & S. Payr (Eds.), Emotions in Humans and Artifacts (pp. 333-365). Cambridge, MA: MIT Press. Sträfling, N., Fleischer, I., Polzer, C., Leutner, D., & Krämer, N. (2010). Teaching learning strategies with a pedagogical agent: The effects of a virtual tutor and its appearance on learning and motivation. Journal of Media Psychology: Theories, Methods, and Applications, 22 (2), 73-83. doi: 10.1027/1864-1105/a000010 Sunder, S., & Nass, C. (2000, December). Source orientation in human-computer interaction: Programmer, networker, or independent social actors? Communication Research, 27(6), 683-703. doi:10.1177/009365000027006001 Sutcliffe, A. (1989). Human-Computer Interface Design. Basingstoke, UK: Palgrave Macmillan.
213 Szarowicz, A., Francik, J., Mittmann, M., & Remagnino, P. (2005, May 13). Layering and heterogeneity as design principles for animated embedded agents. Information Sciences, 171(4), 355-376. doi: 10.1016/j.ins.2004.09.010 Takeuchi, A., & Nagao, K. (1993). Communicative facial displays as a new conversational modality. Proceedings of the 1993 SIGCHI Conference on Human Factors in Computing Systems (pp. 187-193). Amsterdam, The Netherlands: ACM Press. doi: 10.1145/169059.169156 Takeuchi, A., & Naito, T. (1995). Situated facial displays: towards social interaction. In I. Katz, R. Mack, L. Mark, M. B. Rosson, & J. Nielsen (Eds.), Proceedings of the International Conference Human Factors in Computing Systems (pp. 450-455). Denver, CO: ACM Press. doi: 10.1145/223904.223965 Talukder, A. (2007, May 8-12). Web – “Greatest Equalizer” for the Developing World. Proceedings of Sixteenth International World Wide Web (WWW2007) Conference. Banff, BC, Canada. Tao, J., & Tan, T. (2005). Affective computing: A review. In J. Tao, T. Tan, & R. W. Picard (Eds.), Proceedings of the First International Conference on Affective Computing & Intelligent Interaction (pp. 981–995). Beijing, China: Springer. doi: 10.1007/11573548_125 Tatai, G., Csordás, A., Kiss, Á., Szaló, A., & Laufer, L. (2003). Happy Chatbot, Happy User. Lecture Notes in Computer Science, 2792/2003, 5-12. doi: 10.1007/978-3-540- 39396-2_2 Tecuci, G., & Keeling, H. (1999). Developing an intelligent educational agent with disciple. International Journal of Artificial Intelligence in Education, 10(3-4), 221- 237. The Daily Telegraph. (2008, May 25). Half of world's population has a mobile. Retrieved March 17, 2010, from The Daily Telegraph: http://www.news.com.au/dailytelegraph/story/0,22049,23755088-5001028,00.html Thrun, S. (2002, March). Probabilistic robotics. Communications of the ACM, 45(3), 52- 57. doi: 10.1145/504729.504754 Tian, Y-I., Kanade, T., & Cohn, J. (2001, February). Recognizing action units for facial expression analysis. Pattern Analysis and Machine Intelligence, 23(2), 97-115. doi: 10.1109/34.908962 Tomkins, S. (1962). Affect, imagery and consciousness (Vol. 1). New York, NY: Springer. Tomkins, S. (2008). Affect Imagery Consciousness: The Complete Edition. New York, NY: Springer. Tomkins, S. (1995). Exploring Affect: The Selected Writings of Silvan S. Tomkins. In V. Demos, (Ed.), New York, NY: Cambridge University Press. Turk, M., & Robertson, G. (2000, March). Perceptual user interfaces (Introduction). Communications of the ACM, 43(3), 32-34. doi: 10.1145/330534.330535 UGOBE Inc. (2008). What is Pleo? [Brochure]. Retrieved March 17, 2010, from PleoWorld: http://www.pleoworld.com/discover Valdes, D., Logan, J., Parkes, W. F., MacDonald, L., Leibovit, A. (Producers), & Wells, S. (Director). (2002). The Time Machine [Motion Picture]. US: Parkes/MacDonald. van Mulken, S., André, E., & Müller, J. (1998, August 1-4). The persona effect: how substantial is it? In H. Johnson, L. Nigay, & C. Roast (Eds.), People and Computers XIII: Proceedings of 1998 Conference of Human-Computer Interaction (pp. 53-66). Sheffield, UK: Springer.
214 van Vugt, H. C., Hoorn, J. F., & Konijn, E. A. (2004, August). Adapting empathic agents to user experiences. Proceedings of the 2004 AAMAS Workshop on Empathic Agents. New York, NY. van Vugt, H. C., Hoorn, J. H., Konijn, E. A., & de Bie Dimitradou, A. (2006, September). Affective affordances: Improving interface character engagement through interaction. International Journal of Human-Computer Studies, 64(9), 874-888. doi: 10.1016/j.ijhcs.2006.04.008 van Vugt, H. C., Konijn, E. A., & Hoorn, J. F. (2009, June). Interactive engagement with embodied agents: an empirically validated framework. Computer Animation and Virtual Worlds, 20(2-3), 195-204. doi: 10.1002/cav.312 van Vugt, H. C., Konijn, E. A., Hoorn, J. F., Keur, I., & Eliëns, A. (2007). Realism is not all! User engagement with task-related interface characters. Interacting with Computers, 19(2), 267-280. doi: 10.1016/j.intcom.2006.08.005 Vertegaal, R., Slagter, R., van der Veer, G., & Nijholt, A. (2001). Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes. Proceedings of the 2001 SIGCHI Conference on Human Factors in Computing Systems (pp. 301-308). Seattle, WA: ACM. doi: 10.1145/365024.365119 Walker, J. H., Sproull, L., & Subramani, R. (1994, April 24-28). Using a human face in an interface. Proceedings of the 1994 SIGCHI Conference on Human Factors in Computing Systems: Celebrating Interdependence (pp. 85-91). Boston, MA: ACM. doi: 10.1145/191666.191708 Wallace, R. (2007, January 02). The Annotated A.L.I.C.E. AIML [Fact Sheet]. Retrieved March 17, 2010, from A.L.I.C.E. Artificial Intelligence Foundation: http://www.alicebot.org/aiml/aaa/ Wallace, R. (2003, March 28). The elements of AIML style (Technical Report). ALICE AI Foundation. Retrieved from http://files.ifi.uzh.ch/cl/hess/classes/seminare/chatbots/style.pdf Watson, D., Clark, L., & Tellegen, A. (1988, June). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54(6), 1063-1070. doi: 10.1037/0022-3514.54.6.1063 Webster, J., & Ho, H. (1997). Audience engagement in multimedia presentations. ACM SIGMIS Database, 28(2), 63-77. doi: 10.1145/264701.264706 Winkielman, P., Knutson, B., Paulus, M., & Trujillo, J. L. (2007, June). Affective influence on judgments and decisions: Moving towards core mechanisms. Review of General Psychology, 11(2), 179-192. doi: 10.1037/1089-2680.11.2.179 Yokoi, K., Nakashima, K., Kobayashi, M., Mihune, H., Hasunuma, H., Yanagihara, Y., et al. (2003, October 27- 31). A tele-operated humanoid robot drives a backhoe in the open air. Proceedings of the 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1117-1122). Las Vegas, NV: IEEE. doi: 10.1109/IROS.2003.1248794 Zajonc, R. (1980, February). Feeling and thinking: Preferences need no inferences. American Psychologist, 35(2) 151-175. doi: 10.1037/0003-066X.35.2.151 Zajonc, R., & Markus, H. (1982, September). Affective and Cognitive Factors in Preferences. The Journal of Consumer Research, 9(2), 123-131. Zhang, P., & Li, N. (2005, September). The importance of affective quality. Communications of the ACM, 48(9), 105-108. doi: 10.1145/1081992.1081997 Zimmerman, J., Ayoob, E., Forlizzi, J., & McQuaid, M. (2005). Putting a face on embodied interface agents. Proceedings of the Conference on Designing Pleasurable 215 Products and Interfaces (DPPI) (pp. 233-245). Eindhoven, The Netherlands: Eindhoven Technical University Press.
216 Appendices
Appendix A. Materials of the Survey of Hardware Interfaces of Chatbots
Participant Information Statement and Consent Form
217
218 Questionnaire–Subject Personal Information Component
219
220 Questionnaire – Chatbot on a Computer Component
221
222 Questionnaire – Chatbot on a Portable Device Component
223
224 Questionnaire – Impression of Subject on both Chatbots Component
225 Advertisement
226 Participants’ Intake Script
Phase 1.
Thanks for participating in this experiment. Now I am going to tell you about what you are going to do in this study. Mostly, you need to chat with two ECAs, which have been applied to a computer and iPod Touch. ECA is an abbreviation of embodied conversational agent. ECAs are one type of computer generated artificial intelligent agents with character forms. They communicate with humans by both text and spoken language.
In this research, we are investigating the affective influences of ECAs on humans. In order to measure how different are the affective influences of ECAs, which are applied to different devices for human use, we need you to text chat with both ECAs on the computer [show computer] and the portable device [show iPod Touch].
After having accumulated a certain number of words through chatting with ECAs, the devices will inform you that you have completed the particular chatting. And then you need to fill out questionnaires about both ECAs.
For example, first, you will text chat with the computer ECA on computer. After having accumulated a certain number of words, you need to fill out a questionnaire about the computer ECA. Similar to your conversation with the computer ECA, you will chat with the iPod Touch ECA and complete another questionnaire. Subsequently, there are some multiple choice questions related to this research and a brief interview.
Each conversation with the ECAs will take you 10-20 minutes. The whole procedure will take about 60-90 minutes, including the time you complete questionnaires and undergo an interview.
After completing this experiment, you will be given a 2GB USB flash disk [show flash disk].
Phase 1.
All of these are described on an information sheet I am going to give you. Do you have any questions? [Discuss any questions the participant has]
Also, I would like you to sign this consent form. Please read the information sheet, and afterwards sign the consent form now. [Give the information sheet and consent 227 form][Give participant time to take a look and sign the form][Collect consent form and give the information sheet to participant]
Phase 2.
Firstly, here is an introduction to this experiment and some questions about you. Please take a look and fill it out first. The ECAs and portable devices of this experiment are defined below the introduction. [Give participant the first part of questionnaire, NOTE: PARTICIPANT ID will be pre-printed on it][Show participant the glossary part][Give participant time to read and fill out][Collect the first part of questionnaire]
Phase 3.
Now I am going to show you how to chat with the ECAs. It is very simple, for example, you just type what you want to talk to this computer ECA then press this submit button. It will be sent to the ECA and ECA will reply to you. See? [Type hi and submit as a demonstration] The ECAs have some emotional facial expressions that react to whatever you write, and you can try to see how many facial expressions you can make them display. Also, they are full of the knowledge about giraffes.
When chatting with this ECA after a certain number of words, the device will inform you that you have completed this part of the experiment. Please let me know when the device informs you that you are finished. Then we can continue with the next part of the experiment. Any questions? [Answer any questions] Now, you can start to talk to this computer ECA. [Give participant time to chat with ECA on computer]
Phase 4.
Did you finish the conversation with the ECA? [Check the completed notice on device]
[If no] You need to continue chatting with ECA. After having accumulated enough words you chat with ECAs, the device will inform you that you have completed this part of the experiment. Please let me know when the device informs you that you are finished.
[If yes] Congratulations! You have completed this part of experiment. Here is a questionnaire about you and the computer ECA. Please fill it out and give it to me. Then we can continue the next part of the experiment. [Give the participant the computer ECA part of the questionnaire, NOTE: PARTICIPANT ID will be pre-printed on it][Give the participant time to fill it out][Collect the computer ECA part of the questionnaire]
228 Phase 5.
Similar to your conversation with the computer ECA, please communicate with another ECA in this iPod Touch and then fill out another questionnaire. Like this. [Type hi and submit for demonstrate] Any questions? [Answer any questions] This ECA also has some emotional facial expressions and you can try to see how many facial expressions you can make it display. Now, you can start to chat with this iPod Touch ECA. [Give participant time to chat with ECA on the iPod Touch]
Phase 6.
Did you finish the conversation with the ECA? [Check the completed notice on the device]
[If no] You need to continue chatting with the ECA. After having accumulated enough words you chat with ECAs, the device will inform you that you have completed this part of experiment. Please let me know when the device informs you.
[If yes] Congratulations! You have completed this part of the experiment. Here is a questionnaire about you and the iPod Touch ECA. Please fill it out and give it to me when you have completed it. Then we can continue to the next part of the experiment. [Give the participant the iPod Touch ECA part of the questionnaire, NOTE: PARTICIPANT ID will be pre-printed on it][Give participant time to fill it out][Collect the iPod Touch ECA part of the questionnaire]
Phase 7.
I would like you to fill out this questionnaire. There are a few questions about you and both ECAs. [Give the participant the questionnaire with multiple choice questions, NOTE: PARTICIPANT ID will be pre-printed on it][Give participant time to fill it out][Collect the questionnaire with the multiple choice questions]
Phase 8.
Finally, there are some open questions related to this research. The following interview will be audio recorded. Stored information will be anonymous and you will not be identified from these records. Are you fine with this? Here we go. [Start to record]
1. Did you see different facial expressions appear on both ECAs?
229 2. Do you think they have emotions? 3. Did you notice that the facial expressions on these two ECAs are different? If yes, please describe that. 4. Which ECA do you think displays greater emotion? Why? 5. What is your opinion about the facial expressions of the ECAs affecting the emotions of users? 6. What is your opinion on ECAs on different devices having different ways to express emotions? And how should they express their emotions respectively? 7. Do you think these two ECAs are different characters? 8. Which ECA do you like more? Why? 9. Now ECAs can be installed on your computers and portable devices. Which device do you want to install? Why? 10. Please describe the relationships between you and these two ECAs respectively. 11. Now ECAs can be developed with different characters and functionality on computers and portable devices. For example, a sexy secretary, an all-purpose assistant, an intimate friend and a lovely pet. What is your opinion about what kinds of ECAs should be applied to computers, portable devices, and other devices respectively?
[Stop recording]
Phase 9.
Thanks again for helping out with this project. Here is the 2G USB flash disk for you.
230 Static Images of Chatbots’ Animations
Character Emotion Static images of animation Gary Neutral
Joy
Fear
Gary Sadness
Anger
Gavin Neutral
Joy
Fear
Sadness
Anger
231 Emotional Keywords of Chatbots
Some emotional keywords are listed as follows. When any emotional keyword appears in the response sentences of chatbots, the emotional keyword will trigger the facial expression exchange of chatbots to match the textual response. Otherwise, chatbots show their neutral face.
Happy: happy, in high spirits, cheerful, lighthearted, glad, delighted, jubilant, gay, jolly, joyful, merry, contented, satisfied, pleased, bright, blissful, radiant, well, blessed, carefree, optimistic, pleasant, flushed, content, ha!, exciting, kidding, heh, joke, thank you, hey, fun, hi, hello, welcome, love, enjoy, wow, lucky, wonderful, cool, smile, awesome, like, good, nice,
Sad: sad, poignant, heartrending, distressing, heartbreaking, cheerless, depressing, sorrowful, unhappy, dejected, depressed, blue, melancholy, downcast, discouraging, gloomy, somber, glum, morose, sullen, grievous, miserable, pathetic, unfortunate, forlorn, downhearted, down, doleful, wretched, tragic, tearful, afflictive, sorry, elegiac, penitent, painful, cry, pity, crying, dislike, uh-huh,
Angry: wrathful, wroth, irate, ireful, incensed, enraged, raging, infuriated, infuriate, furious, livid, heated, mad, sore, provoked, indignant, peeved, annoyed, vexed, irritated, piqued, galled, riled, nettled, offended, displeased, indignant, bitter, pissed, teed off, ticked off, in a lather, irked, irritable, cross, surly, petulant, peevish, testy, choleric, touchy, huffy, splenetic, spleenful, bilious, ill-tempered, bad-tempered, irascible, quick-tempered, short-tempered, explosive, hot-tempered, hot- headed, miffed, displeased, galled, nettled, stormy, violent, strong, savage, tempestuous, turbulent, wild, rough, boiling, seething, raging, blustering, blustery, squally, steamed, stewing, burning, bristling, ballistic, bent out of shape, hot under the collar, fuming, enflamed, angry, difficult, hate
Fear: dread, afraid, fright, alarm, dread, panic, terror, funk, tension, suspense, expectation, apprehend, anxiety, horror, mystery, miraculous, embarrassing, secret, surprising, shocking, astonishing, amazing 232 Sample Scripts of Gary – PI-Bot
234
238
241
242 Appendix B. Materials of the Survey of Character Interfaces of CompanionBots
Participant Information Statement and Consent Form
243
244 Questionnaire – Subject Personal Information Component
245
246 Questionnaire – Preference before Interactions Component
247 Questionnaire – CompanionBot Andrew Component
248
249 Questionnaire – CompanionBot Ben Component
250
251 Questionnaire – CompanionBot Chris Component
252
253 Questionnaire – User Engagement Component
254
255 Participants’ Intake Script
Introduction
Thanks for participating in this experiment. I am going to tell you about what you are going to do in this experiment. Mostly, you need to chat with three learning CompanionBots with whom you can practice English conversation. You will communicate with them by text online at home via computer.
In this research, we are investigating the affective influences of ECAs (Embodied Conversational Agents) on humans. ECAs are one type of computer generated artificial intelligent agents with character forms. They can communicate with humans by both text and spoken language. CompanionBots are one type of ECAs. We developed CompanionBots for humans to practice English conversations with more convenience. You can chat with them anytime and anywhere online. We have also given IELTS speaking materials to these three CompanionBots. They can help you to practice for your IELTS speaking test.
This experiment intends to investigate the interaction between you, the CompanionBots, and your affective influences toward CompanionBots. Firstly, we need you to choose your top five characters who you want to be English learning CompanionBots (from 18 characters), and then rank them. In order to measure the effect of different CompanionBots on humans, we need you to text chat with three CompanionBots online at home via computer. You need to chat with each of them at least ten minutes per day and five days per week. After you finish chatting with each CompanionBot, you need to fill out a questionnaire about that CompanionBot. This experiment will take you about three weeks to complete.
For example, in the first week, you text-chat with CompanionBot A. After five days of chatting (at least ten minutes per day), we will send you a URL and a questionnaire by email. The URL is for you to chat with CompanionBot B in the second week and the questionnaire includes some questions in relation to the interactions between you and the CompanionBot A. Similarly, after another five days of chatting with CompanionBot B, you will receive another URL and another questionnaire. The questionnaire contains questions related to CompanionBot B and the URL to CompanionBot C. Finally, after chatting with CompanionBot C, you will get the final questionnaire about CompanionBot C and a date for a five to ten minute interview.
256 Phase 1.
All of these are described on an information sheet I am going to give you. Do you have any questions? [Discuss any question the participant has]
I would like you to sign this consent form. Please read the information sheet and sign the consent form now. [Give the information sheet and consent form][Give the participant time to take a look and sign the form][Collect the consent form and give the information sheet copy to the participant]
Phase 2.
Now I am going to show you how to chat with CompanionBots. It is very simple. For example, you type what you want to say to this CompanionBot, and then press the submit button. The CompanionBot will then reply to you. See? [Type hi and submit for demonstration purposes] you can practice English conversation and IELTS Speaking tests with CompanionBots. We also need you to choose five characters/avatars from a choice of 18 characters, and rank them. I will then give you the questionnaire. Please take a look and fill it out. [Give the participant the first part of the questionnaire. NOTE: PARTICIPANT ID will be pre-printed on it][Show the participant the glossary part][Give the participant time to read and fill it out][Collect the first part of the questionnaire]
Phase 3.
The CompanionBots with whom you will chat in next three weeks have emotional facial expressions that react to what you write. You can try to see how many facial expressions you can make them display. I will send you the URL of your first CompanionBot on the next day and you can start chatting with it. Please check your mailbox frequently, and you will receive some questionnaires and URLs for chatting with the CompanionBots later. After three weeks, you will come back for a five to ten minute interview and collect your reward.
See you in three weeks. Have a good day.
Phase 4.
Congratulations! You have now completed most of this experiment. Finally, I have some open questions related to this research. The following interview will be audio recorded.
257 Stored information will be anonymous and you will not be identified from these records. Are you fine with this? Here we go. [Start to record]
1. Did the CompanionBots help you learn English? How? CompanionBots 㚱⸓≑ぐ劙㔯⬠佺╶烎⤪ỽ⸓≑ぐ烎 2. Which CompanionBot did you like to chat with most? Why? ⒒ᶨᾳ CompanionBot ぐ㚨╄㬉冯⬫Ṍ婯烎䁢Ṩ湤烎 3. Which CompanionBot do you think helped you learn English most? Why? ⒒ᶨᾳ CompanionBoty ⛐ぐ劙婆⬠佺ᶲ⸓≑㚨⣂烎䁢Ṩ湤烎 4. Which CompanionBot do you want to spend more time with? Why? ⒒ᶨᾳ CompanionBot ぐ゛⣂剙ṃ㗪攻冯Ṿ䚠嗽烎䁢Ṩ湤烎 5. When you talk to a native speaker, a classmate who is learning English as a second language, or a CompanionBot, which one do you feel more relaxed conversing with? How come? 䔞ぐ冯㭵婆㗗劙婆䘬Ṣ炻㬋⛐⬠佺劙婆䘬⎴⬠炻冯ᶨᾳ CompanionBot Ṍ婯炻 ᶱ侭ᷕ炻冯ỽ侭Ṍ婯Ἀ奢⼿庫庽檮烎䁢Ṩ湤烎 6. If the CompanionBot could be customised for you, could you please describe your ideal language learning CompanionBot, including appearance and functionality? ⤪㝄⎗ẍ䁢ぐ⭊⇞⊾姕妰ᶨᾳ CompanionBot炻婳ぐ㍷徘Ἀ⣊゛ᷕ䘬婆妨⬠佺 CompanionBot炻⊭⏓⢾堐冯≇傥ˤ [Stop recording]
Phase 5.
Thanks again for helping out. Here is your reward for your participation.
258 Static Images of CompanionBots’ Animations
Character Emotion Static images of animation Andrew Neutral
Happiness
Andrew Fear
Anger
Surprise
Disgust
Sadness
Ben Neutral
Happiness
Fear
259 Character Emotion Static images of animation Anger
Surprise
Disgust
Sadness
Chris Neutral
Happiness
Fear
Anger
Surprise
Disgust
Sadness
260 Emotional Keywords of CompanionBots
Some emotional keywords are listed as follows. When any emotional keyword appears in the response sentences of CompanionBots, the emotional keyword will trigger the facial expression exchange of CompanionBots to match the textual response. Otherwise, CompanionBots show their neutral face. For example, when users input ‘are you sad’ and CompanionBots response ‘No, I am as happy as ever’, the ‘happy’ keyword will trigger CompanionBots to present a happy face.
Anger: wrathful, wroth, irate, ireful, incensed, enraged, raging, infuriated, infuriate, furious, livid, heated, mad, sore, provoked, indignant, peeved, annoyed, vexed, irritated, piqued, galled, riled, nettled, offended, indignant, bitter, pissed, teed off, ticked off, in a lather, irked, irritable, cross, surly, petulant, peevish, testy, choleric, touchy, huffy, splenetic, spleenful, bilious, ill-tempered, bad-tempered, irascible, quick-tempered, short-tempered, explosive, hot-tempered, hot- headed, miffed, displeased, galled, nettled, stormy, violent, strong, savage, tempestuous, turbulent, wild, rough, boiling, seething, raging, blustering, blustery, squally, steamed, stewing, burning, bristling, ballistic, bent out of shape, hot under the collar, fuming, enflamed, angry, difficult, hate, irresponsible, horrible, upset, anger
Disgust: disgust, revulsion, repugnance, abhorrence, repulsion, antipathy, aversion, loathing, hatred, sicken, offend, repel, revolt, nauseate, shame, scorn, offense, contempt, horror, hatred, boring, worse
Fear: fear, dread, afraid, fright, alarm, dread, panic, terror, funk, tension, suspense, expectation, apprehend, anxiety, horror, mystery, miraculous, embarrassing, secret, surprising, shocking, astonishing, amazing
Happiness: happy, in high spirits, cheerful, lighthearted, glad, delighted, jubilant, gay, jolly, joyful, merry, satisfied, pleased, bright, blissful, radiant, blessed, carefree, optimistic, pleasant, flushed, ha!, exciting, kidding, heh, joke, thank, fun, hello, welcome, love, enjoy, wow, lucky, wonderful, cool, smile, awesome, good, nice, happiness
261 Sadness: sad, poignant, heartrending, distressing, heartbreaking, cheerless, depressing, sorrowful, unhappy, dejected, depressed, blue, melancholy, downcast, discouraging, gloomy, somber, glum, morose, sullen, grievous, miserable, pathetic, unfortunate, forlorn, downhearted, down, doleful, wretched, tragic, tearful, afflictive, sorry, elegiac, penitent, painful, cry, pity, crying, dislike, uh-huh, frustrating, loneliness, depression, tear
Surprise: shock, revelation, bolt from the blue, disclosure, bombshell, blow, shocker, astonish, surprise, astonishment, wonder, amazement, awe, startle, astonish, amaze, astound, flabbergast, dumbfound, nonplus, bewilder, awe, catch unaware, amazement, wonderment, great surprise, stupefaction, shock, bewilderment, lost
262 Sample Scripts of IELTS – IELTS-Bot
Sure. Hello, my name is Gary. Can you tell me your full name, please?
Aha, and what can I call you?
Aha, and what can I call you?
OK. Thank you so much. Fine. Now, in the first part, I'd like to ask you a few more questions about yourself. Let's talk about your work. Where do you work?
263
Aha, have you always had that job?
Aha, have you always had that job?
And have you always felt that way about it?
Well, let's change the subject and talk about your personal habits. What is your favourite food?
Aha, and, what is the most important meal of the day for you?
264
265
266 how it might change you or your life, and explain how you would use your new skill.
267
Appendix C. Materials of the Multi-agent Website
The Annotated ALICE AIML Files (AAA Files)
File Patterns File Description 1 1314 Adverbs.aiml Omits adverbs w/o changing logical semantics 2 230 AI.aiml Knowledge about A. I. and robots 3 195 ALICE.aiml Specifically mentions ALICE by name 4 13 Astrology.aiml Star signs 5 3565 Atomic.aiml Categories with patterns without wild cards 6 38 Badanswer.aiml Client can teach bot new replies 7 559 Biography.aiml Famous personalities 8 36 Blackjack.aiml The Card Game 9 3041 Bot.aiml Knowledge about the bot’s personality 10 35 Botmaster.aiml Knowledge about the botmaster 11 1213 Client.aiml Getting to know the client 12 138 Computers.aiml Knowledge about computers and software 13 45 Date.aiml Date and time using Pandorabots formatted AIML date tag 14 4987 Default.aiml Non-committal replies to imprecisely matched inputs 15 10 Dialog.aiml Displays the recent conversation history 16 57 Drugs.aiml Politically Incorrect Opinions 17 123 Emotion.aiml Emotional responses depend on personality type 18 23 Food.aiml Culinary knowledge 19 843 Geography.aiml Places and locations 20 1 Gossip.aiml Spreading rumours 21 20 Happy.aiml Knowledge about the past 22 10 History.aiml Knowledge about the past 23 230 Human.aiml Replaces AI.aiml for a more “human” bot 24 9 Humor.aiml Take your chances 25 25 Inquiry.aiml Gathers Information about the client without repeating questions 26 110 Integer.aiml Simple Integer Addition in AIML 27 99 Interjection.aiml Yes and No 28 2 IU.aiml Default categories for inputs starting with I and YOU. 29 1459 Knowledge.aiml General knowledge 30 8 Literature.aiml Books and fiction 31 31 Luckyslots.aiml Slot machine game 32 33 Money.aiml Economics 33 61 Movies.aiml Film 34 8 Multiple.aiml Multiple Choice Test 35 17 Music.aiml Musical tastes and trends 36 8980 Parts.aiml Simplifies some past participle expressions 37 67 Personality.aiml Simple personality test 38 16 Philosophy.aiml Epistemology and Metaphysics 39 3 Pickup.aiml Pickup Lines 40 24 Politics.aiml More Politically Incorrect Opinions 41 5 Predicates.aiml Manage client predicates 42 714 Psychology.aiml NLP and Assertiveness Training 43 5366 Reduce.aiml General Purpose Symbolic Reductions 44 880 Reducer.aiml Symbolic reductions using
270 Static Images of CompanionBots’ Animations
Character Emotion Static images of animation Andrew Neutral
Happiness
Fear
Anger
Surprise
Disgust
Sadness
Calla Neutral
Happiness
Fear
271 Character Emotion Static images of animation Calla Anger
Surprise
Disgust
Sadness
Annie Neutral
Happiness
Fear
Anger
Surprise
Disgust
Sadness
Gary Neutral
272 Character Emotion Static images of animation Gary Happiness
Fear
Anger
Surprise
Disgust
Sadness
Pax Neutral
Happiness
Fear
Anger
Surprise
Disgust
273 Character Emotion Static images of animation Pax Sadness
Jacob Neutral
Happiness
Fear
Anger
Surprise
Disgust
Sadness
Angel Neutral
Happiness
Fear
Anger
274 Character Emotion Static images of animation Angel Surprise
Disgust
Sadness
Chris Neutral
Happiness
Fear
Anger
Surprise
Disgust
Sadness
Evan Neutral
Happiness
275 Character Emotion Static images of animation Evan Fear
Anger
Surprise
Disgust
Sadness
Sunny Neutral
Happiness
Fear
Anger
Surprise
Disgust
276 Character Emotion Static images of animation Sunny Sadness
277