Exploring the effects of multi-sensory stimulation on the anticipation of ice cream flavours
Ting-Chih, Wang
Project report submitted in part fulfilment of the requirements for the degree of Master of Science (Human-Computer Interaction with Ergonomics) in the Faculty of Brain Sciences, University College London, 2015.
NOTE BY THE UNIVERSITY
This project report is submitted as an examination paper. No responsibility can be held by London University for the accuracy or completeness of the material therein. ACKNOWLEDGEMENT
I am sincerely grateful to my supervisor Paul Marshall, as without his patience, insights, guidance and contribution this project would not have been possible. I would also like to thank my co-supervisor Dr. Bob Hurling and Unilever for the inspiration, help and opportunity for this interesting project.
I would like to express my gratitude to Professor Charles Spence, Charles Michel, and Janice (Qian) Wang from Oxford University Crossmodal Research Laboratory and Professor Adrian Cheok, Emma Yann Zhang, and Gilang Andi Pradana from City University’s Mixed Reality Lab for their inspiration and kindness.
I would also like to thank all the participants for their valuable time in participating in the study.
Finally, I am so grateful for all the support and friendship from the HCI-E students. Also, thanks to my friends and family who supported and encouraged me all the way through.
2
ABSTRACT
This is an exploratory research project aiming to apply existing research findings to the designing of a novel prototype via the Research through Design (RtD) approach. The purpose of the project is to find out whether stimulations of our senses can anticipate the experience of eating ice cream. The design space was explored through three iterations of prototypes. Different methods of evaluation technique were used to analyse the findings from informal experiments. Final prototypes collected quantitative data that led to the findings of the research questions. The findings, to an extent, evidenced that using visual and auditory cues in the anticipation phase can create matching sensations of flavour to the actual flavour tasted. Moreover, through the prototype, the enjoyment of ice cream consumption can be enhanced.
3
TABLE OF CONTENTS
CHAPTER 1 INTRODUCTION ------6
CHAPTER 2 LITERATURE REVIEW ------9 2.1 Multisensory Perception ------9 2.2 Taste and Flavour ------10 2.3 Taste and Smell ------13 2.4 Taste and Sound ------14 2.5 Taste and Sight ------16 2.6 Visual Advertisement ------19 2.7 Taste and Technology ------20 2.8 Food Expectations ------23 2.9 Theories to Design ------25
CHAPTER 3 METHODOLOGY ------2 7 3.1 Approach ------27 3.1.1 Research through Design (RtD) ------27 3.2 Methods ------29 3.2.1 Data gathering ------29 3.2.2 Design ------30 3.2.3 Evaluation ------30 3.2.4 Data analysis ------31
CHAPTER 4 DESIGNING ------3 2 4.1 Idea Generation ------32 4.2 Prototype 1 ------35 4.2.1 Introduction ------35 4.2.2 Methods ------36 4.2.3 Results ------41 4.2.4 Discussion ------41 4.3 Prototype 2 ------42 4.3.1 Introduction ------42 4.3.2 Methods ------42 4.3.3 Results ------46 4.3.4 Discussion ------46
4
CHAPTER 5 FINAL PROTOTYE ------47 5.1 Introduction ------47 5.2 Methods ------48 5.2.1 Design ------48 5.2.2 Participants ------49 5.2.3 Apparatus & materials ------50 5.2.4 Procedure ------50 5.3 Results ------51 5.4 Discussion ------59 5.5 Limitations ------62
CHAPTER 6 GENERAL DISCUSSION ------64
CHAPTER 7 CONCLUSION & FUTURE WORK ------67
REFERENCES ------69
APPENDICES ------81
5
CHAPTER 1. INTRODUCTION
Our brain is powerful, in that it allows us to create feelings without physical experience. Our imagination is a powerful tool. Ever since we were babies, we have used memories to anticipate and expect future events (Canfield et al., 1997; Haith,
Hazan & Goodman, 1988). When we are in doubt about an event that is outside our previous experience, we use our imagination. This occurs in food tasting very often; when we are uncertain about food that we have never eaten before.
In the area of eating experiences, our senses are powerful, and are crucial in creating the experience. Our eating experiences rely on multiple senses, not only taste and smell, as we may imagine. The sound that the food makes while preparing or eating; the colours that the food is presented in; the touch and texture that the food feels like; all should contribute to the whole eating experience (Taylor, 2012). ‘The Perfect
Meal’ by Spence and Piqueras-Fiszman (2014) has a well-rounded presentation of the concept of different elements and stimulations contributing to our eating experience. From different studies and researches, it has been proven that the tasting or eating experience of a certain flavour is multisensory, and not only based on the taste. How we experience for instance a specific dish, is far more complicated than we can imagine and most of the time we wouldn’t notice it. When we see a food, we integrate our senses to identify the ‘flavour’ and to decide whether to consume it or not (Prescott, 2015).
Ice cream is a popular dessert around the world. It is a complex food matrix with a combination of different elements (Frøst et al., 2005; King, 1994). More and more
6 flavours of ice cream are being developed, and the ice-cream company Unilever is implementing new technologies to improve its ice cream and its eating experience
(Unilever, n.d.). As new flavours of ice cream are introduced, people are sometimes uncertain about what the taste experience is like, and therefore are afraid to try. This is where our senses come to work.
Many studies that have been conducted to determine tastes have been technical and non-commercial. It is difficult to move directly from theories to design. A gap between the theories and technological advancement and practical design applications and the commercial world has been created. This project aims to fill this gap by taking the Research through Design (RtD) approach to create a novel design.
The main research question of this project is: “Is there a way to stimulate our senses to create an anticipated flavour in our mind?” Followed by: “Does this change our enjoyment of the perceived flavour?” which will be the next research question.
Our visual and verbal-auditory cognitive processing systems have a close relationship to each other (Atwood, 1971). Therefore, in this project various visual and auditory prototypes will be produced following the RtD method. The prototypes will be tested and analysed. There will be two stages of testing: 1) gathering qualitative data about the participants’ feelings in anticipating flavour with the prototypes; 2) gathering quantitative data about the accuracy of flavour image with both via prototypes or physical tasting.
7
To answer the research questions, the following chapters will cover the background literature to the area of study, and the methods used to answer the research questions.
Afterwards, a design section will go through the design process, artefacts and testing that was conducted, which leads to the final prototype testing. The results and limitations of the study will then be discussed, followed by a discussion of the project overall. Lastly, the project will be concluded and future work discussed. In the next chapter, literature regarding different aspects about our senses and eating experience will be reviewed.
8
CHAPTER 2. LITERATURE REVIEW
2.1 Multisensory Perception
We experience the world with our senses – taste, smell, sight, hearing, and touch.
Our senses help us perceive signals and estimate external events around us in achieving goals or preventing harm (Otto, Dassy & Mamassian, 2013). We often associate a single sense to an event, however, this is untrue. The information received by different senses is correlated (Driver & Noesselt, 2008; Small & Prescott,
2005; Driver & Spence, 2000). An example is the Ventriloquist Effect, which is an illusion that when visual and auditory information is in separate locations, the sound is shifted towards a visual stimulus (Vroomen & De Gelder, 2004). Sensory signals can interact with each other crossmodally (Macaluso & Driver 2005). This is called multisensory perception. We go through multisensory processing in order to form perceptions (Stein & Meredith, 1993). Additionally, it is also essential for robust perception to use a combination of senses (Ernst & Bülthoff, 2004). By integrating these senses, it allows the information we receive to be optimised, which create a more complete experience in different sensory aspects such as hearing a fire alarm and seeing where to go in escaping, as an example for events that are important to our survival (Prescott 2015). Flavour is considered as a multimodal or a multisensory processing object (Stevenson, 2009). It is not, as what we imagine only relating to our taste sense.
9
2.2 Taste and Flavour
In our five senses, taste is more commonly related to flavour (Pandurangan & Hwang,
2015). Our tongue is sensitive; Dublon & Paradiso (2012) from the MIT Media Lab developed Tongueduino that uses the tongue as a medium to experience sight and hearing, especially for the blind and deaf. Moreover, it is also the medium for our sense of taste. In the tasting aspect, an old idea was that our tongue has a “tongue map” (Bartoshuk, 1993). This was a misconception that different tastes have different reception areas on the tongue. Bartoshuk (1993) discussed that, regarding the anatomy of the tongue, “the tongue map” is not accurate, and that the tastes are widely spread on the tongue (Spence & Piqueras-Fiszman, 2014; Neil, 2004;
Stillman, 2002). Our tongue can be referred as an “organ of flavour” (Michel et al.,
2014).
Nevertheless, many studies have demonstrated that flavour perception is not just in our mouth, but also from our nose (Bremner, Lewkowicz & Spence, 2012). Prescott
(1999) defined flavour as “a unique sense” that is not simply a taste in our mouth as we would imagine. The International Organisation for Standardisation (ISO) defined flavour as a “complex combination of the olfactory, gustatory and trigeminal sensations perceived during tasting. The flavour may be influenced by tactile, thermal, painful and/or kinaesthetic effects” (ISO, 2008). Furthermore, studies have shown that flavour perception can be more than the ISO definition; it can be a combination of all five senses, including visual and auditory cues (Bremner,
Lewkowicz & Spence, 2012; Stevenson, 2009; Auvray & Spence, 2008).
10
In research, taste and smell has been constantly discussed together in terms of flavour. The sensory characteristics of foods are mainly odour (Prescott, 1999) instead of the ‘taste’ that we would imagine. Flavour perception and smelling share the same sub-personal mechanism and part of the stimuli. The molecules of the candy, for instance, will be in contact with the receptors by floating to the retro-nasal passage and back to the throat (Richardson, 2013).
Auvray and Spence (2008) stated two types of the perception of flavour – analytic and synthetic. Analytic is when two stimuli are mixed together but still keep their individual qualities. Synthetic perception is where two stimuli are mixed together to form a third sensation by losing their individual qualities. This shows the potential of manipulating stimuli can affect our perception of flavours.
Other than taste and smell, there are other sensory modes that contribute to flavour.
From different studies, it has been evident that the taste sensations are highly related to tactile touch (Bult, de Wijk & Hummel, 2007; Lim & Green, 2007; Lim &
Johnson, 2010; Todrank & Bartoshuk, 1991; Woods, 1998). The temperature on the tongue can elicit sensations of different tastes (Spence & Piqueras-Fiszman, 2014).
Sounds and colours are also factors that would influence the perception of a food or drink (Clydesdale, 1993; Maga, 1974; Moir, 1936; Spence & Shankar, 2010; Spence,
2015; Woods, 1998).
Furthermore, our brain has a great role in determining the taste from these sensory modes. When we choose what food to consume, we use both our conscious and subconscious processing, as well as our economic and social situations, to make the
11 decision (Clark, 1998). This is illustrated graphically by Shepherd (1985) in figure 1.
Our taste perception towards food is also related to our memory and other cognitive processes of its flavour. For example, you experience the sweetness of caramel in your mouth, so when you smell caramel somewhere at a later date it creates the
“odour-induced taste” which has been seen as a close similarity to actual taste
(Stevenson, 2009; Small & Prescott 2005). This was also identified by White and
Prescott (2007) and Prescott et al. (2004) in their study.
figure 1. factors affecting food choice
The next three sections will discuss taste with our senses – smell, sound and sight.
12
2.3 Taste and Smell
Rozin (1982) mentioned that taste is one of the sensory modes that are always confused with the olfactory system. Thus, our olfactory system is strongly related to what we taste. The distinctive properties of a flavour are most likely to be from olfaction rather than the other senses. Stuckey (2012) argued that smelling is an enhancement instead of interference to the experience of flavour and provides pleasure from foods along with other sensory modes.
In multisensory flavour perception, the olfactory sense has two sensory systems – ortho-nasal and retro-nasal. The former one is associated with the inhaling of external odours; and the latter one with smell detection from chewing and swallowing (Spence & Piqueras-Fiszman, 2014). Both of the systems contribute to the tasting experience.
Smell has an important role in tasting, this is commonly proved by holding your nose when having different flavours of for instance sweets, and they will have the same taste experience. When you hold your nose, you are preventing the olfactory component to play its role in perceiving flavours (Neil, 2004; Richardson, 2013).
The everyday conception of senses is when we make judgments with our senses at the instances of perception (Richardson, 2013).
There are many more studies that illustrate the strong connection between taste and smell. There are misconceptions that the taste sense is only related to our mouth, however Richardson (2013) explained perception of taste with water and citral water
13 to show that flavours are not only perceived in our taste buds, but with our nose too.
It is perceived with both smell and taste. Stevenson (2009) also described the combination of “odour-taste” by arguing for the sweetness enhancement of food.
However, in a commercial context, smell is yet harder to simulate. Thus, the next section will focus on sight and sound.
2.4 Taste and Sound
Hearing is another crucial sense for humans to create experiences. According to the traditional information processing approach, the auditory system of a person enables them to construct a mental representation of the environment they are situated in
(Spence & Zampini, 2006). Lageat, Czellar and Laurent (2003) proposed a guideline in their study to use sound as a stimulus in product design to enhance the consumer hedonic experience. There are many more studies that investigate the use of auditory elements to enhance the product design (Norman, 2004; Keiper, 1997; Blauert &
Jekosch, 1997). In addition, the study of Lundborg, Rosén and Lindberg (1999) used processed acoustic stimuli, such as processing the friction sound that represents each finger, then transmitting by a headphone to identify the fingers by sound. The different friction sound on different surface allowed the identification of texture. This can be can be used as an effective substitution to generate artificial sensibility, such as texture in prostheses.
Hearing is an important sense, yet a relatively new idea, in the eating experience.
Our perceptions toward food and drink can be affected by what we hear, even if it is
14 the sound produced when we consume the food (Spence, 2012; Spence & Shankar,
2010; Verhagen & Engelen, 2006; Vickers, 1983). A famous experiment by Zampini and Spence (2004) showed that the perceived crispiness and freshness of potato crisps could be modified by sound levels or frequencies. When the sound level was increased or with amplified high frequency sound, the crisps were perceived as crispier and fresher. The results significantly highlighted that the perception towards food can be modulated by auditory cues.
The characteristics of the audio itself could have an impact on our sense of taste.
Some studies demonstrated the relationship between certain flavours and sound pitch.
The study of Crisinel and Spence (2009) related high-pitched and low-pitched sounds to names of sour and bitter taste foods respectively. Condiment Junkie, a sensory branding and design agency, created soundtracks that could enhance the tastes of sweetness and bitterness (Jones, 2014). The frequency of the sweetness soundtrack was around 22kHz, whereas the bitterness soundtrack was around 66Hz.
Sweetness as more related to smooth and continuous vowel sounds, whereas bitterness and sourness were more matched with staccato sounds (Simner, Cuskley &
Kirby, 2010). Sourness had the highest frequency out of all four flavours (Simner,
Cuskley & Kirby, 2010). Moreover, music can also be a medium to change the level of arousal of a person. Guéguen et al. (2008) and Guéguen, Hélène and Jacob (2004) showed that high volume of music increases the amount of alcohol consumption due to the arousal level with high volume level. A higher sound level is associated with a higher arousal level, inducing a higher level of alcohol consumption.
15
There are restaurants that have implemented auditory aspects into their dishes to enhance customers’ dining experience. For example, the famous dish ‘The sound of the sea’ from Heston Blumenthal’s The Fat Duck restaurant, in which the soundscape was developed by Condiment Junkie. They have successfully used a miniature iPod and headphone to make people focus on the texture and play of the dish. The experience has created emotional impacts such as tear breaking, feeling of pleasantness and focused attention to the experience on the diners toward the dish
(Spence & Piqueras-Fiszman, 2014).
2.5 Taste and Sight
We, as humans, have visual-dominant perceptions (Hoegg & Alba, 2007; Spence,
Shore & Klein, 2001; Driver & Spence, 2000; Posner, Nissen & Klein, 1976). We use visual images effortlessly and efficiently to be immersed in the information surrounding us (Van Essen, Anderson & Felleman, 1992).
Our visual sense involves and supports many more senses than we experience.
Tsakiris and Haggard (2005) revisited the well-known Rubber Hand Illusion (RHI).
The RHI (Botvinick & Cohen, 1998) study is about people who would receive a similar tactile stimulation on their own hand when it was the rubber hand being stimulated. They realised that the illusion is not only the synchronisation of visual and tactile stimulation, but is also modulated by our visual sense, for instance, an ownership feeling (Michel et al., 2014; Tsakiris & Haggard, 2005).
16
Michel et al., (2014) conducted a similar experiment to the Rubber Hand Illusion
(RHI) – the Butcher’s Tongue Illusion. They conducted the experiment with tongues instead of hands. The participants experienced tactile stimulations on their tongues when the dummy tongue was stimulated. This showed that what we see could be a strong stimulation to our physical feelings. However, this is not associated with taste factors.
The flavour that we experience has been proved by many studies to be related to the colour that we see (Spence & Piqueras-Fiszman, 2014; Piqueras-Fiszman, Giboreau
& Spence, 2013; Murray & Wallace, 2012; Zampini, Sanabria, Phillips & Spence,
2007; Moir, 1936). Colour has been seen as the most important sensory cue for people’s expectations of the taste of food and drink (Spence, 2015). One famous study was conducted by Harrar and Spence (2013) where they investigated the effect of different colours, weights and materials of spoons to the taste of yoghurt. Some of the example findings was that white yoghurt tasted from white spoons was sweeter than that from black spoons due to colour contrasts, and also, cheese on a knife tasted saltiest compared with different shapes of cutleries. Furthermore, Ranasinghe,
Lee and Do (2014) created FunRasa, that used an electrical stimulation on the tongue and virtual colouring in a drink. With this system, participants reported the sensations of tingling, as well as tastes that were spicy, sour, salty, bitter and some sweet. They also associated blue to salty, green to sour, and red to bitter.
Moreover, the representation of the food is also important. This is the stage before ingestion, the first impression that we create for the food we are about to consume.
17
For example, the “Kandinsky on a plate” by Charles Michel, is where the arrangement of the plate is inspired by the work of Vasily Kandinsky, in particular
Panel for Edwin R. Campbell No.4 (MoMA, n.d.). This result of the study showed that the arranged dish was more preferable for the participants than when the ingredients were just piled up on a plate (Spence & Piqueras-Fiszman, 2014). See below for the comparison of the arranged and piled plate (figure 2).
figure 2. piled up ingredients and ‘Kandinsky on a plate’
Additionally, other than colours, the shape, size and haptic aspects are also significant to the eating experience (Spence & Piqueras-Fiszman, 2014).
18
2.6 Visual Advertisement
When we put our visual sense into the commercial context, advertisement is a good example. Advertisement is a way that consumers are persuaded into a purchase or belief. One of the purposes of an advertisement is to stimulate purchases (Park,
Shenoy & Salvendy, 2008). Purchases are strongly related to expectations.
Expectations are formed by implicit learning, where the consumer is not aware of the learning process, and explicit learning, from past experience (Simons & Noble,
2003).
In an advertisement, especially in the digital age, visual and auditory aspects are being used more and more. Park & Young (1986) conducted a study that examined the use of music in advertisements. It was found that main concept for them in the advertisement could be facilitated and communicated by visual or auditory cues; however, it can be distracting if the visual or auditory cues are not effective. In the digital age, still images have now become continuous images. Animation is a popular way to create an advertisement. In Bush, Hair, and Bush’s (1983) research they found that approximately 20 percent of TV advertisements contain total or partial animation. In addition, animations are more commonly used in low risk products.
Visual elements are essential to an advertisement, as it creates strong and positive attitudes in the audience and increases the desire to purchase (Clow et al., 2006). The study by Posner, Nissen and Klein (1976) found that visual processing was strongly affected by extra auditory stimulus. An advertisement utilises both visual and verbal
19 aspects to create attitudes from the audiences towards the product or services being advertised (Mitchell & Olson, 2000; Mitchell, 1986).
2.7 Taste and Technology
This section will review different studies in Human-Computer Interaction (HCI) research that have aimed to augment our eating experience by creating virtual tastes.
Food is essential in our lives. With the development of new technologies, future food experiences can be imagined. The interaction of humans and food has become more about the engagement of people with potential technology involvement (Comber et al., 2014; Spence & Piqueras-Fiszman, 2013). One future food experience is digitally replicated food (Schoning, Rogers & Kruger, 2012). The advancement of 3D printing technology provides the potential for digitally produced food. For example,
CandyFab (CandyFab, n.d.) can create a customised shape of sugar by caramelising it; and there is Choc Edge (Choc Edge, n.d.), where chocolates can be produced in customised patterns or shapes. However, there are many limitations to what a 3D printer can produce with edible materials. Technologies can also be employed to change eating behaviours. FoodWorks (Ganesh et al., 2014) was a study that aimed to encourage children to eat healthy foods that they usually dislike, such as carrots.
Animated light and projections were used to distract and change their attention on to the lights on the vegetable, rather than the vegetable itself. The results were encouraging, showing that FoodWorks can enhance and make the food more appealing for children to eat. However, this is changing the expectation towards the vegetables for the children, in that when they see the original form, it is still possible
20 that they reject the vegetable. Their anticipation towards the vegetables has been changed.
From the literature mentioned before, the utensil used could also be a factor that affects the eating experience. Kita and Rekimoto’s (2014) study created two utensils aimed at enhancing the food eating experience. First, the utensil HEAT is a plate that heated up the food to keep the temperature. Most of the participants preferred the food with the managed temperature; the second utensil PAFUMA enhanced the aroma of champagne by using vibration on the glass. All participants agreed that the device strengthened the aroma. Moreover, Hirose et al. (2015) created a spoon that changes weight. The study showed that when the spoon was heavier, participants perceived that there was a larger amount of food content to consume. The heavier spoon also affected quality perception, such as a marshmallow feeling harder and more elastic. This showed that the weight of food affects our perception towards food. The designs of these utensils are still in an experimental stage.
There are more studies that are still being developed that aim to create virtual tastes in our mouths. National University of Singapore (NUS) engineers have developed a cup and spoon that would enhance or restore taste by electric pulses on the tongue
(Cuthbertson, 2015). There is another project from the NUS by Dr Nimesha
Ranasinghe at the Keio-NUS CUTE Centre. It is a device that uses different combination of electrical currents and temperatures to simulate the sweet, sour, bitter and salty sensations (National University of Singapore, 2014). Professor Adrian
David Cheok from City University, London has been doing research with a similar device to digitise taste through electrical and thermal stimulation (Zhang, 2015).
21
Nakamura and Miyashita (2011) used a similar concept by using an electrical stimulus to create electric tastes with both drink and food, through a straw and fork or chopsticks respectively. Furthermore, Keio University in Japan created the
TagCandy device that can augment the taste of a lollipop by using vibrations (Mills,
2010). Meta Cookie created by Narumi et al. (2010) tried to create virtual taste through visual and olfactory cues. They provided plain cookies with virtual flavour for participants to select by visual display, and to manipulate the olfactory information by the scent released to the participant. There is progress being made regarding creating virtual tastes, however, these studies are still focusing on creating the taste in our mouth rather than in combinations with other senses.
In Obrist et al.’s (2014) study they examined the experiences of the basic five tastes
(Schiffman, 2000) that can be used as a guideline in designing the eating experience.
They discussed the five tastes in regard to time. Sweetness is a pleasant taste, however through time it can be unsatisfying; sourness is usually unpleasant at first but the fast decay of its taste makes us want more; bitterness is an unpleasant taste that takes longer to decay which make us not want to experience it again; umami has both likes and dislikes but it is still confusing when determining the taste; and lastly, saltiness is not linked to any extreme reaction towards it.
22
2.8 Food Expectations
When we are experiencing something new, we have expectations or anticipations prior to the actual experience. According to Simons and Noble (2003), our purchase decision is affected by our expectations, which are based on factors like previous experience. The feelings of pleasantness and inclination towards a product are closely related to our expectation of the characteristics of the product (Schifferstein,
Kole & Mojet, 1999). In food consumption, expectation is strongly involved.
Similarly to Simons and Noble (2003), Tuorila, Cardello and Lesher (1994) also mentioned that past experience and sensory cues creates expectations in our mind relating to whether to accept or reject the food (Piqueras-Fiszman & Spence, 2015).
The expectations of food can be divided into two types – imagining the sensory attributes of the food, or whether the food will be liked or disliked. When there is a gap between the expected and actual attributes, this could have both a negative and positive impact depending on whether they are better than expected or worse
(Cardello & Sawyer, 1992). The expectations can be formed with the physiological factors as well (Costell, Tárrega & Bayarri, 2009). In the study of Rebollar et al.,
(2012) they found that the colours of the packaging influenced the taste of gum; warm colours was expected to be more sweet and fruity, whereas grey colours were expected to be more menthol and spicy. The exteroceptive cues that involve visual, auditory and olfactory aspects usually occur before the consumption of food
(Piqueras-Fiszman & Spence, 2015). Expectation is important for food consumption because it might create an improvement or degradation towards the food even before tasting (Deliza & MacFie, 1996).
23
Collins English Dictionary (Collins English Dictionary, n.d.) defines expectation as
“an attitude of expectancy or hope; anticipation”. Anticipation is the phase before food consumption, which is when the expectation is first formed. Kramer and
Szczesniak (1973) created a sensory circle that was divided into three zones – appearance, flavour and kineasthetics. It showed how our senses could overlap and merge into each other. Hutchings (1977) then added the anticipatory and participatory aspects to it, where the perception can differ depending on the moment of perception. This showed the stages and aspects that food consumption includes, and that the process starts before consumption. Anticipation mostly occurs within the appearance zone and links to flavour perception (as shown in figure 3).
figure 3. Kramer’s sensory circle modified
This showed that our anticipation is closely related to our visual cues. Visual cues are available even before ingestion (Murray & Wallace, 2012). They create an expectation of taste for the food that is about to be consumed (Hutchings, 1977).
McCarthy and Wright (2004) proposed the use of technology as an experience, and created six sense making processes when we encountered a new experience. The six phases are what the user went through when interacting with an experience. The
24 anticipation phase in our mind is illustrated by the first stage of “anticipating” in this process. This is a framework that focused on the experiences “felt by the user”
(Rogers, Preece & Sharp, 2011, p151). This anticipation phase will be a main focus in this project to create the virtual flavour.
2.9 Theories to Design
There has been much technical advancement towards creating augmentation of flavours. However, these technologies are not yet common in the commercial world.
There is a gap between the technical advancement and the commercial use, and between theories and design. Visual and auditory aspects are more straightforward implementation in linking the gap. As humans, we have visual hunger. Visual hunger can be used as a stimulation to fulfill the expectations of people towards food. This is what the food media use to manipulate the consumers into purchasing food. Food images are sometimes also made more appetising and that might increase salivation
(Spence et al., 2015), which can enhance the eating experience and increase enjoyment. Taking from the concept of advertisements, audio is also an essential compliment to a consumer’s expectation. When people’s expectations are closer to the properties of the product, the sensations are closer to the expectations rather than the actual sensory properties of the product (Schifferstein, Kole & Mojet, 1999).
The psychology studies reviewed above were all carried out in controlled lab environment and studies on augmentation of tastes are still experimental. Most of them are focused on the eating experience but not the anticipation stage. It is difficult to translate the proposed theories in the literature to design. This created a gap
25 between the findings from literature and practical design applications to flavour augmentation in different stages. To fill the gap, the project will create a design that would be able to provide an experience to the product prior to consuming it by using multimodal stimuli. This is to manipulate the user’s expectation towards the food given in their anticipation state. This project is aiming to answer the research question of “ways to stimulate our senses to create an accurate anticipated flavour in our mind” by utilising visual and auditory stimulations.
26
CHAPTER 3. METHODOLOGY
This chapter will cover two stages that the project has taken – the design space, which was explored through the Research through Design (RtD) approach, and the design ideas, which were evaluated in the controlled in-situ environment to gather the validity of the design ideas and potential in the problem space.
3.1 Approaches
3.1.1 Research through Design
The question of how design practice can be considered as research has been debated
(Rampino, 2012). Zimmerman, Forlizzi and Evenson (2007) proposed a solution –
Research through Design (RtD). This is an approach to integrating design into HCI research by having design artefacts as an outcome method (Zimmerman, Forlizzi &
Evenson, 2007). The artefact created can be used as a key to a new design space
(Zimmerman, Stolterman & Forlizzi, 2010; Gaver & Martin, 2000). The process involves problem framing and employing methods to develop the artefacts
(Zimmerman, Stolterman & Forlizzi, 2010). The artefacts can involve mock-ups to fully functional prototypes (Keyson, n.d.). One of the challenges in RtD is the requirement of design inquiry into a “Wicked Problem”, which is a problem that is not approachable by science and engineering (Zimmerman, Stolterman & Forlizzi,
2010; Rittel & Webber, 1973). This project is an exploratory research using RtD due to the difficulties of moving from theories directly to design.
27
This project has implemented the approach of RtD. The wicked problem in this project is the formation of anticipated flavours in people’s minds prior to eating. To tackle this problem, research of related fields was constructed to lead to idea generation. During the idea generation stage, different methods such as sketching and storyboarding were used to evaluate various ideas to narrow down to a direction of the final idea. Prototypes were created for the final idea, and through iterative process, the idea was evaluated and tested for further development. Two prototypes were developed, leading to the final idea. Different data collection methods were used to gather data for analysis. The data collected and analysed from the study of the final prototype were leading to answers to the research question. The process of the whole project is illustrated with a flowchart shown below (figure 4).
figure 4. project process
28
3.2 Methods
3.2.1 Data gathering
Different data gathering methods serves different purposes throughout the project.
The two types of data collected were quantitative and qualitative data.
3.2.1.1 Quantitative
There were two questionnaires used throughout the project. Due to the low
cost and short time requirement, during the idea generation stage it was easy
to collect quick quantitative data to explore the designs. OnePulse (OnePulse,
n.d.) was used to gather a large amount of data (100 participants) in a short
period of time. Then, for the study of the final prototype, a quantitative
questionnaire for participants to rate characteristics of the ice cream flavour
with 9-point Likert scale (Wakita, Ueshima & Noguchi, 2012) was used. This
allowed quick and understandable data to be collected from the participants’
experience.
3.2.1.2 Qualitative
Interviews were used to collect qualitative data about the participants’
experience. The pilot study and testing of prototype 1 both used semi-
structured interviews to collect data. Interviews allowed the greater detail of
participants’ experience to be explored (Cassell & Symon, 2004). This
qualitative data was used as guidelines to further development of the design
ideas. Additionally, unstructured interviews were used to gather quick
feedback from potential users with sketches and storyboards. The qualitative
29
data were analysed by categorising similar pattern of data (Rogers, Preece &
Sharp, 2011); responses were categorised to lead to design decisions and
improvements.
3.2.2 Design
There are different design methods that can be implemented to develop through ideas. In this project, the first stage of idea generation used different sketching to explore a wide range of ideas. Then, after different ideas were analysed and evaluated, storyboards were drawn to further develop the idea. Storyboards are an easy way to create visual images of the design task flow and gain feedback from users (Sova & Sova, 2006).
When the ideas were narrowed down, prototypes were created. In this project, the design idea was a video clip with visual and auditory elements, thus prototypes of videos were made. They are serving the purpose of creating visual and auditory stimulation for participants to create the image of the flavour in their mind.
3.2.3 Evaluation
The prototypes produced were evaluated. Data gathered from the evaluation were used for further development. Studies were conducted with all three prototypes including pilot study, controlled-environment study and in-the-wild study. The pilot study was used to test out the initial ideas and gain feedback for development and changes. After the first prototype was created, a controlled-environment study was used. Participants were invited into a room with minimal external influences (Rogers,
Preece & Sharp, 2011). The participants were able to focus on the visual and
30 auditory aspects in the prototype. Then, for the final prototype, to be able to broaden the target audience and reach out to more participants, the in-the-wild study was used. This allowed users to be in the natural settings (Rogers, Preece & Sharp, 2011), that they are experiencing, for instance, in a supermarket prior to purchasing an ice cream.
3.2.4 Data Analysis
The final prototype testing data collected from 9-point Likert scale were tested for significance with Aligned Rank Transform (ART) (Wobbrock et al., 2011) by repeated measures ANOVA. The software used was called ARTool (Wobbrock et al., 2015). This allows nonparametric factorial data with multiple factors to be analysed. The common method of The Friedman test (Corder & Foreman, 2009) would not work in this study to determine the interaction with multiple factors
(Wobbrock et al., 2011). Furthermore, there were two characteristics that had not only significance between the expected and perceived taste, but also between the different simulations of expected and perceived taste. These two contrasts were followed up with the Wilcoxon Signed Rank Test. This allows both the simulation groups and expected and perceived taste, to be compared for significance.
31
CHAPTER 4. DESIGNING
This chapter will present the design process of the project, from the starting phase of the idea generation, to the developing stage of two prototypes, then into the final prototype. The methods and procedures used in each phase will be discussed.
The project followed an iterative process (Rogers, Preece & Sharp, 2011, p330).
Different ideas were brainstormed and sketched using different methods. Then, an idea was chosen to be developed further. Three different iterations have been made to this idea to reach the final prototype.
4.1 Idea generation
The first stage to idea generation, knowledge, was gathered from the literature of the five senses and brainstormed in a mind map (figure 5).
figure 5. brainstorm mind map
32
From the mind map, ideas of testing out different senses were sketched (see figure 6 for example).
figure 6. examples of testing ideas
Then, more physical designs were sketched and evaluated (see figure 7 for example).
figure 7. physical design ideas
From the literature reviewed, it was evident that all senses affected the flavour perception. However, visual and auditory aspects were more straightforward and easier to implement to form anticipation. From the commercial perspectives, the prototype was chosen to focus on these more straightforward elements. Tindall-Ford,
33
Chandler & Sweller (1997) proved in their study with three experiments that information in mixed or audio-visual form is far more effective than visual-only.
Thus, with visual and auditory aspects, the idea development for the final idea was generated.
For the design the Ben & Jerry’s ice cream from Unilever was used as a case study for the studies. Three flavours were chosen and analysed – Minter Wonderland,
Greek Style Strawberry Shortcake and Phish Food. A visual brainstorm was used to develop the idea for all three flavours (figure 8)
figure 8. visual brainstorm of the three flavours
34
The main ingredients were emphasised and evaluated with the visual and auditory aspects. The prototypes produced were referred as the ‘simulation’ of the flavour.
These prototypes were evaluated with informal experiments that gathered data to enable design improvements and research findings.
4.2 Prototype 1
4.2.1 Introduction
Other research has showed that images and sounds are effective cognitive stimulators affecting peoples’ minds (Clow et al., 2006; Tindall-Ford, Chandler & Sweller, 1997;
Posner, Nissen and Klein, 1976). Therefore, the first prototype was developed with still images and sound. There were two different simulations generated with the same images but different audio. The simulation was created with iMovie. The storyboard of the testing of prototype is illustrated in figure 9.
figure 9. user testing storyboard
35
4.2.2 Methods
4.2.2.1 Design
The still images were of the main ingredients of each flavour of ice cream: 1)
Greek Style Strawberry Shortcake, 2) Minter Wonderland and 3) Phish Food.
There were four soundtracks used across the two simulations. The
soundtracks were served with three main purposes – 1) enhancing flavour, 2)
emphasising the image of ice cream and 3) elevating the feeling of coldness.
The third purpose was research through the questionnaire distributed to 30
respondents to receive feedback about their feelings towards the sound effects.
A few sound effects of the ingredients were added as well, such as cookie
crunching sound.
Before producing the prototype, a questionnaire was used to gather data about
two sound effects and how they related to the feeling of coldness. These two
sound effects were: 1) wind blowing, and 2) a rod stirring ice cubes in a glass.
Prototype 1 was a pilot study. This prototype contained two different versions
of the same prototype – simulation 1 and 2, with the difference of background
soundtracks.
Simulation 1 used the Sweetness soundtrack from Condiment Junkie, this
soundtrack has higher frequencies, around 22kHz. It is higher than Bitterness
soundtrack (Jones, 2014). For the Greek Style Strawberry Shortcake, an
additional track was added, which was essentially the same soundtrack as the
one used for sweetness but with a higher pitch, according to the literature
(Simner, Cuskley & Kirby, 2010; Crisinel & Spence, 2009). By increasing
36
the pitch of the sweetness soundtrack it would be able to enhance the
sourness in this flavour, which contains yoghurt. Additionally, the sound
effect of cold wind blowing was added to elevate the feeling of coldness. This
sound effect has been voted as a slightly higher feeling of coldness. Figure 10,
showed the components of the simulations in order of flavour 1, 2 and 3 (see
appendix 1 for sample screens).
figure 10. components of prototype 1 simulation 1
37
Simulation 2 used a soundtrack that has a happy atmosphere to emphasise the
joyful ice cream eating experience. Then, the sound effect used here was a
rod stirring ice cubes inside a glass. This was voted as a slightly lower feeling
of coldness compare to the cold wind blowing but was still effective
according to the questionnaire. Figure 11, showed the components of the
simulations in order of flavour 1, 2 and 3 (see appendix 1 for sample screens).
figure 11. components of prototype 1 simulation 2
38
4.2.2.2 Participants
A total of 30 responses were received for the questionnaire of the sound
effects. There were a total of 5 participants involved in the pilot testing. They
experienced both simulations with a short unstructured interview in between
and provided feedback on both of them.
4.2.2.3 Apparatus & Materials
The materials used involved an iPad 2 as a display of the prototype
connecting to a headphone that allowed the participants to listen to the
soundtrack and effects. Small polystyrene ice cream cups were used to
contain the different flavours of ice cream with numbered label (figure 12).
figure 12. numbered label for ice creams
The three flavours of ice cream used in the pilot testing were Ben & Jerry
Greek Style Strawberry Shortcake, Minter Wonderland and Phish Food. Ice
cream spoons were also used for each tasting.
39
4.2.2.4 Procedure
1. The participants were given background information about the project
with the information sheet (appendix 2) and were given the consent form
(appendix 3) to sign.
2. Firstly, the participants were asked to watch one type of the flavour
simulations chosen by the facilitator.
3. After they were asked to close their eyes to prevent colour influence, the
facilitator provided them the ice cream of the flavour in the simulation.
4. Then, a short unstructured interview was implemented to gather
information on their experience comparing the prototype and actual
eating experience.
5. Secondly, the participants were asked to watch the other type of the
flavour simulations.
6. They were asked again, to close their eyes to prevent colour influence,
and the facilitator provided them the ice cream of the flavour in the
simulation.
7. Then, a short unstructured interview was implemented to gather
information, both on their experience comparing the prototype and eating
and comparison to the first simulation they experienced.
8. Lastly, the participants were given a free ice cream as their compensation.
40
4.2.3 Results
There was some positive feedback, such as that they could imagine what each ingredient tastes like, however, it was hard to imagine the taste of the combination of the ingredients, therefore difficult to imagine the overall flavour. For the flavour
Minter Wonderland specifically, three of the participants stated that they didn’t know what a mint leaf looked like. Moreover, few of the participants pointed out that it would be helpful if there were some description of the ingredients or flavour. For the feedback of the soundtrack, two participants mentioned that the happy soundtrack was distracting, as because of the beats they were paying more attention to the sound then the images. Two other participants mentioned that the wind-blowing soundtrack sounded too depressing and did not match the feeling of an ice cream. Most of the participants’ found the ingredient sound effects useful in creating a more vivid image.
4.2.4 Discussion
These feedbacks were useful for developing new design requirements. For example, the need for descriptions of either the ingredients or the flavour should be taken into consideration for the next prototype. The coldness sound effects were not as effective as imagined. Also, a requirement of the feeling of combination of ingredients needs to be added. By using these feedbacks, the second prototype can be generated and developed.
41
4.3 Prototype 2
4.3.1 Introduction
To develop from the idea of still images, an idea to use animation was developed.
One of the issues from prototype 1 was that the participants couldn’t imagine the ingredients mixing together; thus when they could visualise the process of how ingredients relate to each other inside an ice cream cup, it helped them form the image. The animation was produced using Flash Professional CC and the simulation video was created in iMovie. An informal experiment was implemented to gather design feedback.
4.3.2 Method
4.3.2.1 Design:
There were two simulations made for this prototype. The main purposes of
these two simulations were to test out the idea of the animation and the
difference between using sound effects and verbal description. Sound effects
were added because, according to the literature, flavour perception can be
positively affected by the natural sound produced when consuming food
(Vickers, 1983). Both simulations have the same animation and sweetness
soundtrack.
Simulation 1 focused on using sound effects, for instance, the dropping sound
when the chocolate dropped into the ice cream cup. Figure 13, showed the
components of the simulations in order of flavours 1, 2 and 3 (see appendix 4
for sample screens).
42
figure 13. components of prototype 2 simulation 1
Simulation 2 focused on using verbal descriptions to describe the ingredients
that were added to the cup and all ending with a phrase ‘all swirled into one
delicious ice cream.
The verbal descriptions were:
Flavour 1: Mellow milk, juicy strawberries, natural Greek yoghurt and
crunchy shortbreads.
Flavour 2: Mellow milk, refreshing mint and crunchy chocolate chunks.
Flavour 3: Fresh milk, mellow melted chocolate, rich caramel, tasty melted
marshmallow and fish shaped chocolates.
Figure 14, showed the components of the simulations in order of flavours 1, 2
and 3 (see appendix 4 for sample screens).
43
figure 14. components of prototype 2 simulation 2
4.3.2.2 Participants:
This prototype was tested on 14 people. Simulations 1 and 2 were evenly
distributed among the participants. The testing took place in a controlled lab
environment.
4.3.2.3 Apparatus & Materials:
The materials used and involved in this testing was identical to the pilot
testing, with an iPad 2 to display the prototype and a headphone for listening
to the audio. Small polystyrene ice cream cups were used to contain the
different flavours of ice cream with the same label (see figure 12). The three
44
flavours of ice cream used were the same – Ben & Jerry’s Greek Style
Strawberry Shortcake, Minter Wonderland and Phish Food. Ice cream spoons
were also used for each tasting.
4.3.2.4 Procedure:
1. The participants were given background information about the project
with the information sheet and given the consent form to sign.
2. The participants were asked to watch one of the flavour simulations
chosen by the facilitator.
3. After they were asked to close their eyes to prevent colour influence, the
facilitator provided them all three flavours of ice creams in random order.
4. The participants were asked to open their eyes and state which one of the
flavours was the one he or she experienced in the flavour simulation.
5. Then, a short semi-structured interview was implemented.
6. Lastly, the participants were given a free ice cream as their compensation.
45
4.3.3 Results
Qualitative data was collected from the short semi-structured interview. There was positive feedback regarding the ability of the prototype to anticipate flavour, with most participants mentioning that they were able or to an extent to imagine, what the flavour could taste like. Simulation 1 had slightly more positive feedback from the feedback about the sound effects used. However, the verbal description had positive feedback too. The participants also found the sound effect useful in creating a more realistic atmosphere.
4.4.4 Discussion
The qualitative data gathered in the interviews showed that both sound effects and verbal descriptions were stated by the participants as effective but not strong enough.
Few participants suggested the combination of both could create a more realistic and vivid image in their minds. For the overall flavour, the participants stated that it is hard to imagine the flavour because they did not know about the proportion of each ingredients of the flavour. The animation of the adding of ingredients was relatively effective; however, when the ingredients are added to the ice cream cup they disappear when the next one is coming along, some participants mentioned that this disrupts their image of the flavour. Moreover, all participants had a different perspective towards some of the ingredients from their past experiences. For example, one of the participants saw the mint leaves and directly thought it should be mint flavoured chocolate, however, the actual ice cream is mint flavoured ice cream with chocolate chips. Also, the three flavours were very distinct which made it easier to state the one that they experienced in the simulation.
46
CHAPTER 5. FINAL PROTOTYPE
This chapter will cover the whole study for the final prototype including the methods used, the results and discussion.
5.1 Introduction
The final prototype was produced with qualitative data analysis from prototype 2.
Most of the participants tested with prototype 2 mentioned that they could imagine and paint a picture of the flavour they saw in the simulation in their mind. Therefore, this idea was continued to the final idea. For the final prototype, the study focused on how different the perceptions were towards the flavour in the simulation and the actual eating experience by rating the characteristics of the flavour such as sweetness and sourness; also, the effects of auditory and visual perception. From prototype 2, the Phish Food flavour simulation was the most effective among all three flavour simulations. Therefore, for the final prototype, this flavour was chosen.
47
5.2 Method
5.2.1 Design:
The improvements that were made from prototype 2 were that, when the ingredients were added into the ice cream cup they stayed on screen until the end. Both the sound effects and verbal description were added in the background. The Ben &
Jerry’s logo was removed as it would be more obvious and easier to link to the Phish
Food flavour from the participant’s past experience. Also, overall improvements on the quality of the animation were made.
There were three different simulations created for different purposes:
Simulation 1 is a purely verbal description of the flavour by stating the ingredients; this is the controlled condition.
Simulation 2 will use the animation with sweetness soundtrack, sound effects and verbal description. This will be the main condition video. Figure 15 shows the components of the video (see appendix 5 for sample screens).
figure 15. components of final prototype simulation 2
48
Simulation 3 will use the same animation, sound effects and verbal description, but with the bitterness soundtrack. This is to test the effect of the Sweetness and
Bitterness soundtrack (Jones, 2014); if there will be any effects on the bitterness with the manipulated soundtrack when experiencing the prototype with the same controlled condition. Figure 16 showed the components of the video (see appendix 5 for sample screens).
figure 16. components of final prototype simulation 3
There were 14 characteristics of ice creams (appendix) chosen to be rated by the participants on a 9-point Likert scale (Wakita, Ueshima & Noguchi, 2012). They were sweetness, sourness, bitterness, spiciness, crunchiness, freshness, fruitiness, creaminess, smoothness, coldness, chocolate flavour, marshmallow flavour, caramel flavour and eagerness. They were typical adjectives that would be used to describe ice cream flavours.
5.2.2 Participants:
This prototype was tested on 51 people (17 per simulation). Simulation 1, 2 and 3 were evenly distributed among the participants. The study implemented an in-the- wild strategy.
49
5.2.3 Apparatus & Materials:
The materials used involved in this were similar to the previous testing. An iPad 2 was used to display the prototype and a headphone for the audio. Small polystyrene ice cream cups were used to contain the ice cream, and ice cream spoons were used for tasting. Ben & Jerry’s Phish Food flavour was used for the study. A sheet of paper with 9-point Likert scale was printed for the participant’s reference when answering questions. Also, a notebook was used to record the findings (see appendix
6 for sample).
5.2.4 Procedure:
1. The participants were given background information about the project with the
information sheet and given the consent form to sign.
2. The participants were asked to watch one of the simulations.
3. The facilitator asked the participants to rate all 14 characteristics.
4. Then, the participants were asked to eat the actual ice cream.
5. Again, the facilitator asked the participants to rate the 14 characteristics.
6. Lastly, the participants were given free ice cream as compensation.
50
5.3 Results
The independent variable in the study was the three simulations – the main video, the controlled verbal description and the video with manipulated sound. The dependent variables were the 14 characteristics rated with the prototype and after eating. A separate ART ANOVA was carried out for each dependent variable to test for significance. The ART (Aligned Ranked Transform) allowed analysis of nonparametric factorial data collected with the 9-point Likert scale with multiple factors (Wobbrock et al., 2011).
The R console ran three different contrasts between – 1) three simulations, 2) prototype flavour expectation or eating perceived flavour and 3) both simulations and prototype or eating.
The following are the findings of each characteristic of the three contrasts.
Sweetness:
There was no main effect of simulation versions for sweetness, F(2,48)=1.10, p >
0.05.
There was no main effect between the expected and perceived sweetness,
F(1,48)=1.16, p > 0.05.
There was no interaction between simulations and expected or perceived sweetness,
F (2,48)=0.78, p > 0.05.
51
Sourness:
There was no main effect of simulation versions for sourness, F(2,48)=0.47, p > 0.05.
There was significant main effect between the expected and perceived sourness,
F(1,48)=6.18, p < 0.05. After experiencing the simulations, both viewing the video and hearing the prototype, participants expected the ice cream to taste more sour than they perceived when actually eating the ice cream.
There was no interaction between simulations and expected or perceived sourness,
F(2,48)=0.08, p > 0.05.
Bitterness:
There was no main effect of simulation versions for bitterness, F(2,48)=0.45, p >
0.05.
There was significant main effect between the expected and perceived bitterness, F
(1,48)=5.87, p < 0.1. After experiencing the simulations, both viewing the video and hearing the prototype, participants expected the ice cream to taste more bitter than they perceived when actually eating the ice cream.
There was no interaction between simulations and expected or perceived sweetness,
F (2,48)=0.39, p > 0.05.
52
Spiciness:
There was no main effect of simulation versions for spiciness,
F (2,48)=1.54, p > 0.05.
There was significant main effect between the expected and perceived spiciness, F
(1,48)=23.63, p < 0.0001. After experiencing the simulations, both viewing the video and hearing the prototype, participants expected the ice cream to taste spicier than they perceived when actually eating the ice cream.
There was interaction between simulations and expected or perceived spiciness, F
(2,48)=40.49, p > 0.001. Wilcoxon Signed Rank Test was used for contrast between simulations and expected or perceived spiciness. There was no significant effect on simulation 2 and 3 both expected and perceived flavour. However, there was an effect that approached significance in simulation 1 (p = 0.68) between the expected and perceived ratings of spiciness. Most of the participants rated the ice cream as non-spicy when the verbal description was given to them, but there were few participants rated 2, 4, 5 and 7 for spiciness. After eating the actual ice cream, the perceived spiciness was all non-spicy. This shows that the verbal description of the ice cream would create an expectation of spiciness when the actual tasting perception had no spiciness. At the same time, it showed that the video-simulations could create a better expectation image than verbal-only simulation.
53
Crunchiness:
There was no main effect of simulation versions for crunchiness, F (2,48)=1.38, p >
0.05.
There was no main effect between the expected and perceived crunchiness, F
(1,48)=1.42, p > 0.05.
There was no interaction between simulations and expected or perceived crunchiness,
F (2,48)=0.02, p > 0.05.
Freshness:
There was no main effect of simulation versions for freshness, F (2,48)=0.26, p >
0.05.
There was significant main effect between the expected and perceived freshness, F
(1,48)=13.33, p < 0.001. After experiencing the simulations, both viewing the video and hearing the prototype, participants expected the ice cream to be less fresh than they perceived when actually eating the ice cream.
The perceived freshness is higher than expected.
There was no interaction between simulations and expected or perceived freshness, F
(2,48)=1.14, p > 0.05.
54
Fruitiness:
There was no main effect of simulation versions for fruitiness, F (2,48)=2.37, p >
0.05.
There was significant main effect between the expected and perceived fruitiness, F
(1,48)=51.51, p < 0.001. After experiencing the simulations, both viewing the video and hearing the prototype, participants expected the ice cream to taste fruitier than they perceived when actually eating the ice cream.
There was interaction between simulations and expected or perceived fruitiness, F
(2,48)=64.42, p > 0.001. Wilcoxon Signed Rank Test was used for contrast between simulations and expected or perceived fruitiness. There was no significant effect on simulation 2 and 3 both expected and perceived flavour. However, there was a significant effect in simulation 1. Most of the participants rated the ice cream as non- fruity when the verbal description was given to them, but there were a few participants who rated 2, 3 and 5 for fruitiness. After eating the actual ice cream, the perceived spiciness was all non-fruity. This shows that the verbal description of the ice cream would create expectation of fruitiness when actual tasting perception had no fruitiness. At the same time, it showed that the video-simulations could create a better expectation image than verbal-only simulation.
55
Creaminess:
There was no main effect of simulation versions for creaminess, F (2,48)=0.06, p >
0.05.
There was no main effect between the expected and perceived creaminess, F
(1,48)=2.80, p > 0.05.
There was no interaction between simulations and expected or perceived creaminess,
F (2, 48)=1.82, p > 0.05.
Smoothness:
There was no main effect of simulation versions for smoothness, F (2,48)=2.48, p >
0.05.
There was significant main effect between the expected and perceived smoothness, F
(1,48)=16.72, p < 0.001. After experiencing the simulations, both viewing the video and hearing the prototype, participants expected the ice cream to be less smooth than they perceived when actually eating the ice cream.
There was no interaction between simulations and expected or perceived smoothness,
F (2,48)=0.08, p > 0.05.
56
Coldness:
There was no main effect of simulation versions for coldness, F (2,48)=0.21, p >
0.05.
There was no main effect between the expected and perceived coldness, F
(1,48)=1.60, p > 0.05.
There was no interaction between simulations and expected or perceived coldness, F
(2,48)=1.86, p > 0.05.
Chocolate flavour:
There was no main effect of simulation versions for chocolate flavour, F (2,48)=1.73, p > 0.05.
There was significant main effect between the expected and perceived chocolate flavour, F (1,48)=16.39, p < 0.001. After experiencing the simulations, both viewing the video and hearing the prototype, participants expected the ice cream to be of less chocolate flavour than they perceived when actually eating the ice cream.
There was no interaction between simulations and expected or perceived chocolate flavour, F (2,48)=2.04, p > 0.05.
57
Marshmallow flavour:
There was no main effect of simulation versions for marshmallow flavour, F
(2,48)=2.16, p > 0.05.
There was significant main effect between the expected and perceived marshmallow flavour, F (1,48)=22.94, p < 0.0001. After experiencing the simulations, both viewing the video and hearing the prototype, participants expected the ice cream to have more marshmallow flavour than they perceived when actually eating the ice cream.
There was no interaction between simulations and expected or perceived marshmallow flavour, F (2,48)=0.53, p > 0.05.
Caramel flavour:
There was no main effect of simulation versions for caramel flavour, F (2,48)=0.24, p > 0.05.
There was significant main effect between the expected and perceived caramel flavour, F (1,48)=7.18, p < 0.1. After experiencing the simulations, both viewing the video and hearing the prototype, participants expected the ice cream to have more caramel flavour than they perceived when actually eating the ice cream.
There was no interaction between simulations and expected or perceived caramel flavour, F (2,48)=1.01, p > 0.05.
58
Eagerness:
There was no main effect of simulation versions for eagerness, F (2,48)=0.51, p >
0.05.
There was significant main effect between the expected and perceived eagerness, F
(1,48)=11.04, p < 0.01. After experiencing the simulations, both viewing the video and hearing the prototype, participants expected their eagerness to be less than they experienced when actually eating the ice cream.
There was no interaction between simulations and expected or perceived eagerness,
F (2,48)=0.21, p > 0.05.
5.4 Discussion
The results showed that some aspects of the prototype were effective and some of were not.
The four characteristics that had no statistical significance (sweetness, crunchiness, creaminess and coldness) showed that the expected and perceived flavours were similar, and that it is easy to imagine the level of them without tasting. Most of the ice creams that exist are quite creamy and cold. Thus, this is easier for people to judge the creaminess and coldness just by a given description or visual images.
Sweetness is a common taste of ice creams, so with the ingredients shown, especially chocolate, it is easy for people to predict the level of sweetness from their past experience with the ingredients. The image of chocolate chunks and description of
“crunchy chocolate chunks” creates an accurate image of the crunchiness from
59 peoples’ past experience of biting into chocolate chips.
There were more characteristics that the participants found stronger in the expected taste than was perceived when eating the ice cream. The six characteristics were sourness, bitterness, caramel flavour, spiciness, fruitiness and marshmallow flavour.
People gave more expectation to the characteristics; however, the actual experience was at a lower level. Sourness, bitterness and spiciness are all from the basic five tastes (Schiffman, 2000), and the level of these flavours usually varies between people. People have different tolerances of sour, bitter and spicy tastes. These flavours are relatively easy to recognise, thus people would expect them to be at higher level when the question was asked. Fruitiness only means the resemblance of fruit tastes but can also be defined as “rich in flavour” (Dictionary.com, n.d.). Thus, when the participants were asked the question of fruitiness, it could be interpreted in different ways. If fruitiness was seen as meaning rich in flavour, by showing all the ingredients that one could be stronger or weaker than others, the prototype type would create a richer flavour than the eating experience. Marshmallow and caramel flavours are not the main ingredients that have a vivid taste, thus by providing a description and images of these two ingredients, it created a larger picture of the taste than it should have.
Freshness, smoothness, chocolate flavour and eagerness: these four characteristics were expected to be at a lower level than the perceived. People found it fresher, smoother, more chocolate flavour and eager after eating the ice cream. Smoothness is harder to imagine where it could mean the physical texture or the eating experience as a whole. The chocolate chunks description and images weren’t strong enough to create the crunchiness of the ice cream overall. However, sometimes the frozen ice
60 cream could create some ice flake crunchiness; therefore, people would only feel this crunchiness when eating the ice cream. By showing a mixture of ingredients without each proportion, it is harder to imagine chocolate flavour as the main taste of the ice cream. They can only notice chocolate being the strongest and main taste of the ice cream after eating. Freshness is also another characteristic that has multiple definitions and it is harder to create an image of freshness with the prototypes, it is more towards a physical feeling. Thus, the feeling of freshness was clearer after eating. During the study, multiple people mentioned that they were not big fans of ice cream or the flavour doesn’t seem like their preference. However, after consuming the ice cream most of them enjoyed it and the eagerness of eating increased. The prototypes didn’t allow the participants to accurately imagine the mixture of tastes.
The only two characteristics that had significance between three the simulations, expected and perceived flavour were spiciness and fruitiness. The statistical difference is with simulation 1. Participants rated the ice cream experienced with some spiciness and fruitiness when the verbal description was given to them, but after eating the ice cream they found it to be non-spicy and non-fruity. This showed that with verbal descriptions, it is harder to determine the spiciness and fruitiness compared to containing visual images and effects. A mixture of audio and video elements created a more accurate expectation. According to the literature, for example Tindall-Ford, Chandler and Sweller’s (1997) study, it proved that mixture of audio-visual is way more effective than audio or visual-only.
From the study, it is evident that the video prototype could possibly create an image of flavour in people’s mind. They were able to determine some of the characteristics of the ice cream flavour from visual or audio stimulation and also their past
61 experiences or memories. In a commercial context, it has the potential to provide customers with an opportunity to construct a general image of the flavour before purchasing.
5.5 Limitations
There were some limitations that affected the effectiveness of the study. Firstly, most of the past studies in the taste augmentation area were lab-based study; however, this study was in-the-wild. Factors can be easily controlled in a lab environment, but there were many external factors that cannot be controlled in the in-the-wild study, for instance, noise on the road or people talking, sun glare where people cannot see the screen completely and sound or visual distractions (Spence & Shankar, 2010).
All these could potentially affect the effectiveness of the prototype (Rogers et al.,
2007). Then, the logistics of ice cream was one of the issues during the study. It was difficult to keep the ice creams in their best condition when doing the study; therefore people from different times could experience the ice cream in different states. For example, the coldness could change due to the melting of ice cream.
During the study where participants were given the ice cream to eat, they all ate different amounts of ice cream. The perception of the flavour could change after eating different amounts (Obrist et al., 2014).
Additionally, different parts of the ice cream could give different perceptions of taste as well. For example, there were some parts of the ice cream that had more marshmallow or caramel taste and some parts had more chocolate crunches, thus participants who ate these specific parts could experience crunchiness, marshmallow
62 and caramel flavour differently. The different interpretation or understanding of the fourteen characteristics that were asked could be affecting the participants’ response as well. Furthermore, some participants had previous experience with the ice cream.
They made a guess that it was Phish Food from Ben & Jerry’s after seeing the prototype. This could possibly affect how they expected the ice cream to taste like, based on what they had experienced previously.
Although visual and auditory stimulation could be strong stimuli to expecting flavour, it is not strong enough in comparison with the combination of texture and smell. This limited the possibilities of accurate expectation, however, as for commercial purposes, this study provided a potential of possible augmentation of taste just with auditory and visual simulation.
63
CHAPTER 6. GENERAL DISCUSSION
The findings throughout the project have provided some insights to the problem space created. The findings, to some extent, answered the research question. From the findings of the evaluation it was evident that visual and auditory cues were potentially able to stimulate our senses to create anticipated flavours. For example, sweetness, crunchiness, creaminess and coldness were perceived as what was expected. Thus, a realistic sensation of these characteristics can be created using visual and auditory cues. From the literature, we understand that flavour perception is not just about our taste but multiple senses (see Bremner, Lewkowicz & Spence,
2012; Stevenson, 2009; Auvray & Spence, 2008; Driver & Spence, 2000). The findings have proven these aspects of flavour perception. The final prototype utilised the bitterness soundtrack to elevate the bitterness sensation. Obrist et al. (2014) described bitterness as an unpleasant taste; however, from the findings bitterness didn’t have a discouraging effect. This might be both the soundtrack not creating an overall effect or that audio stimulation of bitterness would not create as much unpleasantness as actual taste. Further research needs to be done to be certain.
One of the main purposes of the project is to create the flavour perception in the anticipation stage. The anticipation stage in Hutching’s (1997) modified sensory circle (see 2.7) showed involvement in appearance, which is our visual sensation and flavour. From the findings, it can be positively correlated to the sensory circle, as visual cues were used to create imaginary flavour prior to consumption. Multiple studies mentioned that our expectations are formed partially by past experience. The
64 observation throughout the project can support this concept. In Deliza and MacFie’s
(1996) study, they stated that expectation could create improvement to food consumption, which is evident in this study by the higher levels of eagerness to consume the ice cream when eating it after viewing the prototype. The increase of eagerness answered the second research question, which showed that rather than change our enjoyment, it elevated our enjoyment. It has also been mentioned that when there is a gap between expected and perceived flavour, it could have an either positive or negative impact (Cardello & Sawyer, 1992). In this project, the gap between expected and perceived, either higher or lower expectation than perceived, all resulted in higher eagerness when consuming the ice cream.
In general, the project has proven the potential in the problem space of using sensory cues, such as visual and auditory, to create an accurate expectation of flavour. The
RtD approach was useful to resolve the difficulty of going directly from theories to design, which is often a challenge. A problem space was able to be created to utilise the theories as a basis for design works. However, as this is an exploratory research, more aspects have to be considered within the problem space to resolve the wicked problem (see Zimmerman, Stolterman & Forlizzi, 2010; Rittel & Webber, 1973).
This project also implemented in-the-wild studies; in section 5.3, some negative impacts on these studies were mentioned. However, Rogers et al., (2007) argued that despite the disadvantages, in-the-wild studies are effective in demonstrating behaviours of people in their intended setting. There were a few challenges during the project. RtD is a relatively new approach, so it is possible that the project could utilise the approach more effectively, such as more concrete research on the design space, and a greater variety of ideations and idea testing. More ideas could have been
65 explored in the design space to allow more iterations and development around the research question. Overall, the project has led to interesting findings. More formal experiments could be implemented to gather more perspectives and aspects of data to expand the design space in order to solve more or larger wicked problems.
66
CHAPTER 7. CONCLUSION & FUTURE WORK
This exploratory research project implemented the Research through Design (RtD) approach to create a design space for creating anticipated tastes through our senses.
Three iterations of prototype were created, data was collected and evaluated, and findings were presented. The findings showed that the visual and auditory cues, to an extent, were able to create accurate sensations of the anticipation of flavour. Also, the prototype has shown that the experience of ice cream consumption has positively increased after viewing the prototype. The stimulation of anticipation can lead to positive sensation of the actual tasted flavour.
This project is a starting point within the problem space, connecting the theories to the design, with the aim of this design being used for commercial purposes.
This project provided potential evidence that through visual and auditory stimulation, it is possible for people to imagine a certain flavour of ice cream. In the commercial context, there are some potential possibilities for application. The prototype could be developed further with the study results to increase accuracy. This can be used as a
“taster” for ice cream. Customers in a supermarket, for example, see a new flavour of ice cream but have no idea if they would like the taste or not. As it is all packaged, there is not any way to try the ice cream. So, as they scan the QR code on the shelf or ice cream box, this will link to this “taster” and provide the customers with a general idea of the taste for purchasing decisions. Spence and Piqueras-Fiszman (2014) mentioned the use of QR codes to modify the interaction with food, either to have the
QR code on or in the food for diners to scan that would lead them to different aspects
67 of multisensory stimulation that would enhance or change their experiences. For example, Qkies (Qkies, n.d.) provides information about the cookies or improve social interaction by customising the QR codes. Furthermore, it would be interesting to look at the differences when applying this onto different flavours of ice cream or even on other foods. If the same effect can be shown in other applications, this would be useful in providing a new way of experiencing foods.
68
REFERENCES
Atwood, G. (1971). An experimental study of visual imagination and memory. Cognitive Psychology,2(3), 290-299. doi:10.1016/0010- 0285(71)90015-6
Auvray, M., & Spence, C. (2008). The multisensory perception of flavor. Consciousness And Cognition, 17(3), 1016--1031.
Bartoshuk, L. (1993). The biological basis of food perception and acceptance. Food Quality And Preference, 4(1-2), 21-32. doi:10.1016/0950-3293(93)90310-3
Bartoshuk, L. (1993). The biological basis of food perception and acceptance. Food Quality And Preference, 4(1-2), 21-32. doi:10.1016/0950-3293(93)90310-3
Blauert, J., & Jekosch, U. (1997). Sound-quality evaluation--a multi-layered problem. Acta Acustica United With Acustica, 83(5), 747--753.
Botvinick, M., Cohen, J., & others,. (1998). Rubber hands' feel'touch that eyes see. Nature, 391(6669), 756--756.
Bremner, A., Lewkowicz, D., & Spence, C. (2012). Multisensory development. Oxford: Oxford University Press.
Bult, J., de Wijk, R., & Hummel, T. (2007). Investigations on multimodal sensory integration: Texture, taste, and ortho- and retronasal olfactory stimuli in concert. Neuroscience Letters, 411(1), 6-10. doi:10.1016/j.neulet.2006.09.036
Bush, A., Hair, J., & Bush, R. (1983). A Content Analysis of Animation in Television Advertising.Journal Of Advertising, 12(4), 20-41. doi:10.1080/00913367.1983.10672859
CandyFab,. CandyFab | A DIY 3D Sugar Printer. Wiki.candyfab.org. Retrieved 3 September 2015, from http://wiki.candyfab.org/
Canfield, R., Smith, E., Brezsnyak, M., Snow, K., Aslin, R., & Haith, M. et al. (1997). Information Processing Through the First Year of Life: A Longitudinal Study Using the Visual Expectation Paradigm. Monographs Of The Society For Research In Child Development, 62(2), i. doi:10.2307/1166196
Cardello, A., & Sawyer, F. (1992). Effects of disconfirmed consumer expectations on food acceptability. J Sensory Studies, 7(4), 253-277. doi:10.1111/j.1745- 459x.1992.tb00194.x
69
Cassell, C., & Symon, G. (2004). Essential guide to qualitative methods in organizational research. London: SAGE Publications.
Choc Edge,. Creations - Choc Edge. chocedge.com. Retrieved 3 September 2015, from http://chocedge.com/
Clark, J. (1998). Taste and flavour: their importance in food choice and acceptance. Proc. Nutr. Soc.,57(04), 639-643. doi:10.1079/pns19980093
Clow, K., James, K., Kranenburg, K., & Berry, C. (2006). The relationship of the visual element of an advertisement to service quality expectations and source credibility. Journal Of Services Marketing, 20(6), 404-411. doi:10.1108/08876040610691293
Clydesdale, F. (1993). Color as a factor in food choice. Critical Reviews In Food Science And Nutrition,33(1), 83-101. doi:10.1080/10408399309527614
Collins English Dictionary,. Definition of “expectation” | Collins English Dictionary.Collinsdictionary.com. Retrieved 3 September 2015, from http://www.collinsdictionary.com/dictionary/english/expectation
Comber, R., Choi, J., Hoonhout, J., & O'hara, K. (2014). Designing for human--food interaction: An introduction to the special issue on ‘food and interaction design’. International Journal Of Human-Computer Studies, 72(2), 181--184.
Corder, G., & Foreman, D. (2009). Nonparametric statistics for non-statisticians. Hoboken, N.J.: Wiley.
Costell, E., Tárrega, A., & Bayarri, S. (2009). Food Acceptance: The Role of Consumer Perception and Attitudes. Chem. Percept., 3(1), 42-50. doi:10.1007/s12078-009-9057-1
Crisinel, A., & Spence, C. (2009). Implicit association between basic tastes and pitch. Neuroscience Letters, 464(1), 39-42. doi:10.1016/j.neulet.2009.08.016
Crisinel, A., & Spence, C. (2009). Implicit association between basic tastes and pitch. Neuroscience Letters, 464(1), 39-42. doi:10.1016/j.neulet.2009.08.016
Cuthbertson, A. (2015). Taste+ smart spoon and cup virtually enhance food flavours and restore taste to the elderly. International Business Times UK. Retrieved 14 August 2015, from http://www.ibtimes.co.uk/taste-smart-spoon-cup-virtually- enhance-food-flavours-restore-taste-elderly-1497816
70
Deliza, R., & MacFie, H. (1996). The generation of sensory expectation by external cues and its effect on sensory perception and hedonic ratings: a review. J Sensory Studies, 11(2), 103-128. doi:10.1111/j.1745-459x.1996.tb00036.x
Dictionary.com,. the definition of fruitiness. Dictionary.com. Retrieved 21 August 2015, from http://dictionary.reference.com/browse/fruitiness
Driver, J., & Noesselt, T. (2008). Multisensory Interplay Reveals Crossmodal Influences on ‘Sensory-Specific’ Brain Regions, Neural Responses, and Judgments. Neuron, 57(1), 11-23. doi:10.1016/j.neuron.2007.12.013
Driver, J., & Spence, C. (2000). Multisensory perception: Beyond modularity and convergence. Current Biology, 10(20), R731-R735. doi:10.1016/s0960- 9822(00)00740-5
Driver, J., & Spence, C. (2000). Multisensory perception: Beyond modularity and convergence. Current Biology, 10(20), R731-R735. doi:10.1016/s0960- 9822(00)00740-5
Dublon, G., & Paradiso, J. (2012). Tongueduino: Hackable, High-bandwidth Sensory Augmentation. InACM (pp. 1453--1454). Retrieved from http://doi.acm.org/10.1145/2212776.2212482
Ernst, M., & Bülthoff, H. (2004). Merging the senses into a robust percept. Trends In Cognitive Sciences, 8(4), 162-169. doi:10.1016/j.tics.2004.02.002
Frøst, M., Heymann, H., Bredie, W., Dijksterhuis, G., & Martens, M. (2005). Sensory measurement of dynamic flavour intensity in ice cream with different fat levels and flavourings. Food Quality And Preference, 16(4), 305-314. doi:10.1016/j.foodqual.2004.05.009
Ganesh, S., Marshall, P., Rogers, Y., & O'Hara, K. (2014). FoodWorks: tackling fussy eating by digitally augmenting children's meals (pp. 147--156).
Gaver, B., & Martin, H. (2000). Alternatives: exploring information appliances through conceptual design proposals (pp. 209--216).
GUEGUE, N. (2004). Sound level of background music and alcohol consumption: an empirical evaluation. Perceptual And Motor Skills, 99(4), 34. doi:10.2466/pms.99.4.34-38
Guéguen, N., Hélène, L., & Jacob, C. (2004). Sound Level Of Background Music And Alcohol Consumption: An Empirical Evaluation 1. Perceptual And Motor Skills, 99(1), 34--38.
71
Guéguen, N., Jacob, C., Le Guellec, H., Morineau, T., & Lourel, M. (2008). Sound Level of Environmental Music and Drinking Behavior: A Field Experiment With Beer Drinkers.Alcoholism: Clinical And Experimental Research, 32(10), 1795-1798. doi:10.1111/j.1530-0277.2008.00764.x
Haith, M., Hazan, C., & Goodman, G. (1988). Expectation and Anticipation of Dynamic Visual Events by 3.5-Month-Old Babies. Child Development, 59(2), 467. doi:10.2307/1130325
Harrar, V., & Spence, C. (2013). The taste of cutlery: how the taste of food is affected by the weight, size, shape, and colour of the cutlery used to eat it. Flavour, 2(1), 21. doi:10.1186/2044-7248-2-21
Hirose, M., Iwazaki, K., Nojiri, K., Takeda, M., Sugiura, Y., & Inami, M. (2015). Gravitamine spice: a system that changes the perception of eating through virtual weight sensation (pp. 33--40).
Hoegg, J., & Alba, J. (2007). Taste Perception: More than Meets the Tongue. J CONSUM RES, 33(4), 490-498. doi:10.1086/510222
Hutchings, J. (1977). The Importance Of Visual Appearance Of Foods To The Food Processor And The Consumer1. Journal Of Food Quality, 1(3), 267--278.
ISO,. (2008). ISO 5492:2008 - Sensory analysis -- Vocabulary. Iso.org. Retrieved 1 June 2015, from http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber =38051
Jones, R. (2014). Bittersweet Symphony | Condiment Junkie. Condimentjunkie.co.uk. Retrieved 13 August 2015, from http://condimentjunkie.co.uk/blog/2015/4/27/bittersweet-symphony
Keiper, W. (1997). Sound quality evaluation in the product cycle. Acta Acustica United With Acustica,83(5), 784--788.
Keyson, D. Empirical research through design.
King, B. (1994). Sensory Profiling of Vanilla Ice Cream: Flavour and Base Interactions. LWT - Food Science And Technology, 27(5), 450-456. doi:10.1006/fstl.1994.1091
Kita, Y., & Rekimoto, J. (2014). Digitally Enhanced Utensils: Designing Dynamic Gustation. Springer, 48--57.
72
Kramer, A., & Szczesniak, A. (1973). Texture Measurements of Foods. Dordrecht: Springer Netherlands.
Lageat, T., Czellar, S., & Laurent, G. (2003). Engineering hedonic attributes to generate perceptions of luxury: Consumer perception of an everyday sound. Marketing Letters, 14(2), 97--109.
Lim, J., & Green, B. (2007). Tactile Interaction with Taste Localization: Influence of Gustatory Quality and Intensity. Chemical Senses, 33(2), 137-143. doi:10.1093/chemse/bjm070
Lim, J., & Johnson, M. (2010). Potential Mechanisms of Retronasal Odor Referral to the Mouth.Chemical Senses, 36(3), 283-289. doi:10.1093/chemse/bjq125
Lundborg, G., Ros\'en, B., & Lindberg, S. (1999). Hearing as substitution for sensation: a new principle for artificial sensibility. The Journal Of Hand Surgery, 24(2), 219--224.
Macaluso, E., & Driver, J. (2005). Multisensory spatial interactions: a window onto functional integration in the human brain. Trends In Neurosciences, 28(5), 264- 271. doi:10.1016/j.tins.2005.03.008
Maga, J. (1974). Influence of color on taste thresholds. Chemical Senses, 1(1), 115- 119. doi:10.1093/chemse/1.1.115
McCarthy, J., & Wright, P. (2004). Technology as experience. Interactions, 11(5), 42. doi:10.1145/1015530.1015549
Michel, C., Velasco, C., Salgado-Montejo, A., & Spence, C. (2014). The butcher’s tongue illusion.Perception, 43(8), 818.
Mills, A. (2010). Japanese create device that allows candy to taste like anything – even fireworks. TNW News. Retrieved from http://tnw.to/1CYfh
Mitchell, A. (1986). The Effect of Verbal and Visual Components of Advertisements on Brand Attitudes and Attitude Toward the Advertisement. J CONSUMRES, 13(1), 12. doi:10.1086/209044
Mitchell, A., & Olson, J. (2000). Are Product Attribute Beliefs the Only Mediator of Advertising Effects on Brand Attitude?. Advertising &Amp; Society Review, 1(1). doi:10.1353/asr.2000.0010
73
Moir, H. (1936). Some observations on the appreciation of flavour in foodstuffs. Journal Of The Society Of Chemical Industry, 55(8), 145-148. doi:10.1002/jctb.5000550803
MoMA,. Vasily Kandinsky. Panel for Edwin R. Campbell No. 4. 1914 | MoMA. Moma.org. Retrieved 2 September 2015, from http://www.moma.org/collection/works/79452?locale=en
Murray, M., & Wallace, M. (2012). The neural bases of multisensory processes. Boca Raton: CRC Press.
Nakamura, H., & Miyashita, H. (2011). Augmented gustation using electricity (p. 34).
Narumi, T., Kajinami, T., Tanikawa, T., & Hirose, M. (2010). Meta cookie (p. 143).
National University of Singapore,. (2014). Simulator recreates virtual taste online. News.nus.edu.sg. Retrieved 14 August 2015, from http://news.nus.edu.sg/highlights/7198-simulator-recreates-virtual-taste-online
Neil, M. (2004). A neuroanatomy of flavour. Petits Propos Culinaires, (76), 58-81. Retrieved from http://www.scribd.com/doc/164371604/A-neuroanatomy-of- flavour#scribd
Norman, D. (2004). Emotional design. New York: Basic Books.
Obrist, M., Comber, R., Subramanian, S., Piqueras-Fiszman, B., Velasco, C., & Spence, C. (2014). Temporal, affective, and embodied characteristics of taste experiences: A framework for design (pp. 2853--2862).
OnePulse,. OnePulse - Consumer insight has just gone real- time. Business.onepulse.com. Retrieved 23 August 2015, from http://business.onepulse.com/
Otto, T., Dassy, B., & Mamassian, P. (2013). Principles of multisensory behavior. The Journal Of Neuroscience, 33(17), 7463--7474.
Pandurangan, M., & Hwang, I. (2015). Systemic Mechanism of Taste, Flavour and Palatability in Brain.Applied Biochemistry And Biotechnology, 175(6), 3133- 3147. doi:10.1007/s12010-015-1488-3
Pandurangan, M., & Hwang, I. (2015). Systemic Mechanism of Taste, Flavour and Palatability in Brain.Appl Biochem Biotechnol, 175(6), 3133-3147. doi:10.1007/s12010-015-1488-3
74
Park, C., & Young, S. (1986). Consumer Response to Television Commercials: The Impact of Involvement and Background Music on Brand Attitude Formation. Journal Of Marketing Research, 23(1), 11. doi:10.2307/3151772
Park, T., Shenoy, R., & Salvendy, G. (2008). Effective advertising on mobile phones: a literature review and presentation of results from 53 case studies. Behaviour & Information Technology,27(5), 355-373. doi:10.1080/01449290600958882
Piqueras-Fiszman, B., & Spence, C. (2015). Sensory expectations based on product- extrinsic food cues: An interdisciplinary review of the empirical evidence and theoretical accounts. Food Quality And Preference, 40, 165-179. doi:10.1016/j.foodqual.2014.09.013
Piqueras-Fiszman, B., Giboreau, A., & Spence, C. (2013). Assessing the influence of the color of the plate on the perception of a complex food in a restaurant setting. Flavour, 2(1), 24. doi:10.1186/2044-7248-2-24
Posner, M., Nissen, M., & Klein, R. (1976). Visual dominance: An information- processing account of its origins and significance. Psychological Review, 83(2), 157-171. doi:10.1037//0033-295x.83.2.157
Prescott, J. (1999). Flavour as a psychological construct: implications for perceiving and measuring the sensory qualities of foods. Food Quality And Preference, 10(4-5), 349-356. doi:10.1016/s0950-3293(98)00048-2
Prescott, J. (2015). Flavours: the pleasure principle. Flavour, 4(1), 15. doi:10.1186/2044-7248-4-15
Prescott, J. (2015). Multisensory processes in flavour perception and their influence on food choice.Current Opinion In Food Science, 3, 47-52. doi:10.1016/j.cofs.2015.02.007
Prescott, J., Taylor, A., Roberts, D., et al. (2004). Psychological processes in flavour perception.Flavor Perception, 256--277.
Qkies,. QKies – Sag's mit Keksen. Qkies.de. Retrieved 14 August 2015, from http://qkies.de/
Rampino, L. (2012). Design Research: Between Scientific Method and Project Praxis. Notes on Doctoral Research in Design 2012.
Ranasinghe, N., Lee, K., & Do, E. (2014). Funrasa: an interactive drinking platform (pp. 133--136).
75
Rebollar, R., Lidón, I., Serrano, A., Martín, J., & Fernández, M. (2012). Influence of chewing gum packaging design on consumer expectation and willingness to buy. An analysis of functional, sensory and experience attributes. Food Quality And Preference, 24(1), 162-170. doi:10.1016/j.foodqual.2011.10.011
Richardson, L. (2013). Flavour, Taste and Smell. Mind & Language, 28(3), 322-341. doi:10.1111/mila.12020
Rimm, D., & Bottrell, J. (1969). Four measures of visual imagination. Behaviour Research And Therapy, 7(1), 63-69. doi:10.1016/0005-7967(69)90050-3
Rittel, H., & Webber, M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155--169.
Rogers, Y., Connelly, K., Tedesco, L., Hazlewood, W., Kurtz, A., & Hall, R. et al. (2007). Why it’s worth the hassle: The value of in-situ studies when designing ubicomp.
Rogers, Y., Preece, J., & Sharp, H. (2011). Interaction design. Hoboken, N.J.: Wiley.
Rozin, P. (1982). “Taste-smell confusions” and the duality of the olfactory sense. Perception & Psychophysics, 31(4), 397-401. doi:10.3758/bf03202667
Ryan, T. (1960). Significance tests for multiple comparison of proportions, variances, and other statistics. Psychological Bulletin, 57(4), 318-328. doi:10.1037/h0044320
Schifferstein, H., Kole, A., & Mojet, J. (1999). Asymmetry in the Disconfirmation of Expectations for Natural Yogurt. Appetite, 32(3), 307-329. doi:10.1006/appe.1998.0208
Schiffman, S. (2000). Taste quality and neural coding: Implications from psychophysics and neurophysiology. Physiology & Behavior, 69(1-2), 147-159. doi:10.1016/s0031-9384(00)00198-0
Schoning, J., Rogers, Y., & Kruger, A. (2012). Digitally Enhanced Food. IEEE Pervasive Comput.,11(3), 4-6. doi:10.1109/mprv.2012.40
Shepherd, D. (1985). Dietary salt intake. Nutrition & Food Science, 85(5), 10-11. doi:10.1108/eb059082
Simner, J., Cuskley, C., & Kirby, S. (2010). What sound does that taste? Cross- modal mappings across gustation and audition. Perception, 39(4), 553-569. doi:10.1068/p6591
76
Simner, J., Cuskley, C., & Kirby, S. (2010). What sound does that taste? Cross- modal mappings across gustation and audition. Perception, 39(4), 553-569. doi:10.1068/p6591
Simons, C., & Noble, A. (2003). Challenges for the sensory sciences from the food and wine industries.Nature Reviews Neuroscience, 4(7), 599--605.
Sinclair, A., Morley, J., Vellas, B., & Pathy, M. (2012). Pathy's principles and practice of geriatric medicine (5th ed., pp. 1061-1072). Chichester [England]: Wiley-Blackwell.
Small, D., & Prescott, J. (2005). Odor/taste integration and the perception of flavor. Experimental Brain Research, 166(3-4), 345--357.
Small, D., & Prescott, J. (2005). Odor/taste integration and the perception of flavor. Experimental Brain Research, 166(3-4), 345--357.
Sova, R., & Sova, D. (2006). Storyboards: a dynamic storytelling tool.
Spence, C. (2012). Auditory contributions to flavour perception and feeding behaviour. Physiology & Behavior, 107(4), 505-515. doi:10.1016/j.physbeh.2012.04.022 Spence, C., Okajima, K., Cheok, A. D., Petit, O., & Michel, C. (2015 submitted). Eating with our eyes: From visual hunger to digital satiation. Brain & Cognition. Spence, C. (2015). On the psychological impact of food colour. Flavour, 4(1). doi:10.1186/s13411-015-0031-3
Spence, C. (2015). On the psychological impact of food colour. Flavour, 4(1). doi:10.1186/s13411-015-0031-3
Spence, C., & Piqueras-Fiszman, B. (2013). Technology at the dining table. Flavour, 2(1), 16. doi:10.1186/2044-7248-2-16
Spence, C., & Piqueras-Fiszman, B. (2014). The perfect meal. Wiley-Blackwell.
Spence, C., & Shankar, M. (2010). The influence of auditory cues on the perception of, and responses to, food and drink. Journal Of Sensory Studies, 25(3), 406-- 430.
Spence, C., & Zampini, M. (2006). Auditory contributions to multisensory product perception. Acta Acustica United With Acustica, 92(6), 1009--1025.
Spence, C., Shore, D., & Klein, R. (2001). Multisensory prior entry. Journal Of Experimental Psychology: General, 130(4), 799.
77
Stein, B., & Meredith, M. (1993). The merging of the senses. Cambridge, Mass.: MIT Press.
Stevenson, R. (2009). The psychology of flavour. Oxford: Oxford University Press.
Stillman, J. (2002). Gustation: intersensory experience par excellence. Perception, 31(12), 1491-1500. doi:10.1068/p3284
Stuckey, B. (2012). Taste what you're missing. New York: Free Press.
Taylor, A. (2012). Eating using all your senses. BBC. Retrieved from http://www.bbc.co.uk/food/0/20253159
Tindall-Ford, S., Chandler, P., & Sweller, J. (1997). When two sensory modes are better than one.Journal Of Experimental Psychology: Applied, 3(4), 257-287. doi:10.1037//1076-898x.3.4.257
Todrank, J., & Bartoshuk, L. (1991). A taste illusion: Taste sensation localized by touch. Physiology & Behavior, 50(5), 1027-1031. doi:10.1016/0031- 9384(91)90432-n
Tsakiris, M., & Haggard, P. (2005). The Rubber Hand Illusion Revisited: Visuotactile Integration and Self-Attribution. Journal Of Experimental Psychology: Human Perception And Performance,31(1), 80-91. doi:10.1037/0096-1523.31.1.80
Tuorila, H., Cardello, A., & Lesher, L. (1994). Antecedents and Consequences of Expectations Related to Fat-free and Regular-fat Foods. Appetite, 23(3), 247- 263. doi:10.1006/appe.1994.1057
Unilever,. The science behind ice cream's magic | Our R&D centres | Unilever. Unilever.co.uk. Retrieved 1 June 2015, from http://www.unilever.co.uk/innovation/researchanddevelopment/thesciencebehin dicecreamsmagic.aspx
Van Essen, D., Anderson, C., & Felleman, D. (1992). Information processing in the primate visual system: an integrated systems perspective. Science, 255(5043), 419-423. doi:10.1126/science.1734518
Verhagen, J., & Engelen, L. (2006). The neurocognitive bases of human multimodal food perception: Sensory integration. Neuroscience & Biobehavioral Reviews, 30(5), 613-650. doi:10.1016/j.neubiorev.2005.11.003
78
Vickers, Z. (1983). Pleasantness of Food Sounds. Journal Of Food Science, 48(3), 783-786. doi:10.1111/j.1365-2621.1983.tb14898.x
Vroomen, J., & De Gelder, B. (2004). Perceptual effects of cross-modal stimulation: Ventriloquism and the freezing phenomenon. The Handbook Of Multisensory Processes, 3(4), 1--23.
Wakita, T., Ueshima, N., & Noguchi, H. (2012). Psychological Distance Between Categories in the Likert Scale: Comparing Different Numbers of Options. Educational And Psychological Measurement, 72(4), 533-546. doi:10.1177/0013164411431162
White, T., & Prescott, J. (2007). Chemosensory Cross-Modal Stroop Effects: Congruent Odors Facilitate Taste Identification. Chemical Senses, 32(4), 337- 341. doi:10.1093/chemse/bjm001
Wobbrock, J., Findlater, L., Gergle, D., & Higgins, J. (2011). The aligned rank transform for nonparametric factorial analyses using only anova procedures (pp. 143--146).
Wobbrock, J., Findlater, L., Gergle, D., Higgins, J., & Kay, M. (2015). ARTool. Depts.washington.edu. Retrieved 24 August 2015, from https://depts.washington.edu/aimgroup/proj/art/
Woods, M. (1998). Symposium on ‘Taste, flavour and palatability’ Taste and flavour perception. Proc. Nutr. Soc., 57(04), 603-607. doi:10.1079/pns19980088
Zampini, M., & Spence, C. (2004). The role of auditory cues in modulating the perceived crispness and staleness of potato chips. Journal Of Sensory Studies, 19(5), 347--363.
Zampini, M., Sanabria, D., Phillips, N., & Spence, C. (2007). The multisensory perception of flavor: Assessing the influence of color cues on flavor discrimination responses. Food Quality And Preference, 18(7), 975-984. doi:10.1016/j.foodqual.2007.04.001
Zhang, E. (2015). Adrian David Cheok and mixed reality research featured in City University Research Publication. Mixed Reality Lab. Retrieved 1 June 2015, from http://mixedrealitylab.org/adrian-david-cheok-and-mixed-reality-research- featured-in-city-university-research-publication/
Zimmerman, J., Forlizzi, J., & Evenson, S. (2007). Research through design as a method for interaction design research in HCI (pp. 493--502).
79
Zimmerman, J., Stolterman, E., & Forlizzi, J. (2010). An analysis and critique of Research through Design: towards a formalization of a research approach (pp. 310--319).
80
APPENDICES
Appendix 1
81
Appendix 2
82
Appendix 3
83
Appendix 4
84
Appendix 5
85
Appendix 6
86