The Implicit and Explicit Memory of Static and Interactive Objects in Virtual Reality

DEGREE PROJECT IN MEDIA TECHNOLOGY, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2019

The implicit and explicit memory of static and interactive objects in virtual reality

PETRA OLSSON

KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE The implicit and explicit memory of static and interactive objects in virtual reality

ABSTRACT Virtual reality applications are developing fast and break new grounds every day in new industries. In today's market, most of the virtual reality projects are funded by venture capitalists and are yet to produce any revenue. For an advertiser to be willing to advertise they must see some sort of value or possibility to receive revenue out of the intended commercial. Memory is a common measure of influence from a commercial. Therefore, this thesis aim was to investigate if the memory, in terms of implicit and explicit memory, was affected by objects with different levels of interaction in virtual reality. A user study was conducted where the participants were instructed to go through several virtual rooms, where they were exposed to static, semi- interactive and interactive objects in the categories fruits/vegetables and beverage cans. The static objects were presented as pictures on instruction boards, the semi-interactive objects were presented through point-and-click tasks, and the interactive objects were presented through tasks that utilized body movements in virtual reality.

When the participants were finished with the virtual reality application, they conducted a survey regarding their experiences and was asked to fill out three memory tests; a word-fragment completion task, a cued recall test, and a recognition task. Results show that interactive and semi-interactive objects in virtual reality did have a significant impact on explicit memory, and that the static objects did not. For implicit memory a conclusion could not be drawn whether the objects in the virtual environment had any impact due to no significant difference between the study participants and the control group without influence of the study as a result of not enough data. The results from the study were vastly different for fruits/vegetables and beverage brands, where results for the fruits/vegetables had a clear distinction and the beverage brands had almost no significant data, which could be due to “Inattentional blindness”.

Det implicita och explicita minnet av statiska och interaktiva objekt i en virtuell verklighet

SAMMANFATTNING Virtuella verklighets (Virtual reality) applikationer utvecklas snabbt och bryter ny mark varje dag i nya industrier. På dagens marknad, så är de flesta virtuella verklighets projekten finansierade av riskkapitalister och är fortfarande i ett läge där de inte producerar intäkt. For att marknadsförare ska vara villiga att investera måste de se någon potential i att få intäkter från den avsedda reklamen. Minne, är ett vanligt mått för att mäta påverkan av en reklam. Därav var avsikten for denna avhandling att undersöka om minnet, i form av implicit och explicit minne, blev påverkat av objekt med olika nivåer av interaktion i en virtuell verklighet. Ett användartest utfördes där deltagare blev instruerade att gå genom flera virtuella rum, där de blev exponerade för statiska, semi-interaktiva och interaktiva objekt i kategorierna frukter/grönsaker och dryckesmärken. De statiska objekten representerades som bilder på en statisk instruktionstavla, dom semi-interaktiva objekten representerades genom en peka-och- klicka uppgift, och de interaktiva objekten var representerade genom uppgifter som utnyttjade kroppsrörelser i en virtuell verklighet.

När deltagarna fullföljt den virtuella verklighets applikationen fick de fylla i en enkät angående deras upplevelser, samt fylla i tre minnes test; ett ord-fragmentkompletterings test, ett indikation återkallelsetest, och ett igenkännings test. Resultaten visar på att de interaktiva och de semi-interaktiva objekten i den virtuella verkligheten hade en signifikant påverkan på det explicita minnet, samt att de statiska inte hade det. För de implicita minnet kunde inga slutsatser dras huruvida objekten i den virtuella verkligheten hade någon inverkan på grund av ingen signifikant skillnad mellan studiedeltagarna och kontrollgruppens resultat som en efterföljd av otillräckliga data. Resultaten från studien hade olika utfall för frukter/grönsaker och dryckesmärken, där resultaten för frukter/grönsaker hade en klar distinktion, medan dryckesmärkena hade nästan ingen signifikant data, vilket kan bero på ”Ouppmärksam blindhet” (Inattentional blindness).

The implicit and explicit memory of static and interactive objects in virtual reality

Petra Olsson Division of Media Technology and Interaction Design EECS, Royal Institute of Technology Stockholm, Sweden [email protected]

ABSTRACT market, most of the VR projects are funded by venture capitalists and are yet to produce any revenue [31]. The ones who have already Virtual reality applications are developing fast and break new managed to publish their product on the market tend to use a grounds every day in new industries. In today's market, most of the payment up front or a subscription fee to produce revenue [7]. This virtual reality projects are funded by venture capitalists and are yet gives the market a narrow economic space to operate within, and as to produce any revenue. For an advertiser to be willing to advertise the market is expanding, it will likely search for new possibilities they must see some sort of value or possibility to receive revenue to raise money. The most presumable solution will likely be out of the intended commercial. Memory is a common measure of advertising, and it was only recently that advertisements in the form influence from a commercial. Therefore, this thesis aim was to of static billboards started taking place within the VR world [26]. investigate if the memory, in terms of implicit and explicit memory, was affected by objects with different levels of interaction in virtual As of 2014, more and more high-performance VR devices aimed reality. A user study was conducted where the participants were towards the consumer market have been announced [2]. The market instructed to go through several virtual rooms, where they were is forecasted to 6 million VR device unit shipments during 2019, exposed to static, semi-interactive and interactive objects in the with an estimated shipment of 4,7 million in 2018 and 3,7 million categories fruits/vegetables and beverage cans. The static objects shipped units in 2017 [28]. Also, in 2017, VR consumer software were presented as pictures on instruction boards, the semi- had a revenue of $554 million, where games accounted for about interactive objects were presented through point-and-click tasks, 55% of that [29]. and the interactive objects were presented through tasks that For an advertiser to be willing to advertise they must see some sort utilized body movements in virtual reality. of value or possibility to receive revenue out of the intended When the participants were finished with the virtual reality commercial. Memory is a common measure of influence from a application, they conducted a survey regarding their experiences commercial: “Advertisers and those who measure the impact of and was asked to fill out three memory tests; a word-fragment advertising are obsessed with memory. If advertising is to be completion task, a cued recall test, and a recognition task. Results successful, it must stick in the consumer’s memory—or so the show that interactive and semi-interactive objects in virtual reality saying goes” - David Brandt [3]. We can primarily divide the did have a significant impact on explicit memory, and that the static human memory into two categories, implicit (information that we objects did not. For implicit memory a conclusion could not be are not consciously aware of) and explicit (the facts and events that drawn whether the objects in the virtual environment had any we can consciously access). Nelson [22] proposed that increasing impact due to no significant difference between the study brand awareness is one of the goals of product placement, and that participants and the control group without influence of the study as a suitable measure for effectiveness would be to look upon explicit a result of not enough data. The results from the study were vastly memory-based measures such as recognition and recall. But different for fruits/vegetables and beverage brands, where results implicit memory has also been proposed as an important concept in for the fruits/vegetables had a clear distinction and the beverage advertising research [6]. Since explicit memory tests only can brands had almost no significant data, which could be due to reveal effects of conscious retrieval of an advertisement, implicit “Inattentional blindness”. memory effects are important as well. Even consumers purchase decisions could be influenced not only by conscious processes, but Key words by unconscious processes, especially at the time of purchase [6]. Virtual Reality; Implicit memory; Explicit memory; Advertising; An act of memory depends on the impression perceived from the act, the kind of intensity it had and the associations that could be 1. INTRODUCTION made from it. There are several different impressions that can be Virtual reality (VR) applications are developing fast and break new made, such as visual, auditory, motor, tactual, gustatory and grounds every day in new industries such as the automotive olfactory [16]. Additional sensory input can increase the memory industry, healthcare, retail, tourism, real estate, education etc. [30]. of objects in a VR environment [5]. Creating a product placement VR is being adopted and utilized in many new industries but have in a VR game, where the user can not only interact with the objects had a special impact on the gaming industry where the players are but also increase their multi-sensory input of the object could boost enabled to be immersed in a game in a new way [8]. In today's the chance of the user remembering it. But also utilize the 1 possibilities that VR provides beyond billboards and static 2.1.2 Recall and recognition of objects and words advertisements. This could result in higher value for the The human memory is better at remembering pictures than words advertisements, as VR can engage users in interactions that are in terms of recognition. Pictures engage more activity in some more interactive than a, for example, semi-interactive point and memory-related regions of the brain, which result in more superior click advertisement in a usual video game. Therefore, this thesis or more efficient engagement for pictures than words [10]. aims to investigate if the interactions with the virtual objects may Moreover, effects of words and pictures on a recognition task differ have an impact on the memory of it, and thus generate additional whether the recognition is performed from picture-to-picture, value for an advertiser. word-to-word, picture-to-word or word-to-picture. Meaning, being Research question: What is the difference, in terms of implicit and exposed to a picture then asked if any picture is recognized between explicit memory, of static, semi-interactive and interactive objects a couple of pictures etc. Comparing the performance between these in virtual reality? recognitions task in a study by Jenkins, et al [14], they found that picture-to-picture recognition was far superior to word-to-word 2. BACKGROUND recognition, and that picture-to-word compared with word-to-word was visually equal but significantly superior to word-to-picture. 2.1 Memory They also found that their hypothesis that pictures are encoded and stored with their verbal label is supported by the comparison of 2.1.1 Implicit and explicit memory performance between word-to-word and picture-to-word. In all the The human memory can be divided into two categories, explicit and comparisons they made between the groups, the means are highly implicit memory. Explicit memory (also called declarative similar. Looking at the decrement in performance of the picture-to- memory) is the conscious or intentional recall of factual word group attributed to the change in stimulus mode surprised information, previous experiences and concepts [32]. Explicit them. They state that it may be concluded “that recognizing the memory arises after an exposure event. At the time of exposure, a word labels of pictures is as easy as recognizing the words representation is encoded in memory, and associated with at spatio- themselves“ (p. 307). termal context that links to the exposure incident. Explicit memory is the conscious recollection of an event and represents what he or Comparing recall for pictures and words we can see the same effect she remembers of it. To measure a person's explicit memory of an where pictures are more effectively stored in or retrieved from event, you test whether the person can make a direct reference to long-term memory [25]. In a study by Kirkpatrick [16], participants the past event and the person can also be asked to demonstrate their were exposed to words written on a board at a rate of a new word knowledge of it [18]. every two seconds. They were also exposed to objects held in front of them every two seconds. Looking at the average recall of these Explicit memory can be measured through recognition, free recall items two days later they found that objects were remembered and cued recall tests. A recognition task contains both a target better than the written names. stimulus and a context cue, asking the respondent to determine whether they have seen the target stimulus in an exposure context 2.2 Advertising in games previously or not [27]. A common execution is listing randomly objects including the right one, then asking the respondent to The first product placement in a computer game, was in the game choose the right one [12]. Through free recall, the respondent is Adventureland, promoting it’s followed up-game Pirate Adventure asked to state whatever he or she remembers seeing of the exposure in 1978 [20]. Compared to television and film, where product event without any additional cues. While in cued recall, the placement can appear disturbing, players tend to feel more positive respondents are presented with an additional cue to help about the product placement in games where they enhance the sense respondents access the memory trace, such as a category [27]. of realism to the game, for example in a racing game [22]. For advertisers, one strong argument for advertising in video games is Implicit memory (a part of procedural memory), is something that the hard-to-reach demographic of 18- to 34-year-olds who is well generally is not available to conscious access but can still affect represented among video- and computer game players [19]. thoughts and behaviors [32]. Unlike explicit memory, implicit memory can also appear as an improved performance on 2.2.1 Interaction with advertisements subsequent tasks, without any direct recollections of past events. Within the advertising industry, advertisers have traditionally People can thus be influenced by previous exposed information divided types of advertising into two groups, passive and active. without consciously attempting to retrieve that information. Where broadcast media such as TV and radio have been considered Improved performance as a result of prior exposures implies that passive since the consumer receives the message from the people do have a memory of the previous exposure event, even advertisement passively and does not actively chooses to listen to though they cannot consciously indicate what they remember [18]. or view the message. While other media, print, such as outdoor Respondents can be asked to complete a task, utilizing their prior billboards, newspapers and magazines are considered active. This knowledge, without information beyond the task at hand, and still because of the active, conscious decision of viewing and be influenced by their implicit memory. Examples include consuming the advertisement [15]. Interaction is defined as an sentence-completion or word stem completion tasks in which episode, or series of episodes of reactions and physical actions of a memory tests for the target stimulus are embedded without human with the world, including the environment, beings and reference to the exposure episode. For testing brand names, a word objects in the world [11]. In a study by Michael Girgis [9] on the stem completion task is most appropriate [27]. memory of interactive and static billboards, he found that an interactive billboard does have a significant difference in recall but not in recognition when compared to static billboards.

2.2.2 Memory and in-game advertisement guided through several virtual rooms where they received, and The memory of a brand in product-placement or advertisement performed different tasks with the static, semi-interactive and placed in movies or television have given mixed results where some interactive objects placed out in the VE. Great emphasis was put brands found little improvements on memory and some found into selecting target objects for this study. The categories beverage remarkable improvements [24]. Looking at in-game brands and fruits/vegetables were selected due their simplicity to advertisements, two studies show that players recalled 25% to 30% recognize and their part in our everyday life. A set of respondents of the brand immediately after playing and 10% to 15% five months were asked to produce an item-pool, listing all the objects they after [22]. One study on real-life brands advertisements on could think of within the categories. This pool-of-items where then billboards in videogames shows that in-game advertisement does used as a basis in the selection process for which items to be influence implicit memory of the brands, while the explicit memory selected. In this study we defined interaction with static objects as does not have an equally large effect [33]. objects you cannot interact with, only consume passively. Interactions with semi-interactive objects as a point-and-click 2.3 Virtual reality and cross reality interaction, where you can interact with the objects but in a VR is part of the concept Cross Reality (XR), also known as confined manner. Lastly, interaction with interactive objects where Extended Reality. XR is a term that includes all real-and-virtual you will be able to “fully” interact with the objects using all the combined environments that is generated by wearables, such as senses that VR enables. head mounted displays (HMD) and computer technology [21]. This Before the study, a pilot study was performed to observe any issues includes VR, augmented reality (AR) and mixed reality (MR). with design and to reassure useful data was collected. In addition VR is a computer-simulated alternative reality. It is usually to the VR application, a questionnaire regarding background provided through HMD. VR provides a Virtual Environment, VE, variables, presence and memory was designed. Background where the user is totally immersed in, and able to interact with the variables included age, gender and previous experience with VR, VE and objects in it [21,23]. The range of VR HMD on the market and a presence questionnaire from Igroup [13] was used. To can be categorized into two groups according to performance. measure memory, three different methods were utilized. A word- Looking at the high-performance category, we have HMDs such as fragment test for measuring implicit memory was designed, as well HTC Vive and Oculus Rift. These headsets enable tracking in six as a recognition test and cued recall for measuring explicit memory. degrees of freedom (6DOF), which refers to the movement of a An assumption was made that the overall available recruits had rigid body in three-dimensional space. The body is free to change Swedish descent, and that they would therefore naturally think in position in forward/backward (surge), up/down (heave) and the Swedish. As a design decision, and to facilitate the memory tests, left/right (sway) direction. This translates to three perpendicular Swedish was therefore set as the primary language used throughout axes, and combined with changes in orientation through rotation, the study. The participants were not informed about the memory often termed yaw, pitch, and roll. While the other, low performance aspect of the study until after the questionnaire was filled out. This category, has HMD such as Google Cardboard, Samsung Gear and was to prevent participant bias, trying to “perform well” on the test, Oculus Go that enables only three degrees of freedom (3DOF) and for them to not actively trying to memorize what they saw. tracking. This means that the tracking is only made in three Simultaneously a control group were recruited to take the same dimensions, when you look up, down and sideways, excluding word-fragment test to give grounds for analyzing the effect on the rotation. study group’s implicit memory. After the study was carried out, the results were analyzed through several different methods such as Traditional media does not acknowledge the spatiality of their statistical diagrams, correlation calculations and ANOVA analyses. consumers. Those media such as television and radio exist in the With ANOVA, the null hypothesis that there is no difference physical world where their users are, but the users do not exist in between the groups is assumed. With an F bigger than Fcrit, the their space. However, communication technologies allow us to null hypothesis can be rejected. experience spaces we could not access before, they can invoke a sensation of presence [11]. With XR, the users can not only 3.1 Procedure experience the advertisements in a new way, but also enables the In the study, the participants were instructed to move through experiences to use the spatiality of the consumers. several virtual rooms. Every room (session) was timed, as well as 2.3.1 Multi-sensory input the total time spent in the VE for each of the participants was In VR the sensory cues most primarily consist of visual stimuli. collected. In some rooms they received instructions for the room However, it is sometimes accompanied by audio stimuli and ahead, and in the next they performed some sort of task. The first occasionally haptic stimuli. As we move around a room scaled VE room was equipped with a few different interaction possibilities, interacting with the objects in the environment we are also for the users to get to know the controllers before starting the actual stimulated by motor sensation. study. Through the instructions, the participants were exposed with static objects as either fruits/vegetables or beverage brands on 3. METHOD aluminum cans as pictures on a board. Using the static objects on To examine the implicit and explicit memory of static, semi- instruction boards, was a design decision to assure that the static interactive and interactive objects in VR, a user study was objects appeared in the field of view of the users and that they performed. The study was formed according to information perceived the static objects. The participants were exposed to the retrieved through a literature study. A VR application was same static objects two times but as different instructions for the developed iteratively through two sprints and tested with users. The tasks. The fruits/vegetables or beverage brands used in one task was VE were designed constricted and rather hollow to reassure the replaced with new ones for the second task. Participants were participants stayed focused on the task at hand. Participants were instructed to take part of the given instructions in each scene, and when understood, press a button to start the task. 3

When finished, they were encouraged to press a button with the statement “next” to move on to the next task. The order went: 1. Static instructions for sorting fruits/vegetables, see Picture 1. 2. Semi-interactive point and click sorting fruits/vegetables, see Picture 3. 3. Static instructions for sorting fruits/vegetables. 4. Interactive throwing fruits/vegetables to sort them, see Picture 4. 5. Static instructions for knocking down cans. 6. Semi-interactive point and click knocking down cans, Picture 1 & 2. Static instructions that were placed on a wall in see Picture 5. the VE. 7. Static instructions for knocking down cans, see Picture 2. 8. Interactive using a bow and arrow to knock down cans, see Picture 6. The participants were given time to freely explore the interactions in all sessions. After finishing the VR part of the study, the participants were asked to fill out a survey of their experiences. The study took approximately 15-25 minutes for each participant in total. The questionnaire was built so that the different sections would not interfere with each other. The first section covered Picture 3. Point and click sorting fruits/vegetables task. background variables and experience with VR, they then moved on to a section about presence and immersion in the study. These were placed first so that the short-term memory was cleared prior to the word-fragment test. They then started with the memory tests; a word fragment test first, followed by a cued recall test, and lastly a recognition task. 3.2 Choice of interaction tasks For the interaction with static objects, an instruction board was selected. By taking part of the instructions given, the users visual field was forced to the static objects, ensuring they had been Picture 4. Throw to sort fruits/vegetables task. perceived, see Picture 1 and 2. The interaction with the semi- interactive and interactive objects tasks were selected to be of the same nature, but with different methods of interaction. For the semi-interactive objects, a point-and-click approach was selected. The users were thus forced to interact with the objects, but without utilizing VRs full potential in terms of body range. What could be equated with a VR experience with 3DOF tracking. And for the interactive objects, tasks that required the users to use their full body movements was selected to utilize the capacity of 6DOF tracking. With the selected categories in mind, a few interaction tasks were brain-stormed such as puzzles, sorting tasks, and simpler sport activities. Two constricted tasks were selected to minimize development time. A color-sorting task were selected for the fruits Picture 5. Point and click knocking down cans task. and a tin can alley task were selected for the beverage brands. The colors green, red and yellow were selected for the sorting task in accordance with many appearing’s of these colors on fruits and vegetables. In the point-and-click session, the participants were instructed to point on a fruit and click to select it, then repeat the procedure on the barrel they wanted to put the fruit in, see Picture 1 and 3. For the interactive session, the participants were instructed to pick up the fruits and throw them in the matching barrel, see Picture 4. A bow and arrow were selected for the interactive tin can alley session, this to invoke a new kind of interaction rather than throwing a ball, something that were already utilized in the sorting task, see Picture 6. For the point-and-click tin can alley session, the participants were instructed to aim towards the cans and pull the Picture 6. Bow and arrow knocking down cans task. trigger to shoot a ball, see Picture 5. 4

3.3 Choice of fruits/vegetables and beverage down to fewer questions to suit this study and translated to brands Swedish. Fruits/vegetables and beverage cans was selected as categories in The IPQ has three subscales; accordance with their simplicity to recognize, their degree of 1. Spatial Presence - the sense of being physically present appearances in our daily lives, and their equality in exposures in in the VE. everyday life. They are also equal in terms of interactions that can 2. Involvement - measuring the attention devoted to the VE be made to them, as well as size. One thing separating them is that and the involvement experienced. one of the categories often is only perceived as an object, while the 3. Experienced Realism - measuring the subjective other is tightly associated with a brand. experience of realism in the VE. Firstly, four participants from the to-be-sampled population were As well as one additional general item not belonging to one of the recruited to fill in a short questionnaire where they were asked to subscales; the general “sense of being there”. Which has a high list all the fruits/vegetables they could think of as well as all impact on all three subscales, especially Spatial presence. beverage brands to generate objects that relate to the focal The participants were asked to answer the different statements on a construct. This method was primarily utilized for constructing scale from one to five. The study utilized one Spatial presence, two sufficient enough words for the word-fragment completion test[17], Involvement and one Experienced Realism statements. as well as to reassure that the selected items are common and Additionally, the participants were given the opportunity to freely recognizable among the target group participants. The words leave comments on their sense of presence. collected were used as an item-pool to choose the final objects for the study. 3.5 Memory tests Firstly, the fruits/vegetables were selected from the item-pool according to color, making sure that they fit in the color scheme 3.5.1 Word fragment completion test of implicit used for the sorting task. Only fruits with clear color distinction memory were selected, fruits like Mango were rejected due to their mix in The participants were asked to fill in the blanks from constructed colors. Secondly, these were sorted out according to shape to avoid words such as _ e p _ _, for the word “Pepsi”. When constructing confusion. For example, a round yellow fruit could be an orange or fragments, the goal is to ensure sufficient variance in responses and several other citrus fruit. Lastly, they were selected after available not create words that produce too high response frequency nor to 3D objects and icons online. Berries was mainly taken for the static low. The difficulty in generating a word from a fragment is objects, since these would appear far too small as real objects in the dependent on the location of the missing letters. It is easier to VE. Icons with a similar design were selected, as well as 3D objects generate a word for a fragment where letters are removed from the with similar visual appearances. The set were changed several end of the word rather than in the beginning of the word. It is also times as one decision in the beginning could result in no available important to mix up the construct of words so that all fragments are 3D objects in the end phase, so the selection process was iterated not similar. After a few fragments had been set up, they were tested several times. The final set were as given; through a crossword solution tool online. Each fragment was tested Statics: Pineapple, cherry and broccoli. to make sure that a variety of words could fit [17]. The location of the blanks where then strategically replaced so that they would fit Semi-interactive: Banana, kiwi and tomato. into a lower boundary of at least 10 available Swedish words, and Interactive: Pear, watermelon and corn. a limit of not more than 100 available words for each fragment. Additionally, two fragments were created for each of the target For the beverages brand, the pool of items was first scaled down words to be tested through the pilot study so that the fragment with and deprived from strong brands such as Coca-cola to minimize the most desirable difficulty and variability could be used for the actual effects of brands with high brand awareness, since this would give study. that brand an unnecessary advantage. Just the red color on a Coca- cola can could trigger brand recognition for example. To assure an The participants were encouraged to skip a word-fragment if they equivalence in brand awareness between the nine brands were very did not immediately see a word that could fit. This due to the fact hard. Therefore, a decision was made to take brands with fairly that the responses are more indicative of a participants vocabulary equal brand awareness or brands with equal target groups. Three rather than the accessibility of the target content at an implicit level sparkling water brands was selected due to their equality in target if the participant tries to fill out the word-fragment through a group, three notorious Swedish brands and three popular deliberative trial and error rather than leaving it empty if a word international brands. These were distributed among the three does not immediately unfold [17]. They were also informed that conditions. They were then modeled in the 3D software Blender. once they had put down a word, they were not allowed to go back The set were given as; and change it. Statics: Sprite, Trocadero, Ramlösa All nine of the fruits/vegetables and all nine beverage brands were used in the word fragment completion tests, as well as three foils, semi-interactive: Fanta, Mer, Bonaqua resulting in 21 words to complete. The foils were included to Interactive: Pepsi, Zingo, Loka minimize the risk of participants developing a self-generated validity effect that can occur independent of the participants actual 3.4 Presence Questionnaire standing on the construct of interest [17]. Meaning, if they start For testing how present the users felt in the VR environment, a seeing a pattern in the word fragments, it may lead to a peak in presence questionnaire was used for evaluation [13]. Igroup’s presence questionnaire (IPQ) original questionnaire were broken 5

Picture 7. Example of Swedish instructions. performance once they have received a particular mindset. The foils were all in accordance with the theme nature. The words “Cloud”, Figure 1. Time spent in seconds in each session. “Rock” and “Flower” were used, as well as “Tree” in the example fragment. These fragments were constructed to be easier on 3.7 Pilot Study purpose to clear the minds of the participants. The foils were also The goal of the pilot study was to get some user feedback on the distributed evenly amongst the goal fragments. application as well as finding obvious shortcomings and evaluate if A control group was recruited to take the word fragment test the gathered data was useful. No major issues were found in the without being a part of the study or be exposed to the study. The pilot study was also a possibility to look over the fruits/vegetables and beverage brands as a baseline for the validity fragments that had been created for the word-fragment test. The of the test. The participants received a link to an online fragments that performed undesirably like “cherry” and “broccoli” questionnaire and were able to fill it at any time, and wherever they were exchanged for the alternative fragment before the study. wanted. 3.8 Instruments 3.5.2 Explicit memory tests The test was set up with Acer’s Windows Mixed Reality Headset. The participants were tested through two explicit memory tests, one The Acer headset enables room-scale tracking with 6DOF tracking cued recall test and one multiple choice recognition task. In the within an HMD and a Camera field of view of 100° [1]. The HMD cued recall test the participants were asked to freely recall all the was set up with the gaming laptop Lenovo Legion Y720 with an fruits and/or vegetables, and beverage brands they remembered Intel Core i7-processor, NVIDIA GeForce GTX 1060-graphic card seeing in the VE as objects or pictures. In the multiple-choice and Windows 10. recognition test each question contained one “right” answer as well Both the HMD and computer were selected due to their easiness to as two foils, resulting in 27 fruits/vegetables and 27 beverage transport and easy set-up, so that the study could be conducted brands in total. The participants were also given the alternative to independent of location. The Acer headset was also particularly leave a question blank if they did not recognize any of the items. selected in accordance with its 6DOF tracking. The study highly The foils were also selected from the pool-of-items generated. relied on the motor sensation that was induced during the tasks, 3.6 Choice of Target group therefore an HMD with 6DOF over one with 3DOF was required. For the recruitment of the study a target group was set. The first The study and VE was developed in Unity 2018.3.6 with SteamVR prerequisite for the target group was that the participants needed to Unity Plugin - v2.2.0 to enable VR. have no considerable sight problems, so that they could use the HMD comfortably, as well as not having any physical movement 4. RESULTS AND ANALYSIS limitations so that they could perform all the tasks in the VE. There In total, 30 people participated in the study over a two-week period was also a requirement to have tested VR at least once before, for where 19 (63%) were male and 11 (37%) were female. The age the controllers to be more intuitive and to be able to navigate in the span was between 20 and 33, with a median of 24. All participants VE without any further instructions. The last requirement was that had previous experience with VR, whereas 20% used VR regularly, the participants needed to be fluent in Swedish. Most of the 43% had used it a couple of times and 37% had only used it once available recruits for this study were from Sweden, and therefore or twice. an assumption that most participants would naturally think in The participants had an average total time in the VE of 216 seconds Swedish was drawn. Taking part of the study, watching the and a median of 213 seconds. The participant with shortest total fruits/vegetables, the assumption is that people will think of the time in the VE had 113 seconds, and the participant with the longest Swedish words for what they see, and for simplicity, the entire total time had 300 seconds. In Figure 1 you can see a box plot of study was only constructed in Swedish, due to that assumption. To the time spent in each session. The session with uneven numbers is set the mindset for the entire study all instructions where given in the sessions where the participants are asked to read instructions. Swedish, and Swedish text were placed in the VE, see Picture 7. We can see a clear decrease in time spent understanding the instructions for every new session. We can also see that the average

Figure 2. Average score per user. time spent on sorting fruits/vegetables (Session 2 and 4) was Figure 3. Fragment-test score for fruits/vegetables and relatively equal despite being two different interactions, semi- beverage brands. interactive and interactive. Average time for Session 2 was 55 seconds and average time in Session 4 is 49 seconds. However, the target names were correctly completed by the participants was variance in time is quite different for the two where the max time analyzed using a one-way ANOVA with three conditions, static, in Session 2 was 138 seconds, and Session 4 has a max on 68 semi-interactive and interactive. As well as three one-way ANOVA seconds. analyses comparing each of the three previous conditions with the control groups performance. A one-way ANOVA with the three For the knocking down cans sessions (6 and 8) the average time conditions for static, semi-interactive and interactive was also diverges a bit, where the bow and arrow had a higher average time conducted for both only fruits/vegetables and only beverage of 40 seconds and the point and click had an average of 25 seconds. brands. The calculations are based on number of correctly filled out words where each participant got a score of one for correct word 4.1 Presence and zero for an incorrect word or blank. See the average score per From the IPQ the average responses were considerate on the upper user over the three interactive conditions in Figure 2. All half of the scale after compiling all the answers. This indicates that calculations are made with α=0,05, a significance level of 95%. most of the participants felt present in the VE. Looking first if we can see any significance between the study Participants described their “sense of being there” as both average participants answers and the control groups answers, over the three and median as a 4 on a scale of 1 to 5. There was a small deviation levels of interaction. Comparing the score for the static objects with of two people who selected 2 on the scale. The first proclaimed “I the control group we got an average A=0,83 and variance V=0,28 had a much harder time learning the controllers than I first of the participants answers in study, and A=0,73 and V=0,35 for expected”, and “it is hard to understand the weight of the objects the control group. F=0.33 and Fcrit=4,07, therefore we cannot when I am throwing the fruits”. The second proclaimed he had a reject the null hypothesis that the study participants and the control hard time feeling the “reality” of the constructed environment, “The group have similar answers. background feels like Unity standard”. For the semi-interactive objects we have A=1,33, V=0,51 for the For the Spatial Presence there was an average of 4,7 and a median study participants, comparing with the control group A=0,93 and of 4 on a scale of 5. There was one deviation of a participant who V=0,35. With F=3,51 and Fcrit=4.07 we cannot reject the null answered 1 on the scale. She did not leave any comments. The hypothesis that the groups have similar answers. However, with an remaining participants all answered 4 or 5 on the scale. α=0,1, we get F=3,51 and Fcrit=2,83 and we can reject the null Looking at Involvement we have a median of 3 and 4 and an hypothesis. We can see a significant difference between the control average of 3,2 and 3,6. Here participants left feedback like “the groups answers and the study groups answers with a significance world did not feel real, however it felt very natural”, “I was aware of 90%. of sounds and temperature but still felt surprised when I took off The last analysis is over the interactive objects, where the study the headset and realized I was in a different room”, “it started very group received A=0,5 and V=0,67 and the control group received unrealistic, but when I took off the headset I was still surprised by A=0,33 and V=0,67. With F=0,41 and Fcrit=4,07 we cannot reject my surroundings” and “that people talking in the background was the null hypothesis that the two groups have similar answers. a reminder of the real world”. We can therefore say, with the ANOVA analysis, that the implicit On Experienced Realism we had an average of 2.8 and a median of memory was not affected by neither the static nor the interactive 3. Some of the negative impacts that was mentioned by the objects. However, the semi-interactive objects affected the memory participants that affected the Experienced Realism was that the since we can see a significant difference in the answers between the aesthetics was not persuasive enough, problem with condensation study group and the control group. on the glasses and lack of tactile feedback. Dividing the score from the fragment-test between the 4.2 Implicit Memory fruits/vegetables and beverage brands, see Figure 3, we can see a For examining the implicit memory performance on both the big difference in implicit memory between the two. fruits/vegetables and beverage brands, the proportion that the 18 Fruits/vegetables scores an average of 44,4% while beverage brands scores 0,0%. We can therefore also conclude that there is 7

Figure 4. Recalled fruits/vegetables and beverage brands. Figure 6. Recognized fruits/vegetables and beverage brands. see a clear impact on memory that the fruits/vegetables in the semi- interactive and interactive conditions had. A couple of the participants listed “Apple” as a recalled fruit. This could be due to a mix up with the Tomato. 4.3.2 Recognition test For the analysis of recognition rates, the hit response rate was determined by calculating the proportion of correct recognized brand names appearing in each session. For example, if the participant correctly opted the fruit/vegetable or the beverage brand that was present in the study of the three alternatives, this was considered a hit, whereas a wrong answer is considered a miss. Lastly, they were also given the option to leave the answer blank, see Figure 5. Figure 5. Recognized fruits/vegetables and beverage brands. We can see a clear divergence between the recognition of the static only a significant difference between the control group and the objects and the objects that in some sort where interacted with study participants on the semi-interactive fruits/vegetables, and not (semi-interactive and interactive). The static objects only had a on the beverage brands. recognition rate of 30,0%. Whereas the semi-interactive had a recognition rate of 57,2% and the interactive had a rate on 61,7%. Some of the participants also managed to fill in words that are in The score of 30,0% on the static object may indicate that the the construct of interest (fruits and vegetables), but not target participants may have been guessing given that only four options words. Words that surfaced among the study participants were were provided per question. “Carrot” and “Gooseberry”, that may indicate a self-validity. Similar words also emerged amongst the control group participants. Comparing the number of hits between fruits/vegetables and beverage brands over the three variances static, semi-interactive 4.2.1 Control group and interactive we can see that the recognized beverage brands lies For the control group 15 people were recruited, where seven (47%) on a stable 30,0% over all three variances, see Figure 6. It can also were male and eight (53%) were female, with a similar age be seen that fruits/vegetables have a stable upward curve with distribution as the study group were ages ranged from 18 to 27, with 30,0% recognized static fruits, 84,4% semi-interactive fruits and a median of 24. 93,3% interactive fruits. Again, a score of 30,0% may indicate that the participants might have been guessing instead of leaving the 4.3 Explicit Memory question blank. 4.3.1 Cued recall test 4.4 Memory in Correlation with Time The participants were asked to freely recall all of the The average time in each session was calculated, and the two static fruits/vegetables they had seen in study and then also all of the sessions for both fruits/vegetables and beverage brands were beverage brands. The occurrences of the static, semi-interactive and merged to one. The average time was then divided by 100 to get it static fruits/vegetables and beverage brands was compiled on the same scale as the percentage rate. This was put in a combined separately, see Figure 4. For the beverage brands we can see a diagram with the memory rate in percentage. The results you can decrease in recall over the interactive parameter, with a score of see in Figure 7 below. The first three bars represent 21,1% for the static brands, 14,4% for the semi-interactive brands fruits/vegetables and the remaining represents beverage brands. and then 11,1% for the interactive brands. While we can see a clear We cannot see a clear correlation between time spent in each increase for the fruits/vegetables with 4,4% for static fruits, 72,2% session and the memory rate for the different memories in the for semi-interactive fruits and 80,0% for interactive fruits. We can combination diagram. Looking at the scatter diagram, Figure 8,

Figure 7. Combination diagram of percentage memory rate Figure 9. Recall, recognition and implicit memory of for the different sessions and it ́s time. interactive fruits. Another interesting remark that several of the participants did, was the fact that they could not recall seeing any beverage brands at all. “Ohhhh, I did not even see any brands!” Looking at their performance on the other memory tasks, they did not score at all on the word fragment test, except for one of the participants who managed to write “Pepsi”. Looking at the recognition rates, the average score was 0,7 recognized items for the participants who left the recall question blank, compared to the average of 4,5 for the remaining participants. Another interesting comment was one of the participants who bent over to look closer at the static instructions for the fruit sorting task and went ”Oh look, a pineapple”, and then left it out from the free recall. Even though he managed to fill out the correct answer on both the word fragment-test and the recognition task. Figure 8. Scatter Plot over time and memory in percentage. 4.6 Differences between male/female the same data is presented but sorted after time. We can still not see any clear correlation between time and memory for neither of the participants and experience from VR memories. We can therefore exclude the hypothesis that time spent A one-way ANOVA was done on the average score of the implicit with the interactions have any impact on the memory of it. memory, recall and recognition and the data were divided in two groups, male and female. In none of the cases was F bigger than 4.5 Feedback from Participants and Quotes Fcrit with 95% significance. We can therefore reject the hypothesis Many of the participants expressed confusion over the that there is any difference between male and female participants in fruit/vegetable sorting. They claimed that some of the fruits had two terms of all three memory conditions. colors and that they were confused over which color the fruit “had” Another one-way ANOVA was carried out on the same average in terms of sorting. “This was very hard, what do you define the score of the three different memory conditions, divided into three watermelon as, red or green?”, “Wrong color, what? Oh, they are groups depending on their previous experience with VR. In neither red!”, “Does the kiwi counts as green or brown?”, “Mhm, but is this of these cases was F bigger than Fcrit with 95% significance. We one green or red then?”, was some of the statements the participants can therefore reject the hypothesis that previous experience with left during the study. The fruit that left the most confusion was VR affected any of the three memory conditions. watermelon which was labeled as a red fruit in the study even though it had a green peel. Since performance was not a part of the 5. DISCUSSION study, and only the memory of the objects that was measured this This study aimed to investigate the memory of different objects inconvenient matter did not have any influence on the study itself. according to how interactive they were, and how the different Comparing the memory of the watermelon with the other fruits in memories were affected in different ways, with the research that session we can see a slight difference in the memory between question: What is the difference, in terms of implicit and explicit the three. See Figure 9. For recognition we cannot see any memory, of static, semi-interactive and interactive objects in virtual considerable difference, neither for recall where the differences are reality? The results tell us that the implicit memory is hardly minimal. However, we can see a considerable difference in the affected by the interactive objects we interact with. We could see a implicit memory of the three, where the watermelon scores 26,7% slight difference in implicit memory for the semi-interactive fruits and Pear and Corn only scores 6,7% respectively 0,0%. and vegetables that the participants were able to interact with, but no effect could be seen for neither static nor interactive objects. We could see a clear difference in the results comparing static objects impact with the interactive objects, both semi-interactive 9 and interactive. The recognition curve between hit, miss and blank of them are more popular. Looking at what we first learn as on the two is almost exactly similar. In Michael Girgis [9] study he children, it is not surprising that some fruits/vegetables are easier found that an interactive billboard did have a significant difference to recognize, has several more association cues, and thus are easier in recall rates but not on recognition rates when comparing with to recognize. static billboards. A reason for this could be the sensory inputs. In For the fragment-test, there is no surprise some fragments both studies visual stimuli and motor stimuli is accomplished, but performed better than others. For example, both the Swedish words in very different ways. In Girgis study the visual stimuli is solely a for pineapple (Ananas) and banana (Banan) are quite repetitive, and screen, while in this study the participants are surrounded by the easy to recognize. This may have facilitated the responses. Looking visual stimuli perceiving it as reality. The same for the motor at these particular words in the control groups answers, we could stimuli where in Girgis study they only use their hands to interact see that they were far superior in hit rate. with billboards, whereas in this study they can interact with the objects in several different ways. One aspect to also consider is the 5.2 Method Critique fact that this study utilized VR, an exciting technology that many The tasks given for the two different types of objects people feel astonished by. The mere fact that the study was carried fruits/vegetables and beverage brands may have affected the out in VR could boost the results. results. The interactions made with fruits demanded the participants That the static objects did not receive as high impact on memory to think before doing in accordance with a sorting task. While for could also be due to “Inattentional blindness” [4]. With the beverage brands they were only required to aim at their target Inattentional blindness Simons and Chabris suggest that we only and not give any more thought of it. It also gave a problem with perceive and remember the objects and details that receive focused near/far interaction. While the participants could actually lift the attention. In this case it could mean that the participants disregarded fruits/vegetables up and hold them closely, inspecting them if the pictures on the screen as not useful leading them to not pay any wanted, in the interactive session gave these objects a big attention to them at all. advantage. Even in the semi-interactive session the fruits/vegetables were lying much closer to the participants than for 5.1 Differences in Labels and Objects the beverage cans in both their sessions. What could have been Even if we could see a significant effect on the semi-interactive and done to solve this issue would have been to mix the tasks up interactive objects, this only applied to the actual objects, the fruits between the objects and divided the participants into two groups. and vegetables that were present in the VE. Looking at the beverage Making the objects less depending on the interaction task they were brands, they did not uphold with fruits/vegetables. For recall we coupled with. can see a small decrease in memory as the interaction level A reason to why we could not see any significant difference increase. As for the recognition rates, the results lie undeniably comparing the results from the word-fragment test between the close to guessing rates. study group and the control group could be the number of foils in That pictures are easier to remember than words may explain the the test. Only three foils with the theme “nature” were placed in the big differences that we have seen between the fruits/vegetables and test. These words were made simpler on purpose to clear the minds beverage brands. However, in the research that Jenkins et. al. [14] of the participants so that their earlier responses may not prime their conducted, they could see no significance in the recognition of responses on subsequent word fragments. The foils were picture-to-word compared with word-to-word. As well as distributed evenly between the fragments. The fragments Kirkpatrick’s study [16] where displayed objects had a clear constructed had a high difficulty since most words were short and advantage on recall over displayed words. Assuming that the easy to target with the structure of the word. Despite this, the beverage brands can be compared to words and the control-group, without any context, managed to score high scores fruits/vegetables objects can be perceived the same was as pictures, on the test. The control group were not monitored so it was harder we should not see that big of a difference between the two. Which to supervise if they conducted the study correctly. There is a risk were confirmed by Jenkins et al. [14] drawing the conclusion that that they developed a self-generated validity despite the placed-out recognizing label of pictures is as easy as recognizing the word foils. A way to minimize this would have been to use more foils itself. and maybe mix up the theme for them, so that all words seemed Some of the participants may have been suffering, again, from random. One fact that supports that some participants developed a "Inattentional Blindness” disregarding the information on the cans self-generated validity is that a few participants in the control group as useful given that several of the participants stated that they did reported words that are in the construct of interest, but not a part of not even notice the beverage brands. In this case it could mean that the goal words. difference between the two tasks, the fruits/vegetables receive 5.3 Future Research focused attention as a product of the task at hand. While the As we could see adequate results for the fruits/vegetables objects beverage cans were disregarded as useful information and acted in terms of recognition and recall but not for the beverage brands, only as targets. it would be interesting to advance the research to objects associated What also must be kept in mind looking at the results, is that more with brands. If the objects are easier to remember rather than the favored brands such as Pepsi and Fanta could have been easier to mere brands or label of an object, could an object tightly associated recognize than for example Mer or Bonaqua. This is due to their with a brand increase the memory rates for the brands? every-day exposure of these brands, and that the brand-awareness It would also be interesting to investigate what types of interactions for popular brands could be much higher among the participants. that performs better on memory rates. This study only investigated We cannot underestimate the effect that brand awareness may have three types of interactions, point-and-click, throwing and shooting. had on the participants biases. The same goes for the What more interactions could be done in VR advertising and how fruits/vegetables. Even though all of the fruits are common, some 10 will they affect the memory differently. 6. Charles R. Duke and Les Carlson. 1993. A Conceptual For the implicit memory effects the data were insufficient to draw Approach To Alternative Memory Measures For any overall conclusions. However, since we could see a significant Advertising Effectivness. Journal of Current Issues & effect on the implicit memory caused by the semi-interactive fruits Research in Advertising. and vegetables it would be interesting to see if minor interactions, 7. Charlie Fink and Stephanie Llamas. 2018. How Are such as point and click, could have an advantage on the implicit People Making Money In VR... Or When Will They? memory over more interactive tasks. Forbes. Retrieved June 7, 2019 from The focal construct in this study was based on Swedish users. The https://www.forbes.com/sites/charliefink/2018/01/02/ho study could be extended with changing the focal construct for w-are-people-making-money-in-vr-or-when-will-they/ several different countries or comparing between several different 8. Gadget Flow. 2018. Will VR Change the World of countries. The focal construct could maybe even be extended by Gaming? Virtual Reality Pop. Retrieved June 7, 2019 from including several different age groups. Another extension could be https://virtualrealitypop.com/will-vr-change-the-world- to replace the categories fruits/vegetables and beverage brands, and of-gaming-d8d0e56d3092 see if there are any differences according to category objects. By 9. Michael Girgis. 2012. Assessing the Impact of letting particular brands investigate around their own product, what Interactivity on Recall and Recognition. types of objects are easy to remember and what types of interactions that could really benefit for a particular product. 10. C. L. Grady, A. R. McIntosh, M. N. Rajah, and F. I. M. Craik. 2002. Neural correlates of the episodic encoding of 6. CONCLUSION pictures and words. Proceedings of the National Academy With the help of intense multi-sensory input from the VR HMD as of Sciences 95, 5: 2703–2708. well as utilizing its full potential in terms of motor stimuli, the https://doi.org/10.1073/pnas.95.5.2703 interactive and semi-interactive objects in the VE does have a 11. Carrie Heeter and Carrie Heeter. 2019. Interactivity in the significant impact on explicit memory. We can also draw the Context of Designed Experiences. 2019. conclusion that the memory of the static objects provided on the https://doi.org/10.1080/15252019.2000.10722040 instructions screen did not have a significant effect on explicit memory. 12. Laura Herrewijn and Karolien Poels. 2014. Recall and recognition of in-game advertising: The role of game A conclusion whether the implicit memory was affected by the control. Frontiers in Psychology 4, JAN: 1–14. objects could not be drawn due to insufficient data. We could https://doi.org/10.3389/fpsyg.2013.01023 however see a slight change in the semi-interactive fruits and 13. Igroup. 2019. Igroup presence questionnaire (IPQ). vegetables effect on implicit memory. Retrieved June 7, 2019 from ACKNOWLEDGMENTS http://www.igroup.org/pq/ipq/ Thanks to my supervisors Björn Thuresson and Calle Sténson for 14. JOSEPH R. JENKINS, DANIEL C. NEALE, and their support throughout this thesis. Also, a special thanks to STANELY L. DENO. 1967. Differential Memory for Adverty AB who provided me with workspace as well as Picture and Word Stimuli. Journal of Educational equipment and additional help to accomplish the study. Psychology 58, 5: 303–307. https://doi.org/10.1037/h0025025 REFERENCES 15. Helen Katz. 2000. Interactivity in 2000. Journal of 1. Acer. Tech Specs Windows Mixed Reality Headset Interactive Advertising: 78–85. AH101-D8EY. Retrieved June 7, 2019 from https://doi.org/10.1080/15252019.2000.10722045 https://www.acer.com/ac/en/US/content/model/VD.R05 AP.002 16. E. A. Kirkpatrick. 1894. An experimental study of memory. Psychological Review 1, 6: 602–609. 2. Lisa Avila and Mike Bailey. 2014. Virtual Reality for the https://doi.org/10.1037/h0068244 Masses. 103–104. 17. Joel Koopman, Michael Howe, Russell E Johnson, James 3. David Brandt. 2017. Understanding memory in A Tan, and Chu-hsiang Chang. 2013. A framework for advertising. Nielsen Journal of Measurement 1, 215–218. developing word fragment completion tasks. Human https://doi.org/10.1111/j.1468-0009.2012.00661.x Resource Management Review 23, 3: 242–253. 4. Christopher F Chabris and Daniel J Simons. 1999. Gorillas https://doi.org/10.1016/j.hrmr.2012.12.005 in our midst: Sustained inattentional blindness for 18. Angela Y. Lee. 2003. Effects of Implicit Memory on dynamic events. Perception 28, 9: 1059–1074. Memory-Based versus Stimulus-Based Brand Choice. https://doi.org/10.1068/p281059 Journal of Marketing Research 39, 4: 440–454. 5. H.Q. Dinh, N. Walker, L.F. Hodges, Chang Song, and A. https://doi.org/10.1509/jmkr.39.4.440.19119 Kobayashi. 2003. Evaluating the importance of multi- 19. Kristian Lorenzon and Cristel Antonia Russell. 2012. sensory input on memory and the sense of presence in From apathy to ambivalence: How is persuasion virtual environments. Proceedings IEEE Virtual Reality knowledge reflected in consumers’ comments about in- (Cat. No. 99CB36316): 222–228. game advertising? Journal of Marketing Communications https://doi.org/10.1109/vr.1999.756955 18, 1: 55–67. https://doi.org/10.1080/13527266.2011.620768 11

20. Eve Mahony. 2018. Yhe Evolution of Ads in Video 28. Statista. 2018. Unit shipments of virtual reality (VR) Games. Venatus. Retrieved June 7, 2019 from devices worldwide from 2017 to 2019 (in millions), by https://www.venatusmedia.com/blogs/evolution-of- vendor. Statista. Retrieved June 7, 2019 from video-game-ads https://www.statista.com/statistics/671403/global-virtual- 21. Paul Milgram and Fumio Kishino. 2013. A TAXONOMY reality-device-shipments-by-vendor/ OF MIXED REALITY. December. 29. Haydn Taylor. 2018. Revenue expected to double this year 22. Michelle R. Nelson. 2002. Recall of brand placements in for augmented and mixed reality. gamesindustry.biz. computer/video games. Journal of Advertising Research Retrieved June 7, 2019 from 42, 2: 80–92. https://doi.org/10.2501/JAR-42-2-80-92 https://www.gamesindustry.biz/articles/2018-05-25- revenue-expected-to-double-this-year-for-augmented- 23. North of 41. 2018. What really is the difference between and-mixed-reality AR / MR / VR / XR ? Medium. Retrieved June 7, 2019 from https://medium.com/@northof41/what-really-is-the- 30. Sophie Thompson. 2019. VR Applications: 21 Industries difference-between-ar-mr-vr-xr-35bed1da1a4e already using Virtual Reality. Virtual Speech. Retrieved June 7, 2019 from https://virtualspeech.com/blog/vr- 24. BS Ong and David Meri. 1995. Should product placement applications in movies be banned? Journal of Promotion Management 6491, August 2014: 159–176. 31. Mikey Tom. 2017. VR/AR Breakdown: VCs investing https://doi.org/10.1300/J057v02n03 heavily to make it a reality. PitchBook. Retrieved June 7, 2019 from https://pitchbook.com/news/articles/vrar- 25. Allan Paivio, T. B. Rogers, and Padric C. Smythe. 1968. breakdown-vcs-investing-heavily-to-make-it-a-reality Why are pictures easier to recall than words? Psychonomic Science 11, 4: 137–138. 32. Michael T. Ullman. 2004. Contributions of memory https://doi.org/10.3758/BF03331011 circuits to language: The declarative/procedural model. Cognition 92, 1–2: 231–270. 26. George Paliy. 2018. The Future Of Advertising In Virtual https://doi.org/10.1016/j.cognition.2003.10.008 Reality. StopAd Blog. Retrieved June 7, 2019 from https://stopad.io/blog/future-virtual-reality-advertising 33. Moonhee Yang, David R. Roskos-Ewoldsen, Lucian Dinu, and Laura M. Arpan. 2006. The Effectiveness of 27. H. Shanker Krishnan and Dipankar Chakravarti. 2004. “in-Game” Advertising: Comparing College Students’ Memory Measures for Pretesting Advertisements: An Explicit and Implicit Memory for Brand Names. Journal Integrative Conceptual Framework and a Diagnostic of Advertising 35, 4: 143–152. Template. Journal of Consumer Psychology 8, 1: 1–37. https://doi.org/10.2753/JOA0091-3367350410 https://doi.org/10.1207/s15327663jcp0801_01

TRITA -EECS-EX-2019:456