Guidelines for Mixed Reality Applications for Culture and Learning Experiences

MIRACLE HANDBOOK GUIDELINES FOR MIXED REALITY APPLICATIONS FOR CULTURE AND LEARNING EXPERIENCES

EDITORS Seppo Helle | Teijo Lehtonen | Charles Woodward | Markku Turunen | Hannu Salmi CONTRIBUTORS

© 2017 University of Turku University of Turku University of Tampere Department of Future Technologies Faculty of Communication Sciences Editors: Seppo Helle, Teijo Lehtonen, Charles Teijo Lehtonen Markku Turunen Woodward, Markku Turunen, Hannu Salmi Olli I. Heimo Jaakko Hakulinen Seppo Helle Silvia Rubio Hernandez ISBN: 978-951-29-6884-8 Lauri Härkänen Jobin James Layout: Lauri Järvenpää, Seppo Helle Sami Jokela Tuuli Keskinen Tuomas Jokioja Ville Mäkelä Lauri Järvenpää Santeri Saarinen Timo Korkalainen Erika Tanhua-Piiroinen Jussi Latvala Tuomas Mäkilä University of Helsinki Juho Pääkylä Faculty of Educational Sciences Kaapo Seppälä Hannu Salmi Finnish History Jani Koivula Kirsi Vainio-Korhonen Timo Mäkimattila Lauri Viinikkala Timo Suvanto Laura Yli-Seppälä Helena Thuneberg Mari-Pauliina Vainikainen VTT Charles Woodward Mika Hakkarainen Tatu Harviainen Petri Honkamaa Kari Rainio CONTENTS

1. Introduction...... 1 4. Cases...... 51 4.1 Luostarinmäki Adventure...... 52 2. Mixed Reality in Touristic and Educational Use...... 5 4.2 Sanan seppä / Wordsmith...... 58 2.1 What is Mixed Reality?...... 6 4.3 Museum Explorer...... 61 2.2 Examples of Existing MR Applications...... 10 4.4 Häme Castle...... 63 2.3 Museums and Other Touristic and Educational Sites...... 11 4.5 House Casagrande...... 65 2.4 Mixed Reality in Pedagogical Use...... 17 4.6 Case Mobilia I — the Road Grader Simulator...... 67 2.5 Business Aspects of Culture-Oriented Mixed Reality...... 21 4.7 Case Mobilia II — the 360° Rally Simulator...... 70 2.6 Co-operation in multidisciplinary field...... 24 4.8 Case Vapriikki — the Optical Telegraph...... 72 4.9 Amos Anderson...... 75 3. Production Guidelines...... 27 4.10 Maritime Centre Vellamo / SelfieWall...... 77 3.1 Software and Hardware Platforms...... 28 4.11 Museum of Technology / Augmented Radio Waves...... 79 3.2 Tracking...... 32 4.12 Ilmailumuseo / How to Fly...... 81 3.3 Storytelling...... 34 4.13 Molecule Movements...... 83 3.4 Content Creation...... 35 3.5 User Experience and Usability...... 46 5. Summary...... 85 3.6 Observations of Narrative AR Application Used in Real Learning Situations...... 49 6. Toolkit...... 89

Glossary...... 93 1. INTRODUCTION 1. INTRODUCTION

The purpose of this Handbook is to summarize the knowledge collected in the MIRACLE research project, and deliver application design guidelines based on the project results, all in one compact document. The Handbook provides a summary about possibilities of applying augmented and virtual reality technologies in cultural travel and education.

It includes • Guidelines for designing and building successful Augmented Reality applications • Advice for experts in one profession about working with people of another profession • Toolkit: a list of software components, tools and methods for developing augmented reality applications serving cultural travel

1 1. INTRODUCTION

INTENDED AUDIENCE MIRACLE PROJECT cost for applications are decreas- Research topics in the project ing. included The Handbook is aimed at support- MIRACLE — Mixed Reality Appli- • The project’s intention was to de- • Developing cost-efficient tools ing museums and other cultural cations for Culture and Learning velop ways for multidisciplinary and processes. This includes cre- sites with historical and/or cultural Experiences — was a research pro- team to create attractive, engag- ating pilot apps for varying cases, value and interested in attracting ject that was run between 1/2015 ing mixed reality applications for mainly museums and memory general audience by augmented re- and 4/2017. It was an extension to cultural travel, to create knowl- organizations, and developing ality offerings. It helps the person- the preceding Futuristic History edge clusters connecting profes- improved AR technologies for nel of such organizations in choos- project[1] of 2013–2014. The moti- sionals with different expertise these uses. ing what kind of experiences would vation for this research was based areas, and to collect knowledge • Studying learning experiences suit the particular case, and what on the following facts: into a Handbook and a Toolkit by AR applications. Learning resources are needed to make and • Museums need new attractions that would be distributed openly. is moving out of the classroom, maintain it. as the competition of people’s A central objective in applying and research shows that interac- Companies creating augmented time and interest is getting more mixed reality technologies to tour- tive learning methods are often reality/virtual reality (AR/VR) ap- and more fierce. Augmented re- istic and educational use is to make more efficient than traditional plications and content for cultur- ality can enrich cultural experi- the facts visible and help the visi- teaching methods. al organizations are also potential ences and is therefore a potential tor to understand them more eas- • Seamless connection with social audience for the Handbook. They means of attracting people to ily and deeply than by traditional media. Sharing experiences is should gain practical information places that could otherwise be methods used in museums. Besides important to people, and crowd- about tools and methods that have losing visitors. that, the project was also focused sourcing can be used for collect- been tried in the MIRACLE project. • Interactive applications have on human factors — engagement of ing valuable knowledge about Yet another target group are re- educational potential, so gener- the audience, entertainment factors culture. searchers (especially historians) inal education could also benefit and social interaction. • Business models. Finding suita- terested in using AR/VR technolo- from the approach. ble business models for different gy as a tool in academic work. • Tourists routinely carry smart- kinds of cases was a target. phones and tablets that can run • Creating a network of potential the applications. There’s not nec- actors in Finland. essarily a need for major invest- ment in new equipment at the site. • Development tools are becoming easier and more affordable, meaning that the production

2 1. INTRODUCTION

Project duration was from 1/2015 to 4/2017. The research units that The project was financed mainly by Tekes — the Finnish Funding Agency for participated in it were Innovation. Other participating organizations were • University of Turku: Department of Future Technologies (formerly • Amos Anderson Art Museum, Helsinki Technology Research Center) and Finnish History. • KOy Casagrandentalo, Turku • VTT Technical Research Centre of Finland • The Chemical Industry Federation of Finland • University of Tampere, Faculty of Communication Sciences (formerly • EnkeliGroup Oy, Raisio School of Information Sciences) • Evangelical Lutheran Church of Finland, Helsinki • University of Helsinki, Faculty of Educational Sciences • Finnish Aviation Museum, Vantaa • Kilt Oy, Tampere • Maritime Centre Vellamo / Museum of Kymenlaakso, Kotka • Automobile and road museum Mobilia, Kangasala • The Museum Centre of Turku • Museum of Technology, Helsinki • Muuritutkimus Ky, Kaarina • National Board of Antiquities, Helsinki • Turku and Kaarina Parish Union, Turku • Vapriikki Museum, Tampere • Vucom, Helsinki • Nimble Devices Oy, Espoo • Silencio Oy, Helsinki • Bryggman Foundation, Turku • Finnish Tourist Board, Helsinki • Regional Council of Southwest Finland / Lounaispaikka, Turku

3 1. INTRODUCTION

HANDBOOK CONTENTS REFERENCES Chapter 2 describes the potential 1. Futuristic History: uses of mixed reality in museums http://ar.utu.fi/ar/research/futuris- and other cultural sites, and the tic-history/ pedagogical aspects involved in it. Multidisciplinary co-operation in MR creation and maintenance is discussed. Chapter 3 concentrates in the actual production and maintenance processes, business opportunities and connections to social media. Chapter 4 describes the individual pilot applications that were implemented in the MIRACLE project: how they were planned and built, what were the research targets and what was learned. Chapter 5 summarizes the outcome of the project. Chapter 6 describes the Toolkit, i.e. the collection of software tools and components tested and developed within the MIRACLE project. References and related literature are listed at the end of each chapter. A glossary is at the end of the document.

4 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE

This chapter explains the basics about augmented reality, virtual reality and mixed reality. Some existing culture- and science-related applications around the world are introduced. The potential for using these technologies in cultural heritage related applications is discussed from different points of view: identified benefits in educational use are described, and financial aspects are analyzed. The need of cooperation between people of different skills is introduced.

5 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE 2.1 WHAT IS MIXED REALITY?

Imagine yourself in an open-air like a ghost town than life in the An adventure game shows the mu- By definition of AR by Azuma[1], museum consisting of 18 blocks 18th century city. Imagine then that seum visitor the life in the city in it has three characteristic elements: of an 18th century city. The build- everything comes to life. You can 1850’s by combining the reality and a) it combines real and virtual ob- ings around you are preserved on see the people and the animals of the virtual, the physically existing jects in a real (3-D) environment, their original location. Every ob- the past and feel the bustle of the contemporary buildings and digi- b) it runs interactively and in real ject is an authentic remnant of the city. You can discuss with the peo- tally created characters. time, and c) it registers (aligns) real past. Everything you see has some- ple you meet and have an effect in Virtual Reality (VR) is an envi- and virtual elements with each oth- thing to tell about life more than what happens around you. ronment, created artificially, typ- er. 200 years ago. But yet, something This is not a scene from a science ically using computers, and pre- Augmentation of reality has is missing: the life itself. Streets are fiction story, but instead, using a sented to human users via various been used in different medias and quiet and the buildings are empty tablet computer, already reality in audiovisual displays. In principle, systems for decades, e.g. head-up with the exception of other visitors part at the Luostarinmäki Hand- virtual reality may contain artificial displays in fighter planes or score- and guides. You experience more icrafts Museum in Turku in 2014. stimuli for any human sense, but in boards in sportscasts. Due to the practice the virtual environments computerisation and rapid emer- nowadays are mostly limited to gence and development of mobile visual and auditive content. technology, AR is now available for In Augmented Reality (AR) the common users in their mobile de- real world view is augmented with vices and possibly soonish with AR- additional information. This can glass technology. be computer-generated 2D and 3D images or information superim- posed on the real-world view, seen through data glasses, or captured from the camera of smartphone, computer or other device. Aug- mented image appears to its users like virtual and real objects coex- isted in the same space. In other words, AR immerses its users in Image 1. Luostarinmäki Handicrafts museum, Turku, Finland. virtually enhanced real world. Image 2. Head-up display in an aircraft.

6 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE

Mixed Reality (MR) is the art thetics of AR have the strongest the real world as possible. move freely and explore the con- and technology that is used to mix influence on perceived enjoyment. 2. Be highly interactive in re- tent from any angle and location real-world elements with virtual Also, as expected, South Korea, al-time. The second condition they like. The virtual content reality. In the already classical vir- having high collectivism and high separates for example pre-ren- must always be fixed tightly in tual reality continuum by Milgram uncertainty avoidance culture, dis- dered movie productions from its place in the real world while and Kishino[2], everything between played stronger dependence on so- augmented reality, as even if the the user moves about. the extremes of reality and virtu- cial influence and hedonic charac- contemporary film productions al reality is called Mixed Reality. teristics of AR/MR. feature highly believable com- This area includes augmented re- Smartphone or tablet device puter generated imagery, they do ality as well as augmented virtual- meets the main requirements posed not represent a real-time simula- ity, which means a mostly virtual by AR since it has a camera and cation of reality from the end-us- environment containing some real pability of rendering and display- ers point of view. elements. MR can use any or all the ing the augmented graphics. Hence, 3. Allow users to experience the senses we have, from viewing and with explosive growth of penetra- content with free movement hearing (which are the more com- tion rates of smartphone, applica- in the real world 3D space. The mon methods) to tasting, feeling tion-based AR has been more ac- third condition emphasizes that and smelling — even understand- cessible to users. the end users must be able to ing balance. However, most actual Whereas virtual reality (VR) can mixed reality applications are lim- deviate greatly from the real world, ited to audiovisual content in the augmented reality productions virtual part. however must fit into the physical The interest towards the emerg- context in order to achieve an im- ing AR/MR technology has been mersive and believable experience. rapidly growing among the mu- At the minimum a good AR appli- seums and cultural heritage sites cation should: around the world. However, the 1. Seamlessly combine the phys- acceptance of AR/MR applications ical and virtual pieces of con- can vary in different populations. tent. The first condition is not Lee & al[3]. made a cultural compar- to set limitations to artistic free- ison between South Korea and Ire- dom in any sense, but to high- land, both having high smartphone light the fact that at least the vir- penetration rates but different cul- tual content should in fact react tural profiles, and noticed that aes- to as many changes and parts of Image 3. Virtual Reality continuum.

7 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE

AUGMENTED REALITY glasses, like the Microsoft HoloLens eliminated. ucation, it can act as a guide or in- •• a mixture of real world and digital or Google Glass, are another possi- There is more detailed discus- terpreter, and it can be used to help illusions bility of presenting augmented real- sion about AR-enabled devices in in decision making. •• interactivity with the real environ- ity content. Compared to handheld Chapter 3. A common example is a travel ment devices, such glasses offer the ad- application displaying indications •• on-line effects vantage of freeing the user’s hands, WHAT CAN THE of nearby restaurants and hotels, •• three-dimensional impressions and the augmentations can also be AUGMENTED CONTENT possibly with the price and rating seen over the real environment in- BE? information. Technically a similar MIXED REALITY stead of the device screen. However, example would be recognizing li- •• The combination of material world currently (Q1/2017) AR glasses are In principle, the augmentations can cense plates of cars on a street and and virtual elements between real still an expensive rarity, and their be derived from any information displaying their owners’ names and and virtual environments display properties often leave room that is available and can be asso- addresses, if this information can for improvements. But in not too ciated with the real world objects be extracted over the internet. far future we might see products around the user. Imagination is A practical example presented in ON WHAT KINDS OF that change the game, and then a perhaps the biggest limitation to image 4 translates signs and labels EQUIPMENT CAN MIXED boom of AR applications for glasses what can be presented — AR can be from one language to another. The REALITY APPLICATIONS may be expected. used to entertainment as well as ed- principle in this is to recognize pre- BE USED? Augmented reality could also be presented via virtual reality glass- Mobile devices (smartphones, tab- es — also known as head-mounted lets) have reached the technology displays, or HMDs — such as the level where they can offer satisfacto- Oculus Rift and HTC Vive. The ry mixed reality experiences with- display can show the real environ- out any additional hardware. The ment as video, adding augmented built-in camera, movement sensors, objects in the video feed. However, compass and GPS receiver can pro- these displays obviously aren’t tru- vide location data accurate enough ly mobile — they can only be worn for many cases, and the computing in controlled, safe places, not when capacity is often good enough to walking in the streets. They could keep the lag in MR apps at a satis- be a feasible option in cases where factory level. The display presents the application would be used in a the augmented view to the user. restricted place, and where the risks See-through augmented reality of collisions and other accidents are Image 4. Augmented reality translation example.

8 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE defined objects from the live camera media used in museum exhibitions REFERENCES image and replace them on screen and heritage sites. All aspects of with other graphics, everything the past life are just not reducible 1. Ronald T. Azuma. A survey of augment- happening on the fly. This kind of to words, still images or other tra- ed reality. Presence, 6(4):355–385, 1997. application can be very useful for ditional means. Mixed reality can 2. Paul Milgram, Fumio Kishino: A Taxon- example in museums where addi- connect historical information with omy of Mixed Reality Visual Displays. tional translated signs would be a the material reality, making histo- IEICE Transactions on Information and disturbance in the authentic envi- ry visible in situ and thus make it Systems vol. E77-D, no. 12(12):1321–1329 ronment. From technological point more comprehensible. · December 1994. of view, the task requires the use of sophisticated computer vision al- 3. Hyunae, Lee, Chung, Namho and Jung, gorithms and fairly powerful mo- Timothy (2015) Examining the cultural bile devices. For the user, such ap- differences in acceptance of mobile plications offer a very easy-to-use augmented reality: Comparison of replacement of a dictionary when South Korea and Ireland. Information visiting places in a foreign country. and Communication Technologies in Cultural heritage tourism is an Tourism 2015. Springer International example of a field that has already Publishing, 2015. 477–491 gained from MR technology. MR may help to prevent degradation of historical sites and offers the tourists enjoyable and useful access to cultural heritage. Major attractions such as the British Museum and the Louvre offer MR applications to their customers. Especially narrative MR applications can be very useful in communicating historical knowledge to the public. MR technology can offer information and experiences that would be very difficult, even impossible to represent with traditional

9 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE 2.2 EXAMPLES OF EXISTING MR APPLICATIONS These applications are — or have Natural history applications ARtours REFERENCES been — available for the visitors in American Museum of Natural Histo- The Stedelijk Museum, Amster- some museums and other places of ry. The museum distributes several dam[5]. The app allows you to ex- 1. Streetmuseum Museum of London: interest around the world. mobile apps[3] for visitors acting as perience interactive tours on your https://www.museumoflondon.org. personal tour guide or presenting smartphone and enjoy the rich con- uk/discover/museum-london-apps Streetmuseum theme-specific information about tent (video, audio, photos, stories, AppStore: Museum of London. AR app of his- exhibitions. Some of the apps are: tasks and Augmented Reality addi- https://itunes.apple.com/app/ torical photos[1]. The app displays • Explorer: the general app, allows tions) as you explore the museum id369684330 historical photos in the actual lo- creating personal tours accord- or the streets of Amsterdam. 2. PIVOT: cations, overlaid onto the current ing to own interests, acts as a http://www.pivottheworld.com/ view. The app was released in 2010. guide. Pokémon Go • MicroRangers: AR game to be Pokémon Go[6] was the game that 3. American Museum of Natural History: PIVOT played while in the museum. brought augmented reality as a term http://www.amnh.org/apps/ “Rediscover the history around you. Scenarios involving microor- to wide audiences around the world. 4. KSPACE National Museum Australia: Change your point of view.” PIV- ganisms to be explored. While it perhaps did not fulfil all http://www.nma.gov.au/kspace/ OT[2] (Point of Interest Visual Opti- • Dinosaurs: photos and render- the definitions of augmented reality, kspace_app mization Tool) is a mobile app that ings of dinosaur fossils. it did give a rough idea about vir- reveals to users what places looked • Beyond Planet Earth: AR app to tual creatures in a real environment 5. ARtours Stedelijk Museum, Amsterdam: like in the past — showing authen- accompany the 2011 exhibition. to a huge amount of people. http://www.stedelijk.nl/en/artours/ ticated histories of locations (from artours-app everyday places to popular tourist KSPACE 6. Pokemon Go: attractions), through images, vid- National Museum Australia[4]. The http://www.pokemongo.com eos, and text-audio. End users can application encourages kids to ex- also store and share their most piv- plore the Museum by finding char- otal moments through “shoebox acters from the Kspace games. The archiving” (the method of digitiz- app uses markers to launch the ing memories and old shoeboxes of characters. The visitor is encour- photographs). aged to follow a trail around the Museum and find all eight characters placed there.

10 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE 2.3 MUSEUMS AND OTHER TOURISTIC AND EDUCATIONAL SITES Museums and other culture-related THREE TYPES OF experiences. Many members of this MIXED REALITY attractions are competing fiercely MUSEUM VISITORS group are rather young and willing AND HERITAGE about people’s time with everything to use new technologies, also MR, ORGANIZATIONS else, so they must offer appealing The expectations of the public nat- to gain new experiences. They also and individual experiences to draw urally vary from person to person, expect that modern technologies Currently, the heritage organiza- people’s attention. Mixed reality is but three types of museum visitors are used at least at some extent to tions are awakening to the fact that one possible means of doing that. can roughly be separated, accord- present the sites or museums they larger and larger groups of visitors A good application for cultur- ing to surveys carried out by the are visiting. The content aimed at think of them more as places where al travel sector must measure up Finnish Museum Association[2]. this group must be entertaining but to experience new things and less as to the expectations of the public. The first typeseeks a place for also informative since the group places of mere learning.[3] The com- During both Futuristic History and meditation and quieting down, dis- mainly values educative elements municative mission of the heritage MIRACLE projects user studies of tant from our own time. Members alongside the entertaining ones. organizations is still important but pilot applications were carried out. of this group may visit the sites An individual museum visitor it must be adapted to the new ex- Studies of the Luostarinmäki Ad- also on nostalgic bases. This group has usually at least some character- pectations of the visitors. The her- venture[1] yielded valuable infor- is usually not interested in using istics from all these types or groups, itage organizations, therefore, need mation about what the future users modern technology on site what- which must be taken into account tools to modernize their services may expect of the MR technology soever. Creating MR content that when designing the applications. and to engage their visitors in the in museums and other cultural would please this visitor type may This suggests that a good applica- process of knowing, which is now heritage sites. Combined with earli- be demanding if necessary at all. tion should be well-balanced be- held as interactive and entertain- er visitor studies from various mu- The second type visits museums tween entertainment and education ing. The heritage sector has clearly seums some general features can be to get information about history and and its use should be voluntary. shown its interest towards tech- outlined. the place or site. Many people in this Perhaps in an ideal situation there nologies like MR which can make group are interested in using new might be several different types of the museum experience engaging, technologies which may help them applications available in a museum entertaining and informative. All to increase their knowledge. The or a site from which the visitors these three features are in center of content in applications targeted at could choose. Another possibility the discussion about the modern this group must first and foremost could be an application that could museum exhibition. be informative. adapt the content to the preferences The question of the needs of the The third and, according to sur- of the user. travel sector concerning the use of veys, biggest group is looking for MR applications on cultural travel

11 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE sites requires further research. The OPPORTUNITIES MR applications can be more Enabling this with the help of travel sector is very closely linked to OFFERED BY MIXED entertaining than some more tra- MR was tested with positive results the museum and cultural heritage REALITY ditional means of communicating within the cases of the Luostarin- sectors, which makes it important (historical) knowledge. Therefore, it mäki Adventure and the Louhisaari to study the possibilities of the MR MR can make abstract and invisible might attract people who wouldn’t Stories. Elements added digitally to technology also from its point of things comprehensible and concre- usually visit museums or heritage the view, like a demolished fence or view. At this moment, however, it is tize complex historical phenomena. sites, e.g. young adults. 19th century seastrand made the possible to state that the travel sec- It is possible to combine digital One of the most remarkable as- whole environment more compre- tor, like the heritage organizations, objects and intangible heritage with pects of the MR technology is its hensible. The possibility to inter- conceive of producing experiences tangible heritage (buildings, ob- ability to offer information that act with the environment and the as its overriding aim. The help of- jects). The resulting MR solutions would be very difficult, expensive characters in the Luostarinmäki fered by MR and other technologies may help to prevent degradation or totally impossible to present with Adventure and the feeling of actu- is widely recognized[4]. The concept of historical sites. For example, re- other media used in museum exhi- ally being in the past caused by this of combining the historical content stricted spaces can be entered vir- bitions and heritage sites. There is possibility enhanced opportunities with commercial and other useful tually, and mechanical functions of much information that cannot be for empathic understanding of the information in same applications historical objects can be presented presented or which is difficult to past realities. The same idea of im- has come up and seems promising. virtually instead of moving the ob- grasp in textual form[5]. Using oth- mersing people in the past was also Evidence exists that cultural jects physically. er media, like images, may help in the motivator behind the augment- heritage tourism has already gained MR can be a cost-efficient alter- these cases but some pieces of infor- ed SelfieWall pilot at Vellamo. from MR technology. [4, 6] MR may native for some traditional means mation are inconceivable until they MR offers a relatively low-priced help to prevent degradation of his- used in museums, such as museum are presented in connection with alternative to presenting history torical sites and offers the tourists dramas and physical reconstruc- the physical reality. through museum theatre perfor- enjoyable and useful access to cul- tions. Combining information with mances or re-enactments of past tural heritage[7]. Major attractions In most cases MR doesn’t have the physical reality enables the vis- events. Digital characters repre- such as the British Museum and the to compete with other means of itors to use their own bodies as in- senting people of past times with Louvre do offer MR applications to communicating historical knowl- struments of learning. This makes which it is possible to interact or their customers. edge. MR can be used e.g. alongside it easier for them to comprehend for communicate are more affordable traditional exhibitions and guided example the physical scale, the spa- than live actors although they hard- tours. MR and “real” objects are tial relations and the movements of ly can entirely replace a real person. not options but they rather com- the presented things. It can also in- Using these characters can help the plement each other. The museums crease the opportunities to learning heritage organizations to solve one can choose the best means of rep- by doing and empathic learning of of the most paradoxical and chal- resentation for their purposes. the life in the past. lenging problems they have. Repre-

12 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE senting human life in the past is an self-purposefully. It must earn its early to judge whether stereoscopic technology needed to make some- important mission for the organi- role by bringing in benefits that are movies will again go away, but the thing specific may still be a few zations and yet they have to do this unattainable with other media for percentage of 3D movies from all years away, so it must be considered only with the help of the material representation and communication. movies seems to be declining. The whether it is wise to wait still and culture since the human beings are Until now, MR technology is be- WOW effect and novelty do not perhaps make something else first. gone, and the past life itself exists ing used at cultural heritage sites seem to be able to keep up against There are also other practical no more. The MR technology ena- and museums in many cases most- the problems: inconvenience of the risks (costs) that must be consid- bles bringing the people of the past ly because of its novelty value and glasses and side-effects like visual ered when planning a mixed reality to life, although as recreations and its WOW effect. Creating impres- fatigue, nausea and headaches that service, such as: interpretations of researchers of our sive experiences is easy because the many people suffer.[9] How much updates and main- own time. technology itself is new to most of These risks can be dealt with tenance are expected to be need- Various studies on learning have the visitors. There lies a risk that the by choosing the technologies so ed, and are there enough re- shown that people learn better by technology takes the visitors’ atten- that they really fit to the intend- sources (personnel, funding for doing and by teaching others than tion from the content. Even more ed content, and by trying to avoid subcontracting) for it? by only reading or listening[6,3]. disconcerting risk is that after the the known problems and utilize If there are devices to be loaned The learning results are the better technology has lost its novelty value the technology as well as possible. to visitors, how much expenses the more senses can be involved. and ceased to attract visitors by its Especially, it is very important to would be spent for repair and sto- Using the whole body supports the mere WOW effect it can be deemed keep the content in center when de- len units? How can those costs be learning process and helps memo- worthless. signing the applications. The con- minimized? rizing the learned things. Examples of such cases can be tent alongside the medium of its How much help the visitors need found in history. 3D (stereoscopic) presentation should be the source in downloading the apps to their WHAT RISKS ARE THERE? cinema had a brief period of success of the experiences, not the technol- own devices and in using the apps? in the 1950s. A number of feature ogy alone. The novelty value of the Is there enough personnel to take Mixed reality technology can car- films were made, also by major stu- MR technology should not be re- care of it? ry also some remarkable problems dios and directors, and people went lied upon as a visitor attractor since and even threats with it. Those re- to see them wearing anaglyphic the technology is rapidly becoming sponsible of creating mixed reality (red-cyan colored) glasses. After a more common and people learn to applications for cultural heritage while the novelty was gone, popular- know it. and cultural travel sectors must be ity ceased, and eventually making The current limitations of the aware of these possible threats. of 3D movies essentially ended. A technology and the capacity of the An essential point to note is new boom started in 2000s enabled available equipment may create ob- that mixed reality — or any other by digital projectors and electronic stacles for presenting content which technology — should not be used shutter 3D glasses. It may be too would otherwise be relevant. The

13 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE

THE ROLE OF SOCIAL can be counted as cultural heritage, The Amos Anderson installation, MEDIA what information about the past is the Sanan seppä, the Luostarin- significant and what is not can be mäki adventure, the Transforma- Mixed reality technology can passed from the organizations to tions of the Knight’s Hall and the change the role of museums and the public[8]. Another question is, Make Your Own Exhibition app. other cultural heritage organi- in what extent, if at all, museums However, people did not seem to zations in the communication should take this opportunity. be keen to share their experiences between them and the public, al- A modern trend, however, seems in social media. The median answer though this is not an inevitable to be that heritage organizations for the statement, across all these result of just taking the technolo- use social media actively as a chan- cases, ranged from somewhat dis- gy into use. Especially social me- nel to provide their services. The agree to neither agree or disagree. dia channels in connection with public and the visitors have become This does not necessarily mean that the MR technology could offer the aware of this and they expect to the respondents would be against public a possibility to take part in find the organizations in the so- the idea of sharing such content providing information in the mu- cial media. Using social media can in social media in general. Instead, seums. This is of course possible make the heritage organizations one possible reason for this rather with the MR technology itself and more open, including and involv- negative attitude might be that peo- even with the traditional exhibition ing towards the public. The visitors ple do not see the traditional and methods, but it seems that the more want to share their museum expe- obvious ways of sharing content in modern technologies are involved riences with their friends in social social media feasible for such in- the more open and reciprocal the media and this comprises also the stallations. Thus, new and easy-to- content creation for exhibitions and MR experiences. use ways to share one’s mixed real- for heritage sites will become. The People’s willingness to share ity experiences should be designed technologies enable the heritage or- mixed reality experiences was in- and tested. ganizations to engage and involve quired in several of the MIRACLE Social media offers a promising the public and the museum visitors pilot cases. A questionnaire item “I tool also for crowdsourcing as a to make knowing into an interac- would like to share my experience method for information gathering tive process in which all parties can to my friends is social media (e.g. for the organizations and for shar- learn and share information. This, Facebook, Twitter or Instagram)” ing information between museum in turn, would decrease the notion or an essentially similar one was in- visitors. Using this in the content of the heritage organizations as au- cluded in the questionnaires for the creation process for MR applica- thoritarian and excluding. Part of road grader simulator, the 360 ral- tions for heritage sites should be the authority to determine what ly simulator, the optical telegraph, studied further.

14 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE

VIRTUAL it is the material remains of the past, to words, still images or other tra- space intact. Neither is the amount RECONSTRUCTIONS OF the objects and the archival sources, ditional means. Mixed reality can of information available limited by THE PAST which provide at least a firm basis connect historical information with the number of or space on informa- for this interpretative work of recre- the material reality, making histo- tion boards, since the boards them- No matter what we might think, ation and reconstruction. Because ry visible in situ and thus make it selves can be replaced with digital the material reality around us is the material reality itself does not more comprehensible. versions, in as many languages as mute. Even the most exciting ves- tell anything, it is crucial to develop As the case of Sanan seppä ap- required, or with more advanced tiges of the past remain silent until methods to link the interpretations plication suggests, one of the most methods of communicating infor- we make them speak. We — the peo- made about the past directly to the significant aspects of MR is that it mation. Storytelling with the help ple living in the present — give them material remains of the past that enables combining digital recon- of digital characters is one of these the words, which we then listen as have functioned as sources for the structions and other additions, sto- methods experimented in Sanan if the objects themselves would be very interpretations. In other words rytelling and historical knowledge seppä application. In this case the speaking. it is important to provide visitors directly with actual locations thus chosen method of storytelling, i.e. We easily forget that the past at a historical building or a herit- multiplying the information availa- using fictional characters, proved itself is irretrievably lost. The old age site with historical knowledge ble for the people visiting the loca- to be suitable for explaining the buildings we enter in hope for step- based on and concerning that site. tions. The direct connection of the material reality of the Cathedral, ping back in time won’t take us on It is noteworthy, that texts or lan- information to the material reality the intangible heritage stemming a time travel. We can’t see them as guage in general, may not be the makes the interpretations easier to from the days of the reformation, they used to be. They are worn by best method to communicate this grasp. The multitude of different the length of the reformation pro- time or altered and renovated or information. Currently, mixed re- types of source material available cess and the changes it brought to both, and therefore look different ality (MR) technology is one of the improves the process even further. the society as well as varying mean- than when they were new. They are most promising new methods for To understand a site or an object ings of the reformation for contem- solely part of the present. All what narrativizing material reality. they see, visitors are not compelled poraries. we can say and think about the past Especially narrative MR appli- to read an explaining text and then One more advantage of the tech- is created in the present and in our cations can be very useful in com- to translate the textual informa- nology is, that presentations based minds. It is our imagination that municating historical knowledge to tion into a visual language in their on MR are available any time. They constitutes how we conceive the the public. MR technology can offer minds, but they can actually see, are not dependent on for example past. information and experiences that how these vestiges of the past have timetables of actors or live inter- Nevertheless, the material re- would be very difficult, even impos- changed, how they were used etc. preters. These representations are mains of the past form a link, how- sible to represent with traditional Because all the additions are dig- also more cost efficient than many ever vague, between the past and media used in museum exhibitions ital, the material reality remains un- other methods, especially if the the present. As knowledge about and heritage sites. All aspects of altered. Using MR based on mark- most part of the visitors download the past is constituted in the present, the past life are just not reducible erless tracking leaves the physical the application on their own devic-

15 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE es, and the heritage site or the mu- applications that present the cultur- REFERENCES seum provides devices only to those al heritage too well may substitute missing suitable equipment for us- the actual heritage with a digital 1. K. Seppälä, O. Heimo, T.Korkalainen, J. 6. E. Dale, “Audio-visual methods in ing the application. version and prevent users from see- Pääkylä, J. Latvala, S. Helle, L. Härkä- teaching.” Dryden. New York 1969. The case example of Sanan sep- ing the real heritage instead of the nen, S. Jokela, L. Järvenpää, F. Saukko, 7. A.S. Marcus, J. D. Stoddard, & W. W. pä, however, indicated also some digital representation of it. L. Viinikkala, T. Mäkilä and T. Lehtonen: Woodward, “Teaching History with weaknesses in using MR in com- Examining User Experience in an Aug- Museums. Strategies for K -12 Social municating cultural heritage. The mented Reality adventure Game Studies.” Routledge. New York and most serious issues dealt with the Case: Luostarinmäki Handicrafts Mu- London 2012. current state of the production pro- seum. In 12th IFIP TC9 Human Choice cess. Since established practices for and Computers Conference, September 8. E. Hooper-Greenhill, “Museums and the many phases of building an MR ap- 7–9, 2016. Interpretation of Visual Culture.” Rout- plication such as Sanan seppä were ledge. New York and London 2004 2. “Museokävijä 2011”. Finnish Museum missing, excessive amount of devel- Association. Helsinki 2011. 9. Angelo G. Solimni, “Are There Side opment work was needed to com- Effects to Watching 3D Movies? A Pro- plete the application. On the other 3. E. Hooper-Greenhill, “Museums and spective Crossover Observational Study hand, the findings and solutions Education. Purpose, Pedagogy, Per- on Visually Induced Motion Sickness.” reached during the process will, for formance.” Routledge. New York and PLoS One, 8(2), 2013 their part, improve the state of the London 2007. https://dx.doi.org/10.1371%2Fjournal. MR production process. 4. J. Cabiria, “Augmenting engagement: pone.0056160. As long as MR applications Augmented reality in education.” are used on tablet computers and Increasing Student Engagement and smartphones, the users look their Rentention using Immersive Interfaces: surroundings through camera view Virtual Worlds, Gaming, and Simula- on a small screen. A risk exists that tion.Vol 6, part C. Edited by Charles the users look their surroundings Wankel and Patrick Blessinger. Bingley: only through this screen and for- Emarald, 2012, pp.225–251. get that what they see actually is around them, not only in the digital 5. L. Auslander, “Beyond Words.” Amer- world opened up by the screen. The ican Historical Review 110 (4) 2005, better and more engaging the con- pp.1015–1045. tent of the application, the greater is the danger. Paradoxically, making

16 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE 2.4 MIXED REALITY IN PEDAGOGICAL USE NARRATIVE AR of culture site excursion fits well to subject matter fits well into the idea vironment and direct users focus APPLICATION these requirements set by curricu- of blended learning. Constructivist to make them more aware to his SUPPLEMENTS lum plans. and socio-constructivist learning surroundings. Characters can even CURRICULUM CONTENT Sanan seppä AR-story, produced theory view that information from interact with real environment uti- as part of MIRACLE-project, is a many different sources contributes lizing help of masking and virtual Augmented reality based technol- good example of application that to individual’s active construction props. ogy can be used to produce differ- can be used to supplement histo- of knowledge. These constructivist Believable virtual actors that ent kind of experiences for supple- ry, language or religion courses in viewpoints are also present in cur- feel to be present in same space as menting elementary or high school elementary or high school level. It rently popular idea of phenomena user and exciting dramatization of level teaching and learning. One combines multidisciplinary con- based learning. otherwise distant historical time possible use of AR is to support and tent, dramatically expressive story Thoughtfully written screenplay and events provide engaging learn- enhance educational visits to mu- and use of AR-technology. Applica- for virtual actors can support phe- ing experience. Inquiry made to seums, churches and other cultural tion presents story of historical fic- nomena based learning by provid- students who tested Sanan seppä sites. These kind of visits are impor- tion set in different locations inside ing lots of interesting content and application showed that there was tant part of many modern curricu- the Turku cathedral and utilizes AR details in different areas and knowl- strong correlation between how in- lum courses. as a tool of storytelling. User moves edge levels. Story with high ped- teresting story was perceived, and Newest curriculum plans in actively inside scenes played by agogical quality offers something how willingly it was used to learn Finnish elementary and high school virtual actors. Sanan seppä differs interesting for all. For example in about subject. emphasize utilization of ICT in all from more fact driven and formal Sanan seppä application elemen- Combining narrative drama to subjects to create diverse learning spatial visualizations that educa- tary level student can view story more analytical and formal infor- environments. Curriculum plans tional AR often uses by trying to in more concrete way focusing on mation can invigorate historical encourage teachers to blend new create engagement with historical differences between everyday life context of culture site and help to forms of technological aids and story. of 1600th century and present day, integrate it more efficiently to sub- out of the classroom -learning into AR application telling an inform- but more mature student can con- ject matter and curriculum content. teaching that is more conventional. ative story is a special medium that nect story’s events neatly to larger In general, AR is efficient in creat- Curriculum plans also value highly can be blended with other forms of historical context. One important ing feeling of presence with content phenomena based, multidiscipli- information to create more diverse demand for good educational AR it presents. Active spatial presence nary and active learning. AR appli- and memorable presentation of sub- story is to bind it strongly to its real and engagement with events of his- cation used independently and ac- ject. This mixing of ways of learn- location. Virtual characters can be torical distance is core element of tively by students and pupils as part ing and medias to approach the aware of their real features of en- narrative AR’s educational strength.

17 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE

In ideal situation user, incorporat- AR technology are soon making technologies and the effects of leisure time, and also experiment- ing AR story can create formidable it available for everyday educa- these in the society also change ed- ing things that never exist in the of- and memorable experience. This tion — the threshold seems no more ucation. We believe more and more ficial curriculums. feeling of presence is threatened if to be money or technology, but learning will happen in informal Lifelong learning needs new virtual content feels too much su- mental resources. learning environments, i.e. outside practical forms: the formal educa- perimposed, stands out from real In educational sector AR gener- classrooms, in the future. We learn tion can learn from the informal, environment or if user is not able ally means a modern computer-as- better by doing than by reading and open learning environments. The actively reposition himself in scene. sisted learning — environment that listening to a teacher. The wide va- existing results indicate and en- This places requirement to smooth- combines the observed real world riety of media available to everyone courage for further development ness of tracking and visual assets phenomena with graphically add- will help to learn better, and inter- of Augmented Reality educational quality of embedment. ed information or images, even active applications are a further solutions. Meaningful learning has spatially positioned sounds can be step towards better understanding two components. First, the content TOWARDS used. The meaning of augmented of science, history, art, geography should be meaningful for the learn- EXPERIMENTATION IN information is to enrich the orig- or nearly any form of knowledge. er. Second, the learning process LEARNING inal phenomenon by information The need of producing such mate- should be arranged pedagogically that is useful in many kinds of rial is increasing at the same time in a meaningful way (according to Research conducted by the MIR- revolutionary applications in ed- as the capabilities of consumer ICT the age and the former knowledge ACLE evaluation consortium has ucation, including the study of ar- devices are growing. and skills of the learner and by the shown the possibilities of AR in chitecture, art, anatomy, languages People learn more in informal logical structure of the topic.) All visualizing the invisible by project- decoration, or any other subject in settings and open learning envi- the great innovations in learning ing virtual objects onto real setting. which a graphic, simulation or 3D ronments. In addition, all learning field have been based on putting Several different AR applications model could improve comprehen- is based on motivation. The latest these two principles into practice. created by MIRACLE proved to sion. More concrete examples of us- research literature has clearly indi- MIRACLE was putting together the allow physical “hands on” and in- ing augmented reality are historical cated that augmented reality type real physical environment and vir- tellectual “minds on” experimen- heritage reconstruction, training of of learning technologies can en- tual experiences. One of the key el- tation with instructional learning operators of industrial processes, hance the situational motivation ements is to bring together concrete scenarios. According to pedagogi- system maintenance or tourist vis- into intrinsic motivation: e.g. real objects and abstract phenomenon. cal experts, museum professionals its to museums and other historic interest in new topic based on facts and teachers, the approach repre- buildings. What is noteworthy rela- and deep-learning strategy both for sents moving from teacher or mu- tive to this study is the fact that the young people and adults. This seem seum guide controlled learning to teaching applications of mixed and to be evident also in several MIR- pupil or visitor oriented learning. augmented reality are still minimal. ACLE case studies. People were Usability, availability and price of The fast development of new learning also phenomenon in their

18 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE

MIRACLE & THE 5-E REPORTED LEARNING and learner differences. pupils, who otherwise belong to MODEL RESULTS BY USING AR- Authenticity of science learn- the lowest achieving school suc- SOLUTIONS ing[3] has been found to be an ef- cess group. They were reaching ENGAGEMENT fective motivating factor. AR edu- up the gap with other students Object, event or question used to en- One indicator of the AR boom in ed- cational solutions have been trying while learning science. On the gage users and visitors. Connections ucation is that there have been sev- to use it as the starting point while other hand, the students with facilitated between what visitors eral recent meta-analyses about the designing AR simulations. In a the high-performance school know and can do. topic. A number of those are listed study[4] related to role playing by success gained more challenge in the references section. In Radu’s AR, the topic to be learned was and quality for the learning out- EXPLORATION “Augmented reality in education: disease transmission that the stu- comes. Objects and phenomena are ex- A meta-review and cross-media dents could affect through their • Video games and computer plored. Hands-on activities, with analysis”[5] AR solutions are com- own actions. According to the re- based entertainment and seri- guidance. pared to non-AR solutions. The sults this kind of ‘participatory re- ous pc-educational games have positive outcomes of AR include ality’ game has made new kinds of traditionally been more bene- EXPLANATION increased content understanding, authentic science inquiry experi- ficial for the boys. However, in Visitors explain their understand- learning spatial structure and func- ences possible. This approach has this MIRACLE-case there was ing of concepts and processes. New tion, learning language associa- become known globally during the no gap between boys and girls in concepts and skills are introduced as tions (like the meaning of written Pokémon GO game trend in 2016. post-knowledge testing; thus the conceptual clarity and cohesion are words,[1]), retention in long-term girls benefitted more from the sought. memory, improved physical task THE MAIN RESULTS BY informal learning experience performance and increased student INFORMAL VISITORS than the boys. ELABORATION motivation. On the other hand, the • Girls had a higher relative au- Activities allow learners to apply con- negative outcomes were as follows: • Augmented Reality advances the tonomy experience (RAI) as an cepts in contexts, and build on or ex- attention tunneling (focusing at- learning especially in informal important background factor for tend understanding and skill. tention to a particular channel of science education context. high-performance learning. information, which might to lead to • Making the invisible observable • Meanwhile, situation motiva- EVALUATION neglect of relevant events on other is possible by Augmented Reali- tion was much more strongly AR-appliers assess their knowledge, channels,[2]), usability difficulties, ty. It gives several opportunities inter-connected among the boys. skills and abilities. Activities permit ineffective classroom integration to learn highly abstract things in • AR seems to be also a good tool evaluation of student development a concrete manner. for different learners. It is bridg- and lesson effectiveness. The 5E-model suits very well as the peda- • The MIRACLE results showed ing the gap between formal edu- gogical framework for MIRACLE Augment- that AR-technology experience cation and informal learning in ed Reality approach. was beneficial especially for the an effective way.

19 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE

RESULTS OF TEACHER BY learning environments, such as the REFERENCES TEACHER EXPERIENCES topics of MIRACLE. Radu[5] names two weaknesses 1. Freitas, R., & Campos, P. (2008). SMART: 6. Mattila, A. (2001). Seeing things in a • Moving from a teacher-con- of using AR in education: a) inef- A SysteM of Augmented Reality for new light: Reframing in therapeutic trolled learning environment to- fective classroom integration and Teaching 2nd Grade Students. Pro- conversation. Helsinki: Rehabilitation wards a pupil-orientated learn- b) learner differences. But, actually, ceedings of the 22nd British HCI Group Foundation, Research Reports 67/2001. ing one. we use the re-framing approach[6] Annual Conference on People and • Erno Lehtinen, Marja Vauras, Mar- • Connecting MIRACLE & AR to change the ‘weaknesses’ into Computers: Culture, Creativity, Interac- ja-Kristiina Lerkkanen 2016 Kasva- with and between learning envi- strengths. The ‘ineffective class- tion, 2, 27–30. tuspsykologia ronments. room integration’ gives the oppor- 2. Wickens, Christopher D. and Alexander, • Seeing remarkable changes in tunity for especially out-of-school • Jon Cabiria 2014, Augmenting Engage- Amy L.(2009) Attentional Tunneling the roles and responsibilities of methods to bridge the gap between ment: Augmented Reality in Education and Task Management in Synthetic students and teachers towards formal education and informal Vision Displays, The International • Taina Joutsenvirta, Arja Kukkonen 2009, student centeredness, autonomy, learning via AR. Also, the ‘learn- Journal of Aviation Psychology, 19: 2, Sulautuva opetus — uusi tapa opiskella and peer collaboration. er differences’ is also an important 182–199. ja opettaa • Teachers were also seeing struc- factor. MIRACLE/AR clearly offers tural changes in the learning en- new types of learning paths for dif- 3. Martin Braund & Michael Reiss (2006) • Maijaliisa Rauste-von Wright, Johan vironments. ferent types of pupils. Augmented Towards a More Authentic Science von Wright 2003, Oppiminen ja koulu- • It gives several options for utilis- Reality seems according the MIRA- Curriculum: The contribution of out‐of‐ tus ing the phenomena-based-ped- CLE experiences to be turning out school learning, International Journal • Perusopetuksen opetussuunnitelman agogy outspoken in the new to be one of the very few and rare of Science Education, 28:12, 1373–1388. perusteet 2014 National Curriculum 2016 of pedagogical solutions that benefits 4. Rosenbaum, E., Klopfer, E., & Perry, J. Finland. those pupils who are below average • Lukion opetussuunnitelman perusteet (2007) On Location Learning Authentic in traditional school success and 2015 Applied Science with Networked Aug- The question related to the muse- achievement level. mented Realities. Journal of Science ums, exhibitions, science centres, Education and Technology, 16(1), 31–45. and other informal leisure time activities is whether they are capable 5. Radu, I. (2014) Augmented reality in ed- of managing to orient and enhance ucation: a meta-review and cross-me- the momentary, strong situational dia analysis. Personal and Ubiquitous motivation into a long-lasting in- Computing 18: 1533. trinsic motivation. This is also one of the biggest challenges for open

20 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE 2.5 BUSINESS ASPECTS OF CULTURE-ORIENTED MIXED REALITY MUSEUMS AND Many of the sites and organizations cialism itself is a controversial no- CREATING THE DEMAND CULTURAL HERITAGE today are trying to cope with es- tion within the cultural travel sec- SITES AS CLIENTS pecially decreasing public funding. tor. Consisting mostly of non-profit Augmented and mixed reality is This has made them aware of the organizations the sector has only used to make museum visit more Mixed Reality experiences in Cul- need to raise additional income, gradually awakened to the realiza- interesting and engaging to visitors. tural travel sector will most likely to even from commercial activities. As tion that strive towards profitabili- However, the applications currently become profitable business for en- a result, the field has been divided. ty, at some extent at least, might be available are primarily individual terprises in near future. Currently, On one hand are those who stress unavoidable for maintaining their and expensive productions. there is already a growing need for that museums and other agents activity. When offering MR solutions to companies producing applications on the field must respond to the MR for museums and cultural museums, the key question is how that can provide cultural travellers consumer-led society and provide heritage sites offer another content to convince the customer how MR with both experiences and infor- more what visitors want. On the layer in the overall experience. The helps the museums to make their mation about cultural heritage sites. other hand are the critics accusing MR therefore is a gamified or other- collections and cultural heritage in To be able to answer this need and the organizations of oversimplifica- wise digitalised level of the experi- general more accessible and engag- to deliver their products on a reations and preferring entertainment ence, an experience which itself and ing to visitors by using interactive sonable price, companies require to education. even without the MR has gathered digital content. Referring to the information about 1) needs and Paradoxically, at the same time the audience. Hence the MR expe- usability study results (e.g. publica- purchasing habits of the Museums the reducing public funding has rience is not be-all-end-all solution tions of MIRACLE project) can be and other Cultural Heritage (CH) made the field commercially more but only an augmentation of that of great help. In addition, the mu- sites, 2) cost efficient production involved, it also has made the field what already is and should be con- seum experts need to be convinced workflow which involves collabora- more unwilling to take any risks. sidered as such. This can generate that MR also creates new co-creation and co-creation with both cus- Adapting too novel technologies additional visitors to a museum or tion possibilities for cooperation of tomers and other stakeholders and or business models not tested else- cultural heritage site, but the deliv- cultural heritage professionals, his- 3) the needs of the end users of the where might seem like waste of ery method for acquiring the best torians and technology researchers. applications already scarce resources. The pos- cost-benefit ratio is yet under re- In addition, this technology devel- Museums and cultural travel sibilities of activating new visitor search and thus the business model ops profoundly new ways for curat- sites have their special character- groups or gaining more income is important in finding the optimal ing exhibitions. istics which must be taken into are not viewed as results alluring method. The key issue in getting the cus- account when considering offer- enough or worth the risk. tomer interested is how the supplier ing MR solutions and technologies. Furthermore, even the commer- can communicate mixed reality in

21 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE its various forms as a new genre that CO-CREATION OF THE MR understanding both needs and pur- content for the MR applications will interest audiences, and that its EXPERIENCE chasing behaviour of Museums and that are part of exhibitions — if they possibilities are growing constant- Cultural heritage sites will be essen- only have such tools that are easy ly as the development of the tech- When done correctly, the fields of tial for these companies for access- to use. Therefore, the need for technology proceeds. Mixed reality history, museology and cultural ing this market. nical development work requir- can successfully compete with the tourism will all benefit from MR In order to ensure cost-efficient ing deep technological knowledge traditional storytelling and exhibi- technology as a new medium for production of MR applications, the has to be minimized and could be tion techniques, including relatively communication between history knowledge and content creation outsourced to local SME’s. In ad- modern tools like multimedia. MR experts and their audience. It can process has to be streamlined, and dition, there might be a need to is versatile, interactive and new, so improve the user experience, ac- the development and authoring develop completely new authoring it can offer more than the tradition- cessibility and the way pedagogy in tool support has to be good enough. tools that are targeted directly for al tools. However, it is important to museums, archives, libraries etc. is As the museums are often already the museum staff. Some enterpris- ensure that the quality of content conducted employing researchers, curators es can concentrate in making the is kept high. Even minor problems The emerging ‘MR industry’ in- and other relevant personnel, the applications, creating content and can spoil the entire experience. cludes new roles for existing actors. museum staff can already produce running the services, while some The ways of obtaining informa- It can be visualized by locating action are changing rapidly in the tors with different roles into a val- Content production Visualization of the Delivery of the MR modern society. Curiosity and will ue-network, visualized in image 1. (b2b) content (b2b) experience (also b2c) to learn new things have all but di- In the value network, new inno- minished; they only have taken an- vation capacity for Cultural travel Producers of historical data Production of visualized content Places for delivery · Museum staff · Acting for 3D conversion · Museum other form. Text is losing ground to not only emerges, but also converts · Research institutes · Scanning of spaces and items · Exhibition · Archaeological excavators · Creation of 3D environments · Historical site (ruins etc.) other media, such as different types into actual business between stake- · Architects · Authoring tools of multimedia. Mixed reality can holders in cultural heritage sector · Producers of recreational User content Software · Museum visitor answer to these needs. Combining and local SMEs. While planning · Writers · User interfaces · Event participant · Gamified content design · Embedded software · Researcher (tool for research) physical and digital environments and building the MR applications · Games · Museum staff (authoring tools) Producers of commercial input · gives new opportunities to learning in MIRACLE project, we identified · Linkin commercial content Hardware and installations design Interaction and social experience · Content differentiation (niches) · AR devices · One-way vs. Interactive experiences that would not be pos- local networks of culture and me- · Mobile devices · Recreational vs. Educational sible without mixed reality. dia enterprises suitable for an active New Business (to be validated) · Exhibition design, lighting etc. · Generation of new knowledge tool for researchers role in MR business and involved A) Business models specialized in one part of the value chain relevant companies to creation B) Business models through building and managing networks of the applications as contractors. (e.g. deliveries of complete MR/AR projects) Managing the supply and production together with other actors and Image 1: Value network for Cultural Mixed Reality Applications

22 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE others may develop the tools and tomer runs the exhibitions and oth- REFERENCES methods and make business selling er activities on yearly budget, large them. This process is referred as the investments in exhibition technol- 1. French, Y. and Runyard, S. 2011. Mar- MR contents value chain and it in- ogy can be difficult to justify. This keting and Public Relations for Muse- cludes actors from content creation, can be the case with permanent ums, Galleries, Cultural and Heritage visualization and delivery. exhibitions and infrastructure. In Attractions. Routledge. London and these cases we encourage provid- New York 2011. PRICING AND BUSINESS ers to suggest pay-per-month pric- MODEL ing for whole upkeep. This not only lowers the purchasing threshold but When making business model and it also brings constant cash flow of- pricing decisions, it has to be taken ten important for a small company. into account in which ways muse- It is notable, that museums op- ums organise their funding. In some erate with both project-type and cases, museums will get a specific, yearly budgets. This means that the project-type, budget for building an provider of an MR service needs to exhibition. Regardless of whether be ready for offering the content to this kind of funding comes from a be purchased in both ways. sponsor or from the museum itself, it will be more or less a lump sum and has to be enough for almost everything related to the exhibition. In this scenario, the customer should be offered MR services with one fixed price, including the upkeep of the application. If any extra costs will occur later, the customer may face difficulties in finding the missing financial resources. This is also the case with upkeep and service functions — they have to be included in the price. If, on the other hand, the cus-

23 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE 2.6 CO-OPERATION IN MULTIDISCIPLINARY FIELD KNOWLEDGE AND SKILLS is highly advisable. curators in heritage organizations. required amount of cooperation REQUIRED TO CREATE The most important specialists Therefore, it could be fruitful to en- between different sectors varies AND MAINTAIN MR to be included in projects involv- able the conversation between these remarkably from application type APPLICATIONS ing creation of MR applications for two groups when designing the his- to another. Creating technically cultural heritage sites are curators torical content and the stories to be simple small scale MR applications Experiences gained during the Fu- and various types of IT specialists told in and with the applications to does not necessarily require any ad- turistic History and MIRACLE like programmers and 3D modelers. be created. Involving scriptwriters ditional experts as the needed ex- projects indicate that creating MR The curators have the expertise on could also be extremely useful for pertise can be found in the heritage applications for cultural heritage the historical facts and they usually creating a good and engaging story. organizations. The more complex sites in current state of technical have the clearest vision of what is The visual image of an appli- the technology used is the more IT development requires cooperation suitable and desirable for the mu- cation is extremely important in specialists will be needed and the between experts of various sectors. seums and the organizations they making it appealing for the users. more probably the work should be It is improbable that the situation represent. The IT specialists have Therefore it might be worthwhile assigned to an enterprise with ex- will change dramatically in next the needed technological skills the to have also people with artistic perience of similar processes. The few years. Using the available plat- museum personnel may lack and expertise involved. Cultural herit- more diverse the desired historical forms, software and tools for appli- they should have adequate expe- age sites are closely linked to travel content is the more historians out- cation building and content crea- rience in creating similar applica- sector since travelers are along with side the heritage sector should be tion demands familiarity with the tions. Representing typically com- school classes the most important involved. The higher the commer- technology and special expertise panies or enterprises, they may visitor group at the sites. The exper- cial or attraction potential of the that curators and other museum work on clearly different conditions tise of the travel sector is required application is estimated to be, the staff currently cannot be expected than the museum staff, a fact of to answer to the needs of the trav- more experts from tourism and to possess. On the other hand, IT which the latter should be aware of. elers. School teachers again are the marketing sectors should take part specialists may not have sufficient Besides these two main groups best to advice on how to connect in designing them. It is essential to understanding of museological ba- a few other sectors may have sig- the content in the applications with assess the required size and com- sic principles or museum pedagogy nificant benefits to offer. Research- the school curriculum. Marketing position of the team for application and they need more fundamental ers of history at universities have experts on the other hand could be creation in advance. assistance than delivering mere knowledge on current trends and consulted on how to amalgamate One interesting aspect is the pos- historical facts in designing the ap- latest research results of research of commercial information with the sibility to simplify the creation and plications. In addition, involving history. Usually their point of view applications in a discrete way. maintenance processes of cultural various other sectors in the process differs at some extent from that of It must be kept in mind that the heritage applications so that herit-

24 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE age organizations could insert the information mainly in visual form. of for example marketing and legal more fruitful if some time is re- content they have produced directly The role of language in mediating issues. These institutions have the served for it in the beginning of the to the application platform without the museum experience is small- strongest changes of getting by the project and if emphasis is put on any outside technical support. er than in traditional exhibitions building of small scale MR appli- explaining working methods, val- or when the historical knowledge cations with their own human re- ues etc. of the different sectors. This TAKING ALL SKILLS INTO is published as books or articles. sources. The smaller a museum or became clear during the creation of USE Converting the existing histori- organization is the more probably it the Holy Ghost Church application cal research (published in textual will need help from several sectors in the beginning of the Futuristic The historians, curators and other form) into images and finding suit- and enterprises. History project. The cooperation members of the heritage organi- able visuals for the purposes of the between the different branches in- zations are experts on the histori- content creation will demand time. ORGANIZING THE CO- volved in the project became more cal knowledge, which should be in This must be taken into account in OPERATION effective after the researchers were center in every application created budgeting time and resources for able to understand each other’s for cultural heritage sector. Mem- this kind of a project. The pilot application projects have ways of thinking, without necessar- bers of the museum staff and the Besides the historical knowledge shown that it is most useful if all ily knowing much about even the cultural heritage organizations and the knowledge of the principles the experts taking part in applica- basic facts of the other sectors at have the best knowledge of the prin- of presentation adapted by the her- tion development would have the that point. ciples on which presenting the past itage organizations the historians basic knowledge also of the other The basic rule for efficient distri- is based in their organizations. His- and curators can be expected to sectors involved in the project. This bution of work is naturally that all torians on the other hand might be possess some other expertise that is essential for making the cooper- specialists involved concentrate on better aware of the various aspects might be useful in creating MR ap- ation fluent and for avoiding waste their own sectors of expertise. This in history which could be presented plications for heritage sites. Usual- of time caused by misunderstand- rule must be supplemented with the in a particular site with the help of ly even the smaller museums have ings. Learning to understand work- notion that the experts should also the MR technology. personnel with IT skills beyond the ing methods, mindsets, values and pose questions to the other special- The amount of work needed from basic level. These skills are becom- the basic vocabulary of the other ists about the aspects and facts they historians and curators depends on ing more common also among re- sectors is more important for func- consider unclear. These questions how much suitable material there searchers of history at universities. tioning cooperation than learning may help the specialists to under- is available for the content crea- The heritage organizations have mere facts like names and years, or stand which of the things they hold tion. It must be stressed that textu- also strong expertise on museum using of particular computer pro- self-evident need to be explained al material needs more processing pedagogy and the educational po- grams. and justified more clearly. than images or objects to be fitting tential of the heritage sites and mu- The best way to learn to work Those involved should also ex- for the content creation, since the seums. The large organizations and together is to work together. The press clearly what they need from applications present the historical museums usually have specialists learning process will, however, be the other experts. In practice this

25 2. MIXED REALITY IN TOURISTIC AND EDUCATIONAL USE means that for example the 3D processes should be integrated as modelers should describe in de- part of the duties of the employees tail what kind of reference images involved. they need from the heritage organizations or the historians. Also the organizations commissioning an application must keep in mind that without proper instructions the enterprises or IT specialists responsi- ble for building the application cannot work as efficiently as possible. Constant communication between the different disciplines and sectors is one of the most important factories for efficient and successful process. There should be at least one specialist of each involved sector to be consulted during the whole process. It is difficult to -or ganize the work in succeeding se- quences so that specialists of each sector would be needed only a short period of time. On the other hand it is obvious that all the people involved cannot be and are not needed to be working full time during the entire process. There will be more intensive periods and periods when only some assistance will be required. This means that within the heritage organizations as well as in the enterprises the participation in individual application creation

26 3. PRODUCTION GUIDELINES 3. PRODUCTION GUIDELINES

This chapter describes technologies and processes that are needed in the production of MR applications. Guidelines are given about choosing suitable methods and tools for different cases. Main focus areas are storytelling, scripting and visual content creation.

27 3. PRODUCTION GUIDELINES 3.1 SOFTWARE AND HARDWARE PLATFORMS SOFTWARE and extensions to the system, and and the iOS 11 operating system droid and iOS. typically other separate tools are to be released in the fall 2017. Cur- • Layar[7] includes an application, There are several types of software used in projects to make content. rently it seems to be an Apple pro- a creator and an SDK. The crea- platforms and frameworks that can prietary platform running on Ap- tor is a drag and drop -style ed- be applied to creation of mixed re- In June 2017, i.e. after the MIRA- ple hardware only and not on other itor, which allows users to easily ality experiences. Which ones to CLE project, Apple published the mobile platforms. create AR content for the Layar choose depends on the case and in- ARKit platform[3] for iOS 11. It is a app. The app itself can be used to tended uses. Some of the currently framework for creating augment- AR CREATION AND view all AR content created with available, relatively popular options ed reality experiences for Apple’s PRESENTATION the creator. Layar also provides are briefly described here. mobile devices, iPhones and iPads. SOLUTIONS services for creating custom tai- ARKit is a result of Apple’s 2015 ac- lored AR applications. Android RICH CONTENT CREATION quisition of Metaio and other tech- There are numerous providers of and iOS are supported platforms. & DEVELOPMENT nology companies, and developing AR browsers and mixed reality de- • Wikitude[8] is an augmented PLATFORMS their solutions into a complete plat- velopment systems. Many of those reality solution that supports form. ARKit is expected to have a are consumer-oriented in the sense a wide range of tracking tech- Unity[1] and Unreal Engine[2] are major effect in bringing AR into that content creation does not re- nologies and many AR devices, two widely used 3D game engines. wide use in consumer devices and quire much training or traditional including see-through glasses. Both of them consist of a wide range in many application areas. software development skills. Some Wikitude Studio is an easy-to of tools for building 3D games — or ARKit uses camera data — from examples of those are briefly pre- use tool included in the system, other applications — and deploying both front and back cameras — and sented below. A more extensive list allowing AR content creation them on the common platforms, motion sensor data for tracking. It of current (and also some inactive) without programming skills. including Windows, Mac, Android can find horizontal planes, like ta- MR solutions can be found in the Also professional software de- and iOS. Although intended orig- bles and floors, where virtual ob- MR database[4] made by University velopment frameworks such as inally for game development, and jects may be placed, and it can also of Turku within the MARIN2 pro- Cordova, Titanium and Xama- especially games in 3D virtual en- estimate the lighting in a scene and ject[5]. rin can be used with Wikitude. vironments, these development en- apply simulated light and shadows • Arilyn[6] is a Finland-based com- • ALVAR[9] is an AR/VR software gines can also be used for making to virtual objects. pany that offers a traditional library and development plat- mixed reality experiences. Both ARKit runs on Apple’s high-end AR-browser and Content Man- form developed by VTT, Fin- have a plugin framework that al- processors (A9 and A10) that are agement System (Arilyn Manag- land. It offers both marker-based lows adding many kinds of tools found on latest product generations, er) combo. Arilyn supports An- and markerless visual tracking

28 3. PRODUCTION GUIDELINES

technologies, and it was used in VISUAL CONTENT gyroscopic sensors, GPS) to run Augmented Reality several of the pilot cases in MIR- SHARING even fairly complex augmented re- • Microsoft HoloLens[14]: stan- ACLE. ality applications. Capabilities for dalone augmented reality glasses • Vuforia[10] is a development plat- Many kinds of public services exist movement sensing depend on the with high quality tracking of the form for mobile AR. It can rec- for sharing and distributing visual built-in sensors and software, and surroundings. Can be controlled ognize many types of objects, content. devices on the lowest price cate- with hand gestures or simple like photographic images, mark- • Sketchfab[12] is a site with lots of gories may fail to run demanding hand controllers. ers, text, boxes, cylinders and shared 3D and VR visual con- apps, so it may be necessary to set • Meta2[15]: tethered see-through other 3D objects, to be used as tent, covering all kinds of subject some system requirements to target augmented reality glasses. anchors for positioning the vir- areas from cultural heritage and devices, and perhaps make a list of • Daqri Smart Helmet[16]: hel- tual content. art to technology. recommended devices. It is an im- met with standalone AR glasses. • ARToolKit[11] is an open source • Destinations VR application[13] portant decision where to draw the Daqri makes also smart glasses tracking and recognition SDK by Valve enables creation, shar- line for performance requirements, with a small computing unit, for augmented reality applica- ing and exploring different i.e. what kind of compromise to sensors and HUD technology. tion development. Android, iOS, worlds in VR: photogrammetry make between how sophisticated • ODG[17]: standalone, integrated macOS and Linux are supported scenes, game worlds, or anything the application is and how large AR smart glasses target platforms, and the license else that can be represented by percentage of consumers’ devices • Google Tango[18]: A technolo- allows both non-commercial a 3D model. The virtual worlds can run it. gy platform enabling AR expe- and commercial use. can be explored with VR glasses, It is also possible to design the riences with smartphones and and it is fairly straightforward application to be run on dedicated tablets. Tango enabled devices to create new destinations using augmented reality or virtual reality have sensors for accurate mo- phtogrammetry. devices. There are many types with tion tracking and area learning varying features, and more prod- enabling a wide range of applica- HARDWARE ucts are introduced all the time. tions to be run. Many AR and VR glasses at the mo- A viable approach to offering AR ment are tethered, i.e. connected to Desktop Virtual Reality for the public is to make it work a host computer by a wire. It can be • HTC Vive[19]: outside in tracking on common devices that people assumed that new products in the VR mask and dedicated hand carry with them everywhere, all future will be either wireless or ful- controllers. the time: smartphones and tablets. ly standalone. • Oculus Rift[20]: outside in track- Many current devices are powerful This is a selection of commonly ing VR mask, Oculus Touch enough and equipped with neces- known products as of spring 2017. hand controllers can be added. sary sensors (camera, inertial and

29 3. PRODUCTION GUIDELINES

HTC Vive and Oculus Rift are sim- Mobile Virtual Reality Controllers ilar products with offer a nearly full A popular type of mobile VR prod- • HTC Vive Controllers & Track- field of vision for the user, possibil- ucts are the simple head-worn ers are part of the base system. ity of free moving within a limited masks where a smartphone is used • Oculus Touch hand controllers area, and a good impression of “be- as a display and computing unit. are supported in the Oculus Rift ing there” in a virtual environment. Rotational sensors are used to track platform but (so far) sold as sep- • Microsoft[21] has a mixed reali- the user’s head movements. arate products. ty platform on which different • DayDream[22]: Google’s VR • LeapMotion[25] produces desk- manufacturers have announced mask, supports a simple hand top and laptop hand controllers products. These are also 3D controller. sensing the user’s hand move- headsets that occlude the user’s • GearVR[23]: Samsung and Oc- ments. natural view and are tethered to ulus branded VR mask with a a computer. hand controller support, to be used with a Galaxy smartphone. • Google Cardboard[24] is a sim- TERMINOLOGY ple cardboard mask with lenses, having the same idea of using inside out tracking: the device has a smartphone as the comput- sensors scanning the surroundings ing and display device. Howev- outside in tracking: cameras/sen- er, there are no additional hand sors are set around the usage area, controllers or electric connec- tracking the movements of the mask tions between the viewer mask and controllers and phone, so the viewer can be rotational: the device has sensors extremely simple and cheap, and for rotational movements, but linear many phone models can be used. movements of the device/user are The concept has been copied by not (accurately) tracked many since Google introduced standalone: does not need a con- it. It has been applied in market- nected computer, all processing is ing campaigns, as it is possible done in the device itself to distribute a cardboard viewer tethered: the device is connected to even as part of product packag- a computer with a wire ing with minimal cost.

30 3. PRODUCTION GUIDELINES

REFERENCES 1. Unity game engine: 13. Destinations: 23. Samsung GearVR: https://unity3d.com/ https://developer.valvesoftware.com/ https://www.oculus.com/gear-vr/ wiki/Destinations 2. Unreal Engine: 24. Google Cardboard: https://www.unrealengine.com/ 14. Microsoft HoloLens: https://vr.google.com/cardboard/ https://www.microsoft.com/en-us/ 3. Apple ARKit: 25. LeapMotion: hololens https://developer.apple.com/arkit/ https://www.leapmotion.com/ 15. Meta: 4. MR database: https://www.metavision.com/ https://ar.utu.fi/mrdb 16. Daqri Smart Helmet: 5. MARIN2 project: https://daqri.com/products/smart-hel- https://ar.utu.fi/research/marin2/ met/ 6. Arilyn: 17. ODG: http://www.arilyn.fi/ https://shop.osterhoutgroup.com/ 7. Layar: products/r-7-glasses-system https://www.layar.com/ 18. Google Tango: 8. Wikitude: https://get.google.com/tango/apps/ http://www.wikitude.com/ 19. HTC Vive: 9. ALVAR by VTT: https://www.vive.com/eu/ http://virtual.vtt.fi/virtual/proj2/multi- 20. Oculus Rift: media/alvar/index.html https://www.oculus.com/rift/ 10. Vuforia: 21. Microsoft Mixed Reality: https://www.vuforia.com/ https://developer.microsoft.com/en-us/ 11. ARToolKit: windows/mixed-reality https://www.artoolkit.org/ 22. Google DayDream: 12. Sketchfab: https://vr.google.com/daydream/ https://sketchfab.com/

31 3. PRODUCTION GUIDELINES 3.2 TRACKING

In order to correctly align virtu- accuracy applications. nice-looking color images instead al objects with the physical world, Outside-in camera tracking of b/w markers; this works well for Augmented Reality requires find- means tracking of the device with many close range applications, but ing out the user’s exact position and externally positioned cameras, the operating distance is even more viewing direction (also known as LEDs, lasers or other e.g. magnetic restricted than with markers. the pose), and then keeping track sensors. This works only in limited Instead of 2D surface based of the movements. In practice this spaces and requires special imple- positioning, it is also possible to is equal to tracking the viewing de- mentations. This approach is used use three-dimensional objects as vice’s camera pose. Several methods for example in HTC Vive and Ocu- positioning references. The term are available for the tracking task. lus Rift headsets. “markerless tracking” means using It is application-dependent which Marker based tracking, typ- existing objects in an environment Image 1. A typical tracking marker. methods are actually suitable: for ically using black&white matrix for tracking, so that there is no need some cases the positioning require- shaped patterns, placed in known to attach dedicated prints for this ments can be fairly loose, while oth- locations in the environment, ena- purpose. A recent approach is to ers require location data with a few bles accurate and reliable tracking. capture the features of a known ob- millimeter accuracy. In a changing The input data from a mobile de- ject in 3D space as a “point cloud”. environment the system must be vice’s camera is analyzed in near- Point cloud based tracking, able to adapt to the situation while ly real time and searched for the based on 3D point clouds and fea- the application is running. known image. When the marker is tures sets enables tracking of com- recognized, the relative position of plex and also distant targets. The TRACKING the camera can be calculated from point clouds are typically created TECHNOLOGIES the projection of the marker, and from several photos of the target, then it is finally possible to place using third party tools such as Ag- Sensors, e.g. gyro-compass com- virtual content over the input video. isoft PhotoScan[1]. Main problems bined with GPS (outdoors) or some However, there are limitations with are found with managing of dif- other location-finding system (e.g., the operating range, and in many ferent viewing distances, lighting WiFi indoors) enables to track the environments the markers may be conditions and changes in the en- device’s pose within some accuracy; considered aesthetically too dis- vironment. however the jitter and drift are typ- turbing. SLAM, simultaneous locali- ically too disturbing to enable high Image based tracking, using zation and mapping, works simi- Image 2. Example of a point cloud.

32 3. PRODUCTION GUIDELINES larly to point cloud based tracking, REFERENCES but creating the point clouds by real time scanning of the environ- 1. Agisoft PhotoScan: ment instead of pre processed point http://www.agisoft.com/ clouds. The main downside is the 2. Google Tango: need for a separate initialization https://get.google.com/tango/apps/ step. 3. Microsoft HoloLens: TRACKING PLATFORMS https://www.microsoft.com/en-us/ hololens Mixed reality platforms such as the 4. ALVAR: ones presented in the previous sec- http://virtual.vtt.fi/virtual/proj2/multi- tion include one or more support- media/alvar/index.html ed tracking technologies. Marker based, image based and point cloud based tracking options are common. The SLAM approach is typically used by depth camera based solutions. Google Tango[2] and Mi- crosoft HoloLens[3] are examples of platforms using those. Within the MIRACLE project VTT’s ALVAR[4] library and its point cloud based tracking was used in many pilot cases, and the technology was also developed further during the project. Some technical comparisons were carried out, concluding that ALVAR provided the best performance especially for 3D point cloud based tracking , being strong on 2D marker and image based tracking as well.

33 3. PRODUCTION GUIDELINES 3.3 STORYTELLING

To offer informative and entertain- ed. The experience may consist of days in the production time. We REFERENCES ing mixed reality experiences, in- more or less independent pieces think this kind of tool is useful at 1. Articy:draft: Nevigo: stead of just showroom toys, the that can be browsed in any order, least in games and other stories that http://www.nevigo.com/en/articydraft/ main effort must be put into the or the user may only pick some of have variations in the story. If the overview/ actual story. Historical and/or ed- them. Whatever the pieces are, they content is linear, with no variations, ucational content must be in focus should have an interesting expe- as is the case with traditional plays, when designing the application. rience to give to the user. If there a scripting tool may not offer much The form and content of the appli- are human characters, dialog with extra efficiency — at least if the au- cation must be coherent, but the them is expected. Non-human ob- thor is not familiar with it in ad- technical implementation and form jects should also allow the user to vance. of the application and means of make a contact with them. They representation should be chosen to may be moved, modified or just observe the purposes of the historical served, and there should be stories content, not on the contrary. Even about them that give the user some- narrative and entertaining applica- thing to think. tions containing fictional elements can (and should) be informative SCRIPTING TOOLS and educational. articy:draft[1] is a scripting tool that SCRIPT WRITING has been used in our productions. It offers the necessary functions for When there is a story to be told, it writing game scripts where the sto- should be told well. A good story ry may branch into several different is at least as important as flawless paths depending on the choices of implementation of an app. A pro- the user. fessional scriptwriter may even be The user interface inarticy:draft the best place to spend money in is simple enough to be learned in the project. a few hours or days. It is thus ben- A complete story with a begin- eficial to learn a tool like this if it ning and an end is not always need- saves more than a couple of work-

34 3. PRODUCTION GUIDELINES 3.4 CONTENT CREATION In principle, augmented (virtual) COMBINING DIFFERENT versa. In our Sanan seppä case, for should be a tool that is easy enough content can be an artificial stim- EXPERTISES example, a historian learned to use to learn and capable of producing ulus to any human sense, whether the clothes modeling software Mar- useful data for the application. visual, auditive, tactile (touch), ol- Content creation often requires ex- velous Designer[1]. It took a few days The first condition is likely to factory (smell) or flavor. However, pertise as well in the subject matter to be productive with it, whereas be fulfilled especially when we are most common is visual content (like history, architecture, art etc.) it can be estimated that the time talking about large units, like na- that is produced and displayed by as in the production tools (mode- needed for someone to get a suffi- tional or provincial museums. They digital devices. Visual content may ling, visualization, animation, in- cient level of knowledge about late often have many employees with be two-dimensional — photographs, teraction design etc.). Usually one medieval clothing would have tak- different kinds of expertise, includ- texts and graphics — or three-di- person is not an expert in both en a significantly longer time. Even ing researchers and other personnel mensional, and it may or may not of these, so it must be estimated the historian in question, whose who probably have necessary skills. be animated. whether it is more efficient to edu- fields of expertise didn’t include (For example experts on clothing The skills required to make AR cate a substance expert to use the historical clothes, 16th century or could be textile conservators, ar- content are very much application necessary tools, or to educate a tools sewing, found it much easier and cheologists, researchers who are dependent. First, expertise in the expert about the substance facts, or less time-consuming to learn to specialized in clothing collections subject matter is needed. Second, to have one of both working as a use the modeling software than to and even seamstresses). the story and application design team to produce the content. gather in-depth knowledge of 16th The second condition is heavily must be of high quality to keep us- As a general rule, it can be stat- century clothing. (If the task would dependent of the specific area of er’s interest in the subject up, and ed that when the amount of content have been to model e.g. early 20th production. Clothes design is an ex- to prevent frustrations due to tech- to be created is small, it is efficient century clothes, it could have been ample of a task for which a suitable nical or other problems in the ex- to create it by teamwork of the two vice versa, though. This is because tool could be found. However, there perience. Third, the making of the (or more) experts. The more there there would have been considera- may be tasks for which no such tool application and its elements may is material to be created, the more bly more visual information about can be found — either the tools may require specific artistic and tech- efficient it becomes to train either clothing of the ordinary people and be too complex to be learned in rea- nical skills, again dependent of the the substance expert to use the pro- original patterns to use.) sonable time, or the task is so spe- case. All these requirements are duction tools, or the tools expert to To be successful with this ap- cific that a tool hasn’t been made at discussed in more detail in the fol- master the subject matter. proach, two things are required: all, or isn’t openly available. In such lowing sections. In most cases it is probably easier the person should have the knowl- cases the way to proceed is to hire for substance experts to learn some edge and skills to design the want- a professional who can do the task specific production tools than vice ed things, like clothes, and there and perhaps make the needed tools.

35 3. PRODUCTION GUIDELINES

VISUAL CONTENT 3D CONTENT CREATION greater whole — in other words, variants. CREATION can the amount of content be Making — or purchasing — virtual scaled down? Would it be worse, Photogrammetry and laser There are different ways of visual 3D content may take a major part of as good or even better? Cutting scanning content creation. Choosing the the resources of an AR or VR pro- unnecessary stuff might make Photogrammetry is a potentially most suitable one depends on what duction. the result more focused. more economical way of producing kind of experience is wanted and Especially creating high-quality • What is good enough for this virtual versions of 3D objects and what are the resources and budget human/animal characters requires project, would less polished spaces than manually modelling available for the project. Visual highly skilled designers, and even assets fit the needs just as well? them. Photogrammetry means au- content can be based on 3D mod- then the amount of time required Typically, consistent look is aes- tomated composing of a 3D model elled virtual objects; this allows ob- may be significant. Therefore, an thetically more important than using a large number of ordinary serving the objects from any angle, important part of the production having something very polished photographs as the input data. Typ- and the user may be able to move process is to plan what content is and something quite raw in ically some tens or hundreds of pho- freely in the area where the con- actually needed and what is the best same piece. tos taken from different angles and tent is to be displayed. In case the way overall to get it — purchasing, • Can the look be simplified and distances from the target are fed to main objective is to display an ex- subcontracting, making it in the made easier to execute? Maybe a program that compares the pho- isting environment but there is no project, or going some other way? it’s not important to be realistic. tos, finds corresponding points and requirement for interactive content, The cost issues related to 3D content • Can ready-made assets be finally calculates a 3D presentation 360 degree video or photography creation may eventually affect also bought instead of making new of the target. Agisoft PhotoScan[2] may be an option. It allows just ob- the choice of the overall concept: ones? Maybe it’s not that impor- is an example of such software, serving the environment — towards what kind of story should we have, tant to have a production spe- also used in the MIRACLE project. any direction — from one point at how many characters to include in cific style. At least having such Learning photogrammetry tech- a time, but making such presenta- it, what kind of visual style should ready-made assets at the start al- niques does not require extensive tions is much cheaper than creating they have, and so on. lows fast prototyping right at the training, so it is not always neces- actual 3D content. These two meth- beginning. sary to hire professionals for the job. ods are explained in detail in the Existing or outsourced char- Laser scanning is another option following sections. When budget is limited and acters can be taken as the starting for creating 3D models of existing compromises are needed, you can point, and by reusing and modi- objects. Scanner uses a laser beam consider the following: fying them the workload can be to measure the distance from the • Can the plan be some way reduced. Non-professional design- scanner to every point in the target, reshaped? Has it too wide a ers can be trained to model simple and the resulting data is then con- scope, has it too many compet- objects, and to modify existing ob- verted to a 3D model. ing things that detract from the jects in order to produce new object Both photogrammetry and laser

36 3. PRODUCTION GUIDELINES scanning are useful for modeling ed into an existing space, creating MAKING YOUR OWN 3D artist, such digital sculpting can static, solid objects, whereas liv- a photogrammetry or 3D scanned CONTENT be quite fast way to iterate between ing characters or shape-changing model in addition to accurate meas- different designs until desired look objects require other techniques. urements is very helpful. This allows The visual work on new 3D content is born — naturally the same goes Photogrammetry does not require the artist to use and even examine typically starts by gathering refer- for 2d sketching and concept art. special hardware, whereas laser the possibly historical space on her ences, making sketches and creat- Final design is then made game- scanners are quite rare and expen- workstation for hints and markers ing concept art to define the tar- ready, typically by a process of re- sive devices, usually not worth pur- about where long-lost structures get look. For example, creating 3D topology, where underlying sculp- chasing for occasional projects. have been and where their AR rec- characters and other assets usually ture is remade by continuously reations should go. Such model also starts with drawn 2d designs, iter- adding simpler polygons on the provides a virtual reference to his- ated until final character appears. surface, or by straight low-polygon It is worth noting that to make 3D torians on the project and allows This 2d sketching step may also be modelling. Both of these methods modeled objects appear natural, for fast commentary regarding the skipped, by e.g. sculpting the char- result in a game-ready mesh, which and to make them work in the mo- authenticity of added constructs. acter digitally and thus creating the can then be UV-unwrapped for tex- bile application, may require mod- It should be noted that the pro- sketch in 3D. For an experienced turing or, if fitting, colored by ma- ifications to some objects, instead ject complexity may not be visi- of using completely accurate mod- ble from the start, for example it els. The first issue is that if modeled might be challenging to estimate strictly in the authentic scale, some asset-heavy production times, as things may appear too small in the pipeline construction, testing and application, so it may be better to asset automation all may reveal un- exaggerate some parts or things. seen problems. This also happens The other thing is that complex ob- because people may be optimistic jects may require too much process- about new project and thus coming power to play well in common monly underestimate the workload. mobile devices, so simplifying may Therefore, it is recommended to be needed. Such compromises re- test the asset creation time and, if quire cooperation between applica- needed, adapt the plan to fit what is tion designers, technology experts actually possible in given monetary and the experts of the subject mat- and timeline constraints. ter. In case of augmented reality projects, where something is to be add- Image 3: Roughly sculpted character and final low-poly character with clothing.

37 3. PRODUCTION GUIDELINES terial and vertex colors. Textures, Motion later on. which are 2d images mapped to the All of these steps may have a spe- model and can be thought as pelts cialized artist or tech artist in larg- of it, can be created by hand-painter production, for example, digital ing the unwrapped model, by as- sculptors separate from low-poly signing ready-made materials, by modellers, riggers separate from texturing procedurally, or by paint- texture artists and so on. But on ing the high-poly model and trans- smaller projects, with less demand- ferring the result to game-ready ing final look, all parts may be filled model. While character artist will by a single 3D generalist. weigh the desired result to time Pipelines are typically produc- constraints and decide the best way, tion-specific — not all styles of 3D for stylized, cartoony look the mod- models need unwrapping and tex- els are typically hand-painted while turing and not all styles need high realistic looks might use scanned definition digital sculpts. Choosing Image 4: Motion capture session during Sanan Seppä pilot project. real-world material libraries com- one that doesn’t can cut production bined with procedurally generated time and cost. If lots of similar as- can improve production time de- ments while several cameras record textures in addition to hand-paint- sets are needed, it might make sense pending on project complexity. the movements. While motion cap- ing. to partially automate asset creation, ture can be very realistic the data it For animation, this game mod- for example by morphing between MOTION produces needs manual clean-up to el also needs to be rigged, that is, different character traits (young, fix e.g. broken hand rotations, and bones to control the movement of old, thin, plump, male, female etc.) Discussion about character motion in the end fixing the mocap data body parts must be set inside the or by making the models modular in AR/VR applications typically may take equal resources to man- characters, for example, an arm (like pieces of fitting equipment). comes to a choice between manu- ual animation. While there should bone inside character’s arm. Then Game-ready models are also al animation and motion capture. be less and less capture errors as the model needs to skinned, a sold and some models or model Animating the characters “by hand” technology progresses, the motion process where points of the wire- tools have controls for changing the requires a lot of time even for a is also a question of style — how frame model are assigned to a suit- look of the character from thin to professional animator. Therefore it realistic characters are and how able bone so that they move with plump and from young to old and is common to use motion capture realistically should they move. As it. Rigged and skinned model can maybe even from realistic to styl- (mocap) technology in productions such this is also an aesthetic choice then be manually animated by an ized. These ready-made models and where the characters move a lot and and has no fixed answer, but a few animator or have motion capture character generator tools may well in different ways. In motion capture, pointers can be given. data assigned to it — see section be enough for certain projects and real human actors do the move-

38 3. PRODUCTION GUIDELINES

Consider motion capture if the 3D character models as well as sive. calibrated to the target characters. • The character motion should be the movements can be reused as Motion capture requires some To do this, the actor does a “T-pose” realistic (e.g. human perform- much as possible. Automating some training for the actors, too. They (standing hands lifted pointing to ing martial arts) and there are parts of the process would also help. need to learn how to limit the the sides) whenever entering the necessary resources for manual movements within a limited space; scene in order to let the system clean-up of mocap data if such MOTION CAPTURE movements may need to be re- identify each marker, and the pose need should arise. stricted in order to avoid losing the is afterwards adjusted by the ani- • Mocap equipment is readily In motion capture, real human ac- tracking of markers, and because mator in order to accurately fit the available, there’s lot of animators do the movements while sever- only a few persons can be recorded motion to the target character. tion needed and motion is sim- al cameras record the movements. at a time, scenes with several per- ple enough to avoid motion er- Actors wear simple dark clothing, sons must be done by combining Error handling with motion capture rors or errors in motion data can and point-like reflectors (markers) several takes, and then the timing Despite proper equipment and care- be tolerated. are attached to critical locations in and locations of the actors become ful recording process, the recorded actors’ limbs and body, as seen in challenging. Some of these issues motion data still contains errors. And, alternatively, consider manual image 6. By tracking the reflectors are easier to learn while some may Our experience is that, for instance, animation if in the recorded video by a special require more practicing. limbs may occasionally flip to un- • The motion is unrealistic, styl- software the actual movements of The number of actors that can natural positions when more than ized or cannot be captured (e.g. the actors can be recorded in 3D be recorded simultaneously is lim- one marker gets obstructed, or a dragon). space. The movements can then be ited. This is because if markers get sometimes when the actor is near • Animators are available, there applied to the virtual characters of obstructed, the movement track- the edge of the recording area, even is little animation needed and the AR/VR application. ing becomes unreliable. In simple when markers are visible. there’s no mocap equipment Motion capture can be done us- scenes, it may be possible to record Tracking errors must be cor- readily available. ing consumer-oriented devices (e.g. four persons at a time, but usual- rected by manual editing. This is Kinect[3]) but a better result and ly only two actors can be recorded routine work done by the anima- Basically, this boils down to using less work is needed if it is done in simultaneously. Thus, scenes with tor, but it can take a lot of time. In whatever you have at hand and a proper mocap studio with profes- more people must be done com- our Sanan seppä case, there were what you’re comfortable with, un- sional equipment. While the prin- bining several recordings, and that in total nearly 50 animated human less there is a clear need to use either ciple is simple, the motion capture causes challenges for timing and characters, and the length of act- one. Typically, in large production, process is complex and there is typ- movements of the actors, so that ed animation totalled to about X both methods would be employed. ically a large amount of work need- the virtual characters don’t move hours. Correcting errors from this When animated characters are ed to actually apply the movements through each other and that the material took several weeks of one included in the production, it is a to the virtual characters. This work movements are in sync. person’s time. The work included good idea to plan the work so that requires experts, so it can be expen- All recorded motions must be masking the erroneous movements

39 3. PRODUCTION GUIDELINES with static poses of limbs. ously, to avoid the work needed in entire 360 degree wide area sur- In almost all practical use cases, the If there are a lot of errors in mo- syncing the characters afterwards. rounding the camera. The video user is given a way to choose the cap data, it is advisable to make a On the other hand, recording sev- is fully spherical so that the whole viewing direction. Also, 360 videos new capture rather than correct eral characters causes more track- area is captured, also vertically. can be either regular 2D videos or the errors manually. Manual work ing errors. So, the best solution is a Spherical still images can also be 3D (stereoscopic) videos. 3D videos takes so much more time than re- compromise, and depends on many made — a well-known example are can increase the immersion par- capture that even counting the cost factors, not least the equipment that the street view images in Google ticularly in HMD viewing. of several persons and the studio, is available. Maps[4]. In most use cases the 360 vid- recapture will often become cheap- To avoid unnecessary work in 360 videos can be viewed in dif- eo is displayed to the user with the er. correcting erroneous mocap data, it ferent ways. The most immersive aim of providing as natural view as If the movements of small body might be a good idea to first build experience is when the video is possible. It is also possible to pro- parts like fingers are not essential, the scenes with uncorrected, bro- viewed with a head-mounted dis- vide different distorted views. In it is better not to capture them, as ken data, and then fix just the need- play (HMD), a.k.a., virtual reality some uses, other projections may it would consume the processed parts and reimport them to the glasses. The user can freely look be used. For example, small plan- ing power more than it is actually game engine. around in different directions by et projection (the technical term is worth for. Facial details are another During this project, Unity did simply turning his/her head. 360 stereographic projection) is often example of this dilemma: captur- not contain a timeline tool for easy videos can also be beneficial with used as it provides a very interesting those with high accuracy takes timing control of scenes. Our solu- systems where the projections ing and entertaining view. Other a lot of processing power, but facial tion was to use animation events or displays cover large areas sur- distortions and effects are also pos- expressions may be essential for the triggered by one or more anima- rounding the user. With mobile sible, which may, for example, ena- user experience. So the decision tions, and a specific content manag- devices, viewing is possible with a ble making the video more interac- must be made case by case, com- er script that reacts to those events so called “magic lens” interface, i.e., tive. To store 360 videos, the entire paring the costs and benefits. and drives other animations in the orientation of the hand-held de- spherical video is usually projected Inverse Kinetics (IK) can be used turn. Unity will most probably get vice determines, which part of the into a regular, rectangular video when correcting the movements of a timeline in the future, so this task video is displayed. The viewer can using equirectangular projection. limbs. In it, e.g. arms are animated will become easier to handle. then naturally choose the viewing This enables the video to be stored by moving the extremities (hand), direction by moving the device using regular video file formats. which in turn moves the connected 360 IMAGERY around. 360 videos are often also body parts. Unity’s Mecanim sys- viewed with regular desktop com- 360 video versus 3D content tem contains such IK functionality. What is a 360 video? puters. For example, YouTube sup- In immersive virtual reality, 360 It would be economical to make 360 video, also known as omnidi- ports 360 video viewing so that the video can be contrasted to 3D the motion recordings with as many rectional video (ODV) is a video, user chooses the viewing direction content created either by manual- characters as possible simultane- where the captured view covers the by dragging the video with mouse. ly modeling (or utilizing existing

40 3. PRODUCTION GUIDELINES

CAD models etc.) or by capturing the camera sets: automated solutions. a computer-based software is also objects and environments, for ex- The camera sets include a mini- Stereoscopy support, i.e., wheth- used in some systems. ample, with photogrammetry or mum of two cameras with (at least) er the camera can capture 3D video Camera synchronization is im- laser scanning. 180 degree wide angle lenses. (A is a distinct feature separating cam- portant especially with very fast Creating 3D content is in most single camera with a set of mirrors eras into two groups. moving objects and scenes. Best cases a time-consuming process can also provide close to 360 video Live streaming of the 360 video solutions are genlocked, i.e., every which requires high expertise. In but such systems are not very com- is possible with some of the solu- camera captures the frame at ex- contrast, capturing 360 videos mon today.) Many solutions consist tions. However, in many cases, the actly the same time. This combined is possible with very reasonable of more cameras. From the user resolutions of the live video is lim- with global shutter cameras (as op- amount of training. The 360 video point of view, the number of cam- ited. posed to rolling shutter) with the also captures a lot of details, and eras is not particularly important. The resolution of the video is same exposure time in each camera in cases where the content should More cameras may provide fewer very important in 360 videos. Since enables robust stitching of videos match some real environment 360 artefacts from the stitching of the one video file must contain infor- with fast moving objects with any videos can be particularly valuable. separate pictures streams into a sin- mation on all different directions, exposure time setting. The weakness of 360 videos is that gle video. the video must have much higher it limits the user’s movement in the Stitching process, where sepa- resolution than a regular video. At virtual world. The user can only rate videos are turned into one, var- the moment, 4K resolution is con- move between separate videos, un- ies between the systems. Many older sidered adequate for most uses but like with full 3D content, where free systems require some manual work, even that resolution is far from op- exploration can be enabled. like synchronizing the separate timal. Some consumer grade solu- videos and optimizing stitching pa- tions provide only Full HD video, Creating 360 videos rameters. Most newer cameras, in which results in very low quality A special camera or camera set is particular the consumer grade solu- when viewed, e.g., with a HMD. required to capture a 360 video. tions, take care of the stitching au- Robustness of the cameras also Many such systems are available tomatically. The benefit of manual varies. If there is need, e.g., to attach and new ones are being released all stitching is that it enables some op- the camera to the exterior of a mov- the time. Image 5 presents a pop- timizations, e.g., small objects like ing vehicle, some setups may be too ular consumer-oriented 360 cam- part of a tripod holding the camera expensive and fragile. era, Ricoh Theta[5]. Price range for can be removed by carefully select- Usability of the setups varies as 360-capable products starts from a ing which of the overlapping vide- well. In some systems, video cap- Image 5. An example of a consumer-grade few hundred euros and goes to tens os to use for each direction. Using turing may require activation of 360 camera: Ricoh Theta, which has two of thousands of euros. There are high end algorithms can also result each camera while others provide 180-degree cameras covering together the several aspects that differ between in a better picture quality than fast, simple remote control. Control via full sphere around the device.

41 3. PRODUCTION GUIDELINES

When a 360 video is captured, there fully. Having at least a part of the benefits of 360 videos since to achieve. are various challenges and aspects the view stable in the camera it can put the viewer into a po- Compared with the real-time to consider: view can reduce the unpleasant sition they would not be able to graphics used in the comput- • Lighting can be challenging in effects. When the camera is at- go in real life. However, since er games, the difficulty with the 360 video shooting. Since all di- tached to, e.g., a moving vehicle, HMD-based 360 video viewing mixed reality image quality comes rections are captured, it is hard the attachment should also be is usually a very immersive ex- from the need to embed the vir- or impossible to hide any lights robust to minimize camera vi- perience, high camera positions tual elements with the view of the used for the capture. In uncon- brations. may cause feelings of vertigo in real world. In order to have identi- trolled conditions, e.g., outdoors, • Since the videos are stitched some users. This may happen cal visual quality, the mixed reality there are also often great differ- from two or more original vid- even when the camera is in quite application needs to be able to sim- ences in natural illumination in eos captured from slightly dif- natural location but near to a ulate lighting conditions and mate- different directions. On one side ferent points of view (difference ledge, for example. rials of the real world when draw- of a scene there may be direct depending on camera setup), ing the virtual objects. Also, when sunshine, while another direc- objects close to the camera are VISUAL PROCESSING augmenting view of the real world tion may be in complete shadow. most problematic for stitching. captured by a camera, as is the case Depending on the tools, camera Furthermore, especially in im- Quality of the visual output pro- with augmented reality applications parameters may allow adjusting mersive viewing, objects close duced by the mixed reality applica- used with the mobile phones, vir- the lighting parameters but the to camera may feel unnatural to tion can have a significant impact tual elements drawn by the appli- decisions between over-expos- viewers. Therefore, in most cases, on the user’s experience. Often cation should be able to mimic the ing some areas or under-expos- the video should not contain ob- with mixed reality applications, the image quality the camera used for ing others may be necessary. jects near the camera. goal is to create an illusion in which capturing the reality produces. This • Especially when a 360 video is • Positioning the camera is per- virtual elements are looking as real means that virtual elements drawn viewed with a HMD, camera haps the most important de- as the physical objects in the envi- by the application should feature movements can be very chal- cision when 360 videos are ronment observed by the viewer. identical lens distortions, dynamic lenging. Camera movement, es- captured. In many cases, the Realism of mixed reality content range, noise, etc. In practice, what pecially if it is not predictable, camera is placed in a natural improves the user’s immersion is needed in the mixed reality im- can cause cyber sickness in users. viewing position. The location and engagement as virtual content age output is the same kind of visual It is possible to stabilize the video, is such where a person could be blends together with the reality processing and image composition in particular camera rotations standing (or sitting) in real life. seamlessly. However, photorealistic that is used in visual effects pro- can be removed in post-process- It is also possible to position the rendering of virtual objects is a dif- duction used in movies. Compared ing. However, shooting videos camera into an unnatural loca- ficult task, and even with the state with the visual effects production with a moving camera should tion, e.g., floating high in the air. of the art solutions available today, methods used for film production, always be considered very care- This can be considered one of such seamless blending is difficult the challenge for the mixed reality

42 3. PRODUCTION GUIDELINES comes from the fact that in the film blurring. In SelfieWall application, through displays is that in the out- appearing always semi-transparent, productions, for producing a single parameters used for video image put of the image on top of the view degree of transparency depending frame, tens of hours of processing post-processing can be tuned in- of the real world, light can only be on the amount of light coming from time and manual tweaking is of- dividually for each historic photo- added not removed from the light the real world environment. In ad- ten used, but for the mixed reality graph in use, so in the same session, that is already transporting from dition to this transparency short- application same processing needs viewer captured on video can be the real world to the retina of the coming of the optical see-through to be fully automatic and be per- blended with black and white pho- viewer. This causes virtual elements setups, also displays available for formed in real-time, i.e. above 20 tographs with various quality and times per second. color images with different color In the SelfieWall pilot imple- shifts. mented for the Maritime Center In video see-through augment- Vellamo, requirements for the ed reality applications virtual ele- visual processing are slightly re- ments are composited together with laxed, as the operational environ- the view captured by a camera. In ment is more controlled than for comparison, in optical see-through example with a mobile phone aug- augmented reality in which virtual mented reality application that can objects are added directly on top of be used at any environment. In the viewer’s natural sight it is more SelfieWall pilot, the goal is to blend difficult to reach a seamless blend- Image 6: Example of impact various steps of the SelfieWall image post-processing has on image of the viewer as seamlessly as ing of virtual objects with the view the image. possible with historic photographs. to the real world. In order to do so, SelfieWall appli- Optical see-through augmented cation implements a real-time im- reality displays typically used by age post-processing pipeline used the emerging head mounted display for creating similar look for the products such as Microsoft Holo- captured video image as is seen on lens, use some form of beam com- the historic photograph. Post-pro- biners to mix image output with a cessing involves several steps that light emitting display element with include image color space trans- the view the user has to the real formations, adding of noise to sim- world. Beam combiners can be for ulate film grain and image space example half-silvered mirrors that transformations to mimic artefacts cause two views to blend togeth- Image 7: Video image of a viewer processed and composited in real-time with the Self- caused by the camera lens such as er. Problem with the optical see- ieWall application.

43 3. PRODUCTION GUIDELINES these setups are not capable of pro- change of prevailing lighting condi- cially when there is speech in the headphones must be available, and ducing dynamic range of luminos- tions made the problem of correct application, the user should be people don’t necessarily carry them ity or tone range nor resolution to mixing of images quite difficult. At able to hear it clearly. Other natu- all the time. Offering headphones match the sensitivity of the human outdoor environments the range of ral sounds — made by animals, ma- for loan is a possible solution, but visual system. light energy reflecting from physi- chines, moving objects and so on, hygienic issues must be taken into In the Amos Anderson pilot cal objects has very big variation. are important for the experience. account. Cheap headphones for sale which uses a large scale optical With Amos Anderson pilot, this There are different options for how may be a working solution in some see-through display, shortcomings need to dynamically adjusting the to reproduce the sounds, and it de- cases. of the display type have been ad- amount of light output with the pends on many factors which is the Designing the sounds to be dressed by accepting the inherent display element was addressed by most appropriate option. subtle may help in avoiding dis- transparency of virtual elements developing a camera based solu- Using the speaker in the mo- turbances, even when using speak- by not trying to create appearance tion for observing the amount of bile device is an obvious possibil- er-generated audio. This approach of virtual elements but rather us- light reflecting from the physical ity, with the advantages that it is obviously affects the whole concept ing the display to enhance existing object on top of which the virtual always available when the device of the application and may exclude physical object by adding details image was to be added and then itself is, and that more than one some otherwise good ideas for a to it with a virtual content. This controlling the brightness of the person can hear the sounds at the mixed reality experience. approach requires delicate balance display accordingly. same time. In several pilot applica- Making the audio experience as between the mixing of normal view tions in the MIRACLE project the realistic as possible is a technical to the physical object and amount SOUND DESIGN speaker-produced sound was used issue. The speakers of phones and of virtual content blended together with good results. There are, how- tablets can produce sounds with with it. As the pilot was deployed Sound can be an essential part of ever, also potential risks with this more or less limited quality and at outdoor environment, constant a mixed reality experience. Espe- approach: sounds spreading free- volume, and the directions of the ly can be a disturbance in places sounds can’t be controlled much. like museums, churches and so on, Headsets of head-mounted displays, and having many copies of the app on the other hand, may be able to running simultaneously may cause reproduce quite realistic sound en- a cacophony that disturbs people vironment with authentic sound even in less delicate places. volume, frequency response and di- Using headphones is one way to rectional characteristics. avoid disturbance, as it keeps the Even with hand-held mobile sounds in the user’s ears. This is devices it is possible to produce Image 8: Test object without overlaid virtual image on left and object with virtual image also a good solution in noisy envi- some basic sound effects. For ex- added on the right ronments. However, it means the ample, when the virtual charac-

44 3. PRODUCTION GUIDELINES ter turns away when speaking, the REFERENCES voice should become muffled, and when the character moves further, 1. Marvelous Designer: it should become quieter. To realize https://www.marvelousdesigner.com/ this, game engines have audio pro- 2. Agisoft PhotoScan: cessing functions built-in and sup- http://www.agisoft.com/ port third-party solutions. 3. Microsoft Kinect: https://developer.microsoft.com/en-us/ windows/kinect

4. Google Maps: http://maps.google.com

5. Ricoh Theta: https://theta360.com/uk/

45 3. PRODUCTION GUIDELINES 3.5 USER EXPERIENCE AND USABILITY

There are many definitions for the the user experience aspects the in- Some practical guidelines for user zontal orientation can switch on term user experience. These have stallation is trying to evoke may experience and usability design are the map view, but the user might different field-dependent empha- also be much more specific. An • Use interaction style and widgets miss that if it is not explained. ses, for example marketing, hu- obvious design goal for any cultur- that most users are familiar with • During the use of the app, tips man-technology interaction, or al heritage-oriented application, or already. Traditional buttons about possible ways to proceed psychology. Approaching the issue any museum installation for that and sliders on the touchscreen may be precious. If the user in a simple manner within the field matter, is that it should be easy to work well. The button functions seems stuck, tell what could be of human-technology interaction, use, and it should be obvious from should be made easy to under- done next or what to look for. as in [1], user experience can be the start how to use it. Since many stand. • Technical problems for instance thought as a user’s subjective opin- of these applications are mainly • It can be useful to offer more with tracking can spoil the user ion about a certain aspect about the used on location, the users are often than one way to achieve a task: experience badly. If this is at all system in a certain context at that first-time users. If the application for example, putting a virtu- likely to happen, the app should time. The primary users referred to requires any significant learning, al object somewhere on screen recognize the problem and give here are the museum visitors, al- many users may simply stop using could be possible by dragging the user assistance about how to though for instance museum per- it before getting any good experi- and dropping, or it could be work around it. An alternative sonnel maintaining an AR installa- ences from it. done first tapping the object and way to proceed could also be tion can be considered as users as The idea, story and content of an then the place where it is to be offered — for example, a 360-de- well. Thus, user experience can be installation obviously influence a put. gree video can be played in place thought simply as a museum visi- lot how the users experience it. See • For some users the concept of of the actual AR view when factor’s personal impression about an Section 3.3 Content creation for tips AR may be new and unfamiliar. ing serious tracking problems. interactive installation at a muse- and guidelines on the content side. For them there should be suffi- um. cient guidance about the ways of Depending on the purpose of an In museum context, good user usage when starting up the app. installation and its planned de- experience in general means that It may be done with a video tuto- ployment time, especially, it may an installation is attractive, the us- rial or some practice tasks before be highly useful to evaluate its user ers like to interact with it and they starting up the actual app. experience on some level at least. see it bringing some added value • A tutorial is needed for ways of The purpose, and thus the scale, of with regards to their museum visit. interaction that aren’t likely to be such an evaluation may vary greatly Depending on the type of museum familiar to everyone. For exam- based on by whom and for what it is and the purpose of the installation, ple, turning the device to hori- conducted. For example, if museum

46 3. PRODUCTION GUIDELINES visitors’ experiences on specific as- revealed with surprisingly few test stalled in automobile and road mu- er installation, we utilized a single pects, functionalities and solutions users. However, evaluating an in- seum Mobilia (Kangasala, Finland) Buttkicker device[2]. It was installed should be investigated for scientific stallation at least somehow in real provides a pure example of vibra- in the frame under the seat. The de- purposes, the evaluation methods context with real users is important. tion-based haptics adding natural vice was attached to the plywood need thorough and professional This is because the people involved and quite critical element into the frame piece under the seat. Rub- planning. Similarly, an evaluation in the design and development pro- experience. The installation aims to ber feet included in the Buttkicker targeting to investigate the business cess are easily too close to the mat- provide an experience of what it is package were placed between the potential of an application would ter, they may become blind towards like to drive an old road grader. It plywood and the rest of the frame, require business-oriented profes- the pros and especially cons. Tips does not aim for full realism; visi- giving the seat and the plywood sionalism when planning the eval- for user evaluations can be found in tors have only limited control over some freedom of movement. uation methods and content. How- the case study descriptions in Sec- what happens and the physical set- To create the vibrations in the ever, if the whole purpose of an tion 4. up consists only of simple, wood- road grader installation, the same application is to provide something en frame with somewhat realistic audio that is played back via a nor- new and nice to the museum au- ADDITIONAL MODALITIES steering wheel, seat and pedals. The mal audio speaker is also sent to the dience, also the evaluation can be haptic feedback included in the sys- Buttkicker. This audio was origi- kept light. In such a case one could Vibration and other haptic feedback tem is vibration feedback moving nally recorded from the cabin of use, for example, a simple question- Haptic feedback such as vibrations the seat. Since the old road grader is the road grader. There was no need naire to gather feedback from the and force feedback can improve the not a smooth ride, vibrations can be to create separate audio for the vi- users, or interview a few people. The user’s experience. In particular, an considered to be an important part bration purposes as the original re- inquired questions can basically be installation which simulates some of the experience. Installing the vi- cording had suitable, low frequency planned and decided using com- real world object or phenomenon bration element clearly improved content. mon sense without specific exper- can benefit from adding such el- users’ experiences about the simu- Challenge with vibration feed- tise: is the installation able to evoke ements. The vibrations and other lator (see Section 4.6 Case Mobilia back, especially with a strong one the target experience(s), do people haptic feedback are transmitted via I — the Road Grader Simulator for like in our installation, is two-fold. like the installation, why or why some physical element of an instal- details). First, the vibration tends to create not, what could be improved, and lation, e.g., a seat, hand controls or The implementation of haptic some audible sounds as well. These so on. In a light-weight user experi- even floor. Because of this, in most feedback can be complex when ad- sounds can become loud if the vi- ence evaluation the most important cases haptics are experienced by vanced force feedback hardware brating elements start to resonate thing is to find out truly useful in- only the visitors who actively use solutions are used. However, simple with the vibration. Second, the formation, i.e., receive results which an installation. People passing by vibration feedback can be included strong vibrations can cause enough have a practical meaning for possi- can experience them only in some quite easily. Many haptic actuators physical movement to cause harm ble next steps. Possible issues, and specific setups. of different sizes are controlled to the installation. Electrical con- also top features, can be sometimes The road grader installation in- with audio signal. In the road grad- nections can become loose due to

47 3. PRODUCTION GUIDELINES the constant vibration and even tion 4.8 Case Vapriikki — the Opti- REFERENCES screws may loosen, glue may break cal Telegraph for details about the etc. The physical design of the in- installation. 1. Keskinen, Tuuli (2015). Evaluating the stallation can alleviate this problem During the observation sessions, User Experience of Interactive Systems by limiting the vibrations only to it was evident that the switches in Challenging Circumstances. Ph.D. the components visitors are physi- were effective in attracting children. thesis. Dissertations in Interactive Tech- cally touching. Additionally, when being inter- nology 22, University of Tampere. viewed, some users expressed that http://urn.fi/URN:IS- Physical switches the switches were a nice element in BN:978-951-44-9972-2 Tangible interfaces have gained the installation. A couple of them 2. The Buttkicker: popularity as enhancers of gaming said it to be the best part of the in- http://www.thebuttkicker.com/ and other interactive experiences. stallation. One of the visitors said These tangible elements are also “The sounds of the switches” were utilized in museum environments their favorite part. where visitors are encouraged to in- However, there is a draw- teract and explore different exhibits. back with the utilization of toggle In the Vapriikki museum instal- switches in these kinds of installa- lation (Tampere, Finland) — the options. Because of their physical con- tical telegraph — physical switches straints, the switches affect the pos- are used to substitute the rope sys- sibility to provide the same entry tem used to control the shutters in point of interaction to all users. In the original version of the telegraph. other words, new users trying to in- The idea was that the switches would teract with the system will find the be a more robust solution while still switches in whichever positions the providing the fun element to the in- previous user left them, which can teraction with the telegraph. Nine cause confusion in problem-solv- different toggle switches (ON/OFF ing tasks such as the understanding states) are used to control the tele- of the coding principle in the telegraph shutters. In addition, three graph installation. buttons are used to control other aspects of the installation such as displaying information or changing the language settings. See Sec-

48 3. PRODUCTION GUIDELINES 3.6 OBSERVATIONS OF NARRATIVE AR APPLICATION USED IN REAL LEARNING SITUATIONS

To many teachers that do not have part of their high school courses or 45 minutes is good estimation of also induce valuable group learning earlier experience with AR, the con- elementary school program, found maximum length for especially benefits. Other bold option to solve cept of AR story can be hard to un- that this kind of narrative AR could younger students to focus on one overcrowding is to design applica- derstand at first. Prejudice towards serve as either as introduction to form of activity without breaks. If tions story in non-linear way, so this kind of vague tablet application subject matter or as deepening of setting up devices and groups takes that users can choose order of story can even be negative. Testing sim- already acquired knowledge. Re- 10 minutes, this gives story a maxi- progression. This design choice also ilar application or seeing it in use, ception and level of understanding mum length of approx. 30 minutes. increases level of user activity. quickly clarifies teachers vision of of the story’s meanings changes Application has to provide work- In general, AR technology in- how they could integrate this medi- depending on how much context is ing tutorial and helpful UI so users cludes an assumption of active um to their courses. Especially for familiar to user. One way to supply can start using it on their own and user who changes his orientation cases where teacher doesn’t have information that deepens under- quickly begin to progress in story. in augmented space. Following AR opportunity to test application with standing of story’s meaning is to Teachers and staff seldom have re- story is by nature an activating ex- their own hands, it’s recommended link application internally to formal sources to instruct whole class one perience. User has to actively navi- to provide basic information about information from different sources by one to use application. If a big gate in environment and follow the AR as learning medium, contents and have integrated UI for viewing group of students gets frustrated virtual character in scenes. This of the this specific story and vid- this in real time between and dur- outside of the classroom, situation user activity aspect of AR medium eo demonstration of application in ing the scenes. Educational AR sto- can quickly turn overwhelming to also contains a big risk. If finding action. One good rule of thumb for ry should include deeper insights in teacher who tries to control it. the scenes and following the events marketing AR content to teaching its subject matter, but it also should AR applications are highly vul- feels tedious, frustration quickly purposes for teachers is to empha- be understandable without high be- nerable to overcrowding and block- destroys learning motivation. Oth- size that this kind of applications forehand knowledge. Succeeding in ing of vision. This threat heightens er practical consideration related are not designed to replace conven- this makes application a versatile when story progresses in linear or- to active nature of AR is user ergo- tional teaching methods, but pro- and non-restrictive building block der and forces big groups of users to nomics with hand supported tablet vide one new possibility that can for teachers planning their course- same places at same time. Design- devices. If scenes are too lengthy or work well with old ones. work in unique situations. ing applications so that multiple require high viewing angles, hold- Teachers that brought their class- Class excursions to cultural sites users are able to use same device ing the device up can feel exhaust- es to explore the Turku cathedral usually have tight schedule and du- can ease the problem. Option to ing. with Sanan seppä application as ration of story cannot be too long. use application in small groups can Users following AR story might

49 3. PRODUCTION GUIDELINES have troubles to realize that they are Why narrative AR app can requires comprehensive and to be versatile tool for teacher free to orient themselves with high effectively supplement elementary multidisciplinary presentation and support unique teaching degree of freedom. Potential of ac- and high school education of setting and attention to de- needs and plans. tive participation in scenes is often • Combining narrative presenta- tails. If story wishes to utilize • Informative info package can not fully realized. Primary reason tion with analytical informa- AR’s potential, thoughtful de- help to market application to behind this is technical challenges tion is invigorating and effective, sign is required. teachers and break negative prej- that restrict the freedom of track- finding right balance between • Teacher must plan carefully how udices. ing in many situations. If tracking captivating drama and factual to integrate AR story experience • Positive activity and freedom drops, user is programmed with content is important. to course work so that different can quickly turn tedious work negative outcome to keep static • Answers to current curriculum modes of learning support each with bad design and technical pose. With unknown technology, plans requirements for learn- other effectively. Application problems. users are easily mimicking others’ ing environments that experi- should support it’s versatile use • Users can be encouraged to uti- successful actions and static pose ment with new ICT possibilities in different curriculum needs. lize freedom of orientation with that works is often replicated be- and utilize out of the classroom visual and audio clues and inte- tween users. learning possibilities. Practical considerations about grated modules like quizzes and Difficulties to use freedom of • Provides one genuinely new me- narrative AR application in treasure hunts. orientation can also be related to dium for informative storytell- education design choices of storytelling. User ing that fits well with ideas of • Designing balanced duration to can be guided to movement with phenomena based, multidiscipli- story and individual scenes is visual and audio clues and induce- nary and active learning. AR sto- important. ments. Users’ attention can also rytelling has lot of potential to • Overcrowding problems must be be encouraged with integrated ele- activate and motivate user, that acknowledged in design process ments in application like real time can be realized with creative use • Getting smooth start with ap- quizzes or treasure hunts mechan- of medium’s special nature. plication is emphasized with big ics. The more user is encouraged to classes. Good tutorial and help- pick details of his liking more he/ General challenges with narrative ful UI is important. she realizes the different ways of AR app in elementary and high • Story should be understandable following the story. school education and meaningful to users with • Engaging presence with his- different knowledge levels. Sup- torical environment demands porting formal information can smooth tracking and quality be provided inside the applica- visual assets. tion. • Story with pedagogical value • Application should be designed

50 4. CASES 4. CASES

This section describes pilot applications designed during the Futuristic History project (2013–14) and MIRACLE project (2015–17). The intention is to give insight into practical questions about application develowpment, potential problems that may be encountered, and results that can be achieved.

Pilots by University of Turku Pilots by University of Tampere Pilots by VTT Pilots by University of Helsinki • Luostarinmäki Adventure • Road Grader Simulator • Miracle Wall • How to Fly • Sanan seppä / Wordsmith • 360° Rally Simulator • Amos Anderson • Molecule Movements • Museum Explorer • Optical Telegraph • Radio Waves • Häme Castle • House Casagrande

51 4. CASES 4.1 LUOSTARINMÄKI ADVENTURE

Luostarinmäki Adventure is a tab- museum personnel and the project main intact was the Luostarinmäki manity and its environment (ICOM let application that takes a visitor researchers, and developed further area on the outskirts of the city. In Statutes). At its best, a museum is of the Luostarinmäki Handicrafts in the MIRACLE project. a new city plan drawn after the fire, also an experiential and creative Museum in the middle of the 1850’s the Luostarinmäki area was replace where learners can investigate daily life in Turku. The user of the THE LUOSTARINMÄKI arranged and thus the old houses and experiment firsthand. Seen app sees digitally created characters HANDICRAFTS MUSEUM were ordered to be demolished[1]. from this context, it is interesting on screen, in the actual environ- However, rebuilding of the city took to consider how an augmented re- ment, and can have a dialog with In 1827 the city of Turku in south- time and the demolition of the old ality game serves the learning en- them by choosing phrases on the western Finland was mostly de- houses in Luostarinmäki was nev- vironment of the Luostarinmäki device. The adventure was origi- stroyed in the most devastating fire er carried out, and the houses pre- Handicrafts Museum. What kinds nally built in the Futuristic Histo- in the Nordic countries through served their 18th century features. of potential influences could gami- ry project in co-operation with the history. The only larger part to re- In the first half of the 20th centu- fication and augmented reality have ry the cultural-historical value of on the museum experience and the area was recognized and the learning at the site? buildings were conserved. The Lu- A place like the Luostarinmäki ostarinmäki Handicrafts museum Museum can be an unfamiliar and was opened in 1940 to represent challenging environment to pres- traditional handicrafts and life of ent-day people, many of whom have the people of limited means in 18th never lived without running water, and 19th century city[2]. electricity and other necessities of modern housing. Few visitors know GAME-BASED MUSEUM how or for what purpose the tools EDUCATION they see were once used. Augment- ed reality and gamification can According the definition set out by help visitors to process the histori- The International Council of Mu- cal details and connect them with seums (ICOM), the duties of a mu- things learned earlier. Gamification, seum include promoting education, the use of video game elements in Image 1. Looking through the tablet application, museum visitor meets digitally created providing enjoyment and commu- non-gaming systems to improve characters in the streets of the Luostarinmäki museum. nicating information about hu- user experience and user engage-

52 4. CASES

they considered worth representing the day. but which have been impossible to During the first task the player bring forward until now. Already will among other things learn that the first version of the manuscript there were no street addresses in was based on these wishes and writ- the 19th century city, that nearly ten by history and museology stu- every house had some domestic an- dents in co-operation with the mu- imals, that master and mistress had seum staff and history researchers the right to discipline their servants, at the University of Turku. The aim that water had to be brought from a was to integrate interesting facts distance and that there were saunas within the events and the plot, in- inside some town houses. stead of presenting them tradition- The Luostarinmäki adventure ally in textual form. turns a tablet computer into a win- The events in the game take dow to history. Yet, there lies a risk Image 2. Attempting to deliver the wedding crown. place on one Saturday in the sum- in being too realistic. Representa- mer of 1855. The player takes the tions of the past are always only in- ment[3]. An engaging experience physical and digital environment role of Frans Hakala, a 23-year-old terpretations made by the museum has long been recognized as of great gives learners new opportunities to man from the countryside, coming staff, the historians, the artists and significance[4]. Games can help see, hear, feel and experience things to Luostarinmäki to take part in the engineers. The more complete learners find the motivation need- that would not be possible without the wedding of his cousin. Because the given interpretation, the less ed for all learning. A successful augmented reality. the character comes from country- the users need to make their own[6]. game engages the player, becomes side, he, just like the player, is unfa- At worst this may lead to an over- personalised with experience, mo- THE STORY miliar with many things in a 19th simplified and one-dimensional tivates, and creates strong memory century city, and people he meets picture of the past. Visitors have to traces[5]. The Luostarinmäki Adventure rep- must explain them to him. The be encouraged to be critical, and it Although the Luostarinmäki resents daily life in 19th century first task is to deliver the wedding should be made clear that the rep- game is fictional and tied to a sto- city through an entertaining ex- crown of the family to the bride’s resentation is not the absolute truth ryline, learners are provided with a perience. The aim was to create a mother. It is soon revealed that the about the history. In the case of lot of information about history, ob- mobile adventure game for young wedding ring has been stolen from the Luostarinmäki adventure this jects and life in the olden days. The adults, which would be entertaining the groom. The game then takes a means explaining the player the game motivates learners to investi- enough to keep the player engaged form of a detective story in which all but self-evident relationship be- gate and look for more information in playing. The museum staff was the player has to follow clues and tween fact and fiction in the story. about the site. The combination of asked to list things in the 1850s life find the thief and the ring to save

53 4. CASES

Image 3. articy:draft tool was used to create the script Image 4. In Unity Multipurpose Avatar numerous ad- Image 5. Dialogue interface. While discussing with for the story, including discussions and user choices. justments were available to create different appearanc- game characters, the user chooses one of the alterna- es for the virtual characters. tive lines as his/her reply.

Image 6. Map view. Holding the tablet horizontally, user Image 7. The inventory can be opened by tapping the sees the map of the area with indications of the current backpack icon in the bottom right corner of the screen. location and the place that should be found next. The The items carried are now seen on the bottom area of small photo gives a further hint of the place to look for. the screen

54 4. CASES

THE TOOLS AND DEVELOPMENT TOOLS character and cloth base models. When the user gets involved in PROCESSES For animating the characters we a task, there is typically some inter- An application targeted to wide au- used data from the CMU Motion action with virtual objects. Those The goal with the Luostarinmäki diences should work in a variety of Capture Database[11]. No database things are carried in a virtual application was to allow the user mobile devices, thus the concept of of clothes of the era could be found, “backpack”, and they can be given to experience the augmented world multi-platform development was so they were modelled within the to the virtual characters. while moving freely around the adopted from the start. The Unity project using Blender software. museum. There are several AR solu- game engine[8] was chosen as the USER TESTING tions readily available, but many do main software development tool. USER INTERFACE not fulfill our initial requirements. Unity provides much of the initial After the initial software develop- Many solutions are based on high- architecture needed in complex The goal of the project was to devel- ment phase, some user tests were ly visible graphic prints (markers, 3D games and supports common op a gamified adventure, inheriting carried out in parallel with User triggers or targets) placed in the desktop and mobile platforms. The elements from the adventure games Interface and application develop- environment to aid the tracking initial script from student writers genre but still “not obviously being ment. Someone from the develop- process. Such markers would not was refined with a scripting tool a game”. Many of the users of such ment team was following the test be acceptable in the museum en- called articy:draft[9]. With it the an application are first time users, user and helping with technical vironment, so a solution without storylines can be visualized and the so it must be easy to adopt, yet effi- or usage problems. Opinions of added visual aids was needed. The non-technical script writing pro- cient, error-proof and satisfying to the test users were collected with markerless point cloud tracking of cess integrated into the actual im- use. The idea in visual design was a simple questionnaire about the the ALVAR library[7] by VTT Tech- plementation of the application. to skip traditional game elements content, usability and user experi- nical Research Centre of Finland Creating realistic human char- such as crosshairs, health bars and ence of the application. A majority was used in the Luostarinmäki ap- acters is very time-consuming. A minimaps in favor of a clutter-free of the users found the experience plication. From a large amount of single lifelike 3D model of a person, interface of the main adventure of augmented reality pleasant or photographs of the environment a with realistic clothing, fluid mo- view. In the dialogue interface the rather pleasant and thought that virtual representation of the area tions, and natural voice can require virtual characters’ lines of speech the application provided added val- is calculated algorithmically. This weeks of work by several skilled are drawn visible over their heads, ue to the museum tour. However, virtual version, the so-called point professionals. Limited-budget work and the user can reply by tapping since the game and the question- cloud, is then used as the tracking requires more efficient methods. In on one of the alternative sentences. naire were both modified through target, eliminating the need for any this case we used the Unity plugin Supporting elements like task the testing period, and since most additional markers in the environ- Unity Multipurpose Avatar (UMA, hints and a map were placed into test persons were in some way con- ment. currently in version 2)[10] that pro- a separate view that appears when nected to either the museum or vided us with all the necessary basic the tablet is leveled down into hori- the research unit, the results can functionality to reuse and mix both zontal orientation. only be considered indicative. The

55 4. CASES testing primarily served for devel- well as usability data and person- uct that is among the first of its kind terfering with the physical reality opment of the user interface and al experiences, was gathered in a the reception was very favorable. in museums like Luostarinmäki, technical solutions. A clear source structured questionnaire built in Questions about app pricing and where outside areas and interiors of frustration for the users was los- the application from a total of 129 delivery — app downloaded to us- have been kept as authentic as pos- ing the tracking; they both blamed persons that were randomly re- er’s own device vs. device offered sible. It also allows for a much more the software and questioned their cruited museum visitors. An analy- for loan in the site — were also pre- diverse provision of multimedia own computer skills — usually si- sis of these tests is published; see[12]. sented to users. The opinions varied learning content than physical solu- multaneously. One important issue and the data does not provide defi- tions could offer. AR technology noticed during the user testing was nite answers to commercialization, can open up things that are difficult some kind of loss of the tradition- RESULTS AND and it must be kept in mind that the to understand or topics that are ab- al museum experience while using CONCLUSIONS app tested was a prototype. stract, making them more concrete the adventure. Some users felt they An important issue found dur- and easy to understand than tradi- The AR-experience in the Luos- were too immersed in the applica- ing the project is the notion that tional educational methods. tarinmäki was found to be very tion and forgot to pay attention to users tend to forget the museum pleasing and, combined with the the surrounding museum. Accord- around them while concentrating perceived value added, it clearly ing to preliminary assumptions, the in the game. Our view is that the indicates that there is a demand tablet was thought quite heavy and tablet should not be the be-all-and- in enhancing the museum experi- thus the users’ hands got tired after end-all solution, but instead more ence with augmented reality con- some time. Moving from one place about enhancing the experience. It tent. The testees were quite pleased to another was a welcome relief to should only be used occasionally, with the idea of using a tablet for some of the users. One practical is- and otherwise the users should be an augmented reality adventure in sue was that the scenes occurring in let to rest their hands, move safely, a museum-style heritage site even the courtyards could interfere with and enjoy the new, spiced-up mu- though the site itself was not as easy other visitors. Therefore, scenes seum experience. We learned that terrain and thus not as accessible for should be arranged at the sides and the application should guide the everyone as one could hope for, and corners of the courtyards. user better in the game. Technical there were challenges in some tech- Another test period was carried issues have affected user experience nical functionalities. The storyline out after developing an improved significantly, which is not surpris- was found good and the length of version of the adventure. This phase ing considering the application was the adventure (45 min) was found was done as part of the MIRACLE still on prototype level, not a pol- proper. The amount of historical project, and the study was conduct- ished commercial product. elements to the story were found a ed June 1st to June 18th 2015. The Using AR technology, informa- tad wanting but overall for a prod- data, including basic user data as tion can be provided without in-

56 4. CASES

REFERENCES 1. Viitaharju, J. (1990). Stadens bakgård. 6. Marcus, A.S., Stoddard, J.D., Woodward In T. Bergroth and M. Söderström (Ed.) W.W. (2012) Teaching History with Mu- ”Först kom skomakaren”. Hantverksmu- seums. Strategies for K -12 Social Stud- seet på Klosterbacken 50 år. Åbo: Åbo ies. New York and London: Routledge. landskapsmuseum. 7. ALVAR tracking library: 2. Sjöberg-Pietarinen, S. ”Fröken gör ju http://virtual.vtt.fi/virtual/proj2/multi- snart hela stan till museum” In T. Ber- media/alvar/index.html groth and M. Söderström (Ed.) ”Först 8. Unity game engine: kom skomakaren”. Hantverksmuseet https://unity3d.com/ på Klosterbacken 50 år. Åbo: Åbo landskapsmuseum. 9. Nevigo: http://www.nevigo.com/en/articydraft/ 3. Deterding, S., Sicart, M., Nacke, L., overview/ O’Hara, K., Dixon, D. (2011) Gamifica- tion. using game-design elements in 10. Unity Multipurpose Avatar 2: non-gaming contexts. CHI ‘11 Extend- https://www.assetstore.unity3d.com/ ed Abstracts on Human Factors in en/#!/content/35611 Computing Systems. 11. CMU Graphics Lab Motion Capture 4. Hooper-Greenhill, E. (2007) Museums Database: and Education. Purpose, Pedagogy, http://mocap.cs.cmu.edu/ Performance. New York and London: 12. K. Seppälä, O. Heimo, T.Korkalainen, J. Routledge. Pääkylä, J. Latvala, S. Helle, L. Härkä- 5. Kapp, K., Blair, L., Mesch, R. (2012). The nen, S. Jokela, L. Järvenpää, F. Saukko, Gamification of Learning and In- L. Viinikkala, T. Mäkilä and T. Lehtonen struction: Game-based Methods and (2016). Examining User Experience Strategies for Training and Education. in an Augmented Reality adventure San Francisco: John Wiley & Sons. Game. Case: Luostarinmäki Handi- crafts Museum. In 12th IFIP TC9 Human Choice and Computers Conference, September 7–9, 2016.

57 4. CASES 4.2 SANAN SEPPÄ / WORDSMITH The Sanan seppä (Wordsmith) aug- visible on tablet computers. The THE STORY told by using objects and historical mented reality application is locat- goal for this app has been to create characters digitally added in the ed in the Turku Cathedral and tells an experience that is both educat- The Sanan seppä will take the vis- church and made visible by tablet about the changes in the local life ing and entertaining. The story is itors in the Cathedral to the year computers. The characters include caused by the Protestant Reforma- told through eight short scenes in 1514, that is, to the situation before both some prominent figures of the tion in the 16th century. The appli- which Hemminki, a son of a black- the reformation, and then it will Finnish reformation and common cation is connected with the mark- smith, becomes an assistant of Mi- represent them the changes during people. The goal has been to create ing of the 500th anniversary of the kael Agricola and the progenitor of the next 150 years from a viewpoint an experience that is both educat- reformation in 2017, and it is cre- a priest family. Agricola was one of of a fictional family. The story writing and entertaining. The users of ated during the MIRACLE project, the most important characters of ten by author Tytti Issakainen is the application follow through eight in cooperation with the Turku and the Finnish reformation, renowned Kaarina Parish Union and Evangel- for his printed works that started ical Lutheran Church of Finland. the spread of literacy among Finn- An objective for creating the app ish common people. was to study presenting intangible The application runs on a tablet cultural heritage[1] via augmented computer, and follows many of the reality technology, in easily com- design principles used in the earlier prehensible form and in authentic Luostarinmäki Adventure app. The locations. user sees the scenes on the tablet The Sanan seppä Mixed Reality screen through the device camera application will take the visitors view where the virtual characters in the Cathedral to the year 1514, are added. Besides the actual scenes, that is, to the situation before the the app provides information about reformation. It then represents the the characters and events in the changes during the following dec- story as well as references to back- ades from a viewpoint of a fictional ground information in the internet. family. The story, written by author However, in this case there is no in- Tytti Issakainen, is told by using ob- teraction between the user and the jects and historical characters digi- virtual characters — the user is just Image 1. The Sanan seppä app runs on a tablet computer and displays fictional scenes tally added in the church and made an observer of the scenes in the app. from 16th century, when Lutheran reformation changed the lives of Finnish citizens.

58 4. CASES short scenes the story of Hemmin- choose the scene that occurs at or ki, a son of a fictional blacksmith as near that location, and search for he becomes the assistant of Mikael the corresponding point clouds in Agricola, one of the most important order to do the exact positioning. characters of the Finnish reforma- The app displays a guiding image of tion, and progenitor of a renowned the location of the scene to help the priest family. user finding it. For handling the virtual ele- DEVELOPMENT OF THE ments in the app, a software frame- APP work designed within the MIRA- CLE project, described in 4.3, is The augmented reality solution used. It includes database connec- for the app is based on markerless tions, allowing digital content — for tracking technology. In a histori- example 3D models, images or cal environment such as the Turku text — stored in various data sourc- Image 2. Marvelous Designer is an application for designing clothing for 3D games. Cathedral it is essential to avoid es to be retrieved and included in adding any visible markers to aid mobile applications. Virtual world the augmented reality tracking. positions of the objects can be In this app, the coarse position- managed via web interface, which ing solution is based on Bluetooth also allows a selection of a tracking beacons that are discreetly placed technology to be used with particu- in non-museal objects, like lamps lar virtual objects. The framework and speakers. Exact positioning introduces well-defined interfaces and tracking is based on visual using eg. ARML (Augmented Re- tracking of point clouds made of ality Markup Language) standard. permanent structures in the cathe- A goal in the MIRACLE project is dral. Unity game engine[2] acts as to design this framework as a tool the software platform, and ALVAR for museum personnel for updating library by VTT[3] is used for the and adding content in the AR ap- visual tracking. Mobile device first plications used in museums — also calculates the approximate location for persons not highly skilled in in- of the user based on the beacon sig- formation technology and software nals, and the application will then development. Image 3. Some characters with clothing.

59 4. CASES

The app also uses a reference ap- pert on clothing was able to do the Narrative applications with fic- REFERENCES paratus, which allows the user to digital design herself. tional elements can be informative find more information about the Animation of the human char- and an excellent way to convey in- 1. L. Viinikkala, L. Yli-Seppälä, O. I. Heimo, characters and events in the story. acters was done using motion cap- formation about complex historical S. Helle, L. Härkänen, S. Jokela, L. Part of the information is included ture technology. Amateur actors re- phenomena. However, it is crucial Järvenpää, T. Korkalainen, J. Latvala, J. within the app, while more extend- hearsed all the scenes, and they were to inform the users of the applica- Pääkylä, K. Seppälä, T. Mäkilä and T. ed descriptions can be found via then recorded with motion capture tion about the extent and nature of Lehtonen Reforming the Reformation links to web pages, where the con- equipment in a special studio dur- uncertainties or fictional elements Representation. Mixed Reality narra- tent is structured according to the ing one day. The collected motion within the application. This is also tives in communicating tangible and app. capture data required a significant recommended in the “London intangible heritage. DIHA & NODEM The virtual characters in the app amount of manual work combining Charter for the computer-based Special Session / VSMM 2016, Kuala are modelled by a graphical design- and correcting the captured move- visualisation of cultural heritage”[8]. Lumpur, October 2016. er using several tools, including ments before taking them into use. 2. Unity game engine: Blender software[4], Substance de- More about motion capture can be https://unity3d.com/ signer[5] and Substance painter[6]. found in section 3.4. The number of characters appear- The app can be downloaded 3. ALVAR tracking library: ing in the scenes is quite high, but free of charge to user’s own An- http://virtual.vtt.fi/virtual/proj2/multi- the amount of work to make those droid or iOS tablet at Google Play media/alvar/index.html was reduced by making parametric or AppStore, or a tablet can be bor- 4. Blender: https://www.blender.org/ variations from a small number of rowed from the cathedral entrance. basic 3D characters. 5. Substance Designer — Material Author- Clothing for the virtual charac- LESSONS LEARNED ing Tool: https://www.allegorithmic. ters was made by a historian, using com/products/substance-designer a clothing design software called Making an application as extensive 6. Substance Painter — 3D Painting Soft- Marvelous Designer[7]. It took a few as Sanan seppä requires lot of work ware: https://www.allegorithmic.com/ days to get onto productive level by professionals with different kind products/substance-painter with the software, which we consid- of skills, but the result can be worth er to be a good result. The intention the effort. However, if the historical 7. Marvelous Designer: was to avoid iterative phases and content and educational goals don’t https://www.marvelousdesigner.com/ extra work needed in case a graph- necessarily require such a compli- 8. London Charter for the comput- ical designer had done the clothing cated mode of representation, it er-based visualisation of cultural herit- based on descriptions and feedback is reasonable to consider another age: http://www.londoncharter.org/ by the historian. In this case the ex- means of representation.

60 4. CASES 4.3 MUSEUM EXPLORER

Museum Explorer is a set of soft- ple that are experts in the subject • Viewpoints: Define geoloca- tion and objects. Once the data is ware components that together matter of the museum but have no tion or bluetooth beacon trigger fed in or a new application is de- form a platform for creating and technology expertise — to modify based areas where different sets fined, it is available via a Museum displaying augmented reality con- and maintain the digital content of content are available. Navigator app. A prototype CMS tent in museum environments, and and create experiences for the mu- • Overlays: Additional maps to be interface for museum personnel is also outside actual museums. The seum’s visitors. used in the editor and the target shown in image 1. main elements of the platform are The system models required data applications for providing cus- a web content management system in a similar fashion to the ARML tom experience in addition to PRESENTATION for creation and maintenance of 2.0 standard[2]. possible 3rd party map provides PLATFORM digital data targeted to augmented (such as Google, Bing, Apple). reality applications for cultural her- The main object categories are • Annotations: Pieces of textual The ME Unity SDK is used to make itage sector, and a software devel- • Applications: To allow multiple information attached to features. end user applications presenting opment kit for building augmented productions to run on the same the virtual content and data cre- reality applications in the Unity en- platform/installation, all other The system provides a 3D editor for ated via the CMP part of ME. The vironment. content and user management is use in web browsers to create and SDK is built for and tested on the filtered based on the parent ap- edit the Features. The editor can be iOS & Android platforms, but most CONTENT MANAGEMENT plication. used to place Assets into a 3D space functionality should be compat- SYSTEM • Assets: Pieces of visual content and combine them with Targets ible with platforms supported by to be displayed to the user: Im- and Annotations. A simplified in- Unity and the connected 3rd party The MuseumExplorer Content ages, 3D models or targets. Tar- terface is provided for more casual tracker. The components and their Management System (CMS) is a gets provide the AR trackers the and non-technical users, such as functionality follows the scheme MeteorJS[1] based system that pro- registration data such as target the museum staff, allowing limited and roles defined in the Museum- vides technical developers, content images, point clouds or 3D spa- functionality, but having enough Explorer CMS. The SDK can con- experts and eventually even casual tial maps. functions for most common us- nect to the device sensors (camera, users interfaces to access and modi- • Features: Describe how the end age scenarios. The idea is that with gyroscopes etc.) and it has tools fy the recorded data. Dedicated user users access and consume the minimal training museum staff can for creating interactive experiences, interfaces are targeted for each tar- AR content. Combine assets (es- extend and update information in such as following a storyline and get group. A main objective for the pecially ties the targets with oth- databases and create new “applica- providing synchronized animation, system is to provide a simple inter- er assets) into larger contexts for tions”, or tours and exhibitions that audio and subtitles. A plugin for face for museum personnel — peo- example as a 3D scene. are based on the existing informa- the ALVAR tracking library is pro-

61 4. CASES vided, too. was piloted in the Finland 100 cit- ical objects virtually into the space the actual objects. This was a very Using this platform it is possible izen space in Turku, and the muse- as an exhibition. The objects were preliminary trial of the concept, but to implement e.g. a Museum Nav- um personnel interface was tried in selected from the archives of the the feedback indicated that there is igator app, intended for end users, the National Museum in Helsinki. Museum Centre Testers were more interest towards augmented reality for displaying the virtual content or less randomly selected passers-by, applications among museum per- from the CMS. DIY EXHIBITION although a press release about test- sonnel. The trial raised discussion ing was distributed and there was about topics like how much em- PILOTS BASED ON The Museum Centre of Turku was a poster about the app in the space ployee resources would such sys- MUSEUMEXPLORER hosting a “citizen space” in a shop- and some other locations in Turku. tems require and what kind of tasks ping centre in Turku in the winter Feedback was collected with a ques- can be expected to be taken care by MuseumExplorer elements were and spring of 2017. The Make Your tionnaire that the users could fill in non-technical museum staff. No used in some pilot implementations Own Exhibition app was given to after trying the app. definite answers to these questions in the MIRACLE project. Especial- visitors there for testing. The app exist at this point. ly two cases were made to test-drive ran on a tablet computer, and the MUSEUM PERSONNEL the platform: the DIY Exhibition user could set a number of histor- INTERFACE The web interface of the Content Management System was demon- strated to volunteers of the Nation- al Museum’s personnel in Helsinki- in a half-day session. There were a handful of sample museum objects (on display in the actual exhibition) Image 1. Web interface can be used to de- acting as test material that could be sign museum tours and virtual exhibitions. used to create “applications”. The web interface allowed selecting ob- REFERENCES jects, positioning them into a virtual space, and adding information 1. Meteor JS: such as textual descriptions into the https://www.meteor.com/ database. The participants could 2. ARML 2.0: edit the augmented reality data via http://www.opengeospatial.org/pro- the interface, and the results could jects/groups/arml2.0swg Image 2. Virtual exhibition experience. be observed via the navigator app at

62 4. CASES 4.4 HÄME CASTLE

Transformations of the Knights’ be chosen: the middle ages (1500) ing did weaken the experience of tailed than in this pilot, would also Hall is an augmented reality ap- when the hall was the reception many users. very likely improve the experience plication located in the Häme Cas- hall of the castle bailiff, and the A display freezing feature would for the users. tle, Hämeenlinna. It is designed as prison period (19th and early 20th lessen the problem of fatigue while a museum guide experience de- century). The application contains holding the tablet up. With it the scribing the changes in one of the also additional information about user can at any time set the view castle’s halls during past centuries. the hall’s history that the user can on hold, lower the tablet to a more Oldest parts of the castle date from browse. comfortable position and continue the late 13th century but the cas- The applications was in test use observing and browsing the con- tle has experienced many kinds of during February 2017 in the castle. tent. Presenting more reconstruc- building phases and uses through Visitors to the Knigths’ Hall were tions, and making them more de- centuries[1]. given a tablet computer with the The Knights’ Hall got its pres- app, and they could use it while in ent appearance, a single large open the hall. A questionnaire was of- space, after the demolition of the fered after that. Feedback gathered prison-period constructions, which from the questionnaire, and com- was done during the 1950s. When ments from the castle guides, indi- the castle served as a prison, there cate that many (roughly half of the) were several floors and prison cells visitors see augmented reality as a in the space. In the 16th century the potential approach for enriching hall had a vault that made it lower the visitor experience in museums than the current space. and historical sites. Most would The application visualizes the like to see more realistic representa- two periods with skeleton-like tions of the historical constructions drawings displayed in their actual than the simple drawings. Holding positions on top of the real-time the tablet was tiring according to image. The user can freely move in most users. Some felt that it would the hall and look at all directions not feel convenient to carry and use “through” the tablet, to see the aug- a device while visiting a museum. mentations. Two time periods can Technical problems with the track- Image 1. The current form of the Knights’ Hall. Remains of the vaults are visible in the walls.

63 4. CASES

REFERENCES 1. Häme Castle: http://www.kansallismuseo.fi/en/ hame-castle/history-of-the-castle

Image 3. Photo taken during the restora- Image 2. Vaults and walls of the hall dis- tion of the castle, prison-time constructions played as augmentations. being removed.

64 4. CASES 4.5 HOUSE CASAGRANDE The House Casagrande pilot is a computer. So the models were man- modelled and displayed, mask- LESSONS LEARNED tablet application that visualizes ually simplified, removing objects ing the current buildings. This the buildings situated earlier in the inside the buildings that would not would be about the same as pre- This is an example of a case where residential quarter now occupied be visible but would have affected senting the historical quarter as close cooperation between disci- by the House Casagrande in Turku. the app’s performance. Further- virtual reality. plines is needed. Architects, his- The buildings can be viewed in aug- more, textures were added to the • Diminished reality. Means vir- torians and archaeologists had to mented reality from the Linnan- models to make them photorealis- tually removing some real ob- decide on interpretations based on katu street in front of the House tic. jects from the AR scene. This can all the available historical informa- Casagrande. Several periods be- Another issue to be noted with be a complex process, depending tion. Ideally, the models could have tween years 1588 and 1790 can be this pilot is that current building on the size and surroundings of been made as visualization mod- chosen in the application. occupies mostly the same space as the removed object. In this case els, instead of architectural models 3D models of the historical the virtual models of the historical practically everything from the that required modifications. This buildings are based on archaeologi- buildings. It is not a trivial problem current environment should be would have required more com- cal and textual sources. The models to virtually remove existing struc- removed and replaced. munication between the modelling were created by an architect with tures from an augmented reality • Distorting and colorizing the architect and software team, or the guidance from archaeologists and view. If a virtual object is simply area. In this approach the image architect should have learned new historians. drawn in the correct location on structure of the current building skills to make visualization models. The architectural models could screen when the real environment is broken up and some smoke The process was iterative, with not be used directly in the app. This has something located in front of or mist-like effect is added to checks and modifications made could be expected, as architectural the virtual object, the virtual object partially hide the building. This during the project, which is normal. models in general contain informa- seems to be closer to the observ- option was used in the app. The In case a lot of buildings were to be tion about the internal structure of er and smaller than intended. The positive arguments for it are modelled, it would be efficient to the building and other details that physical objects should be masked that it is relatively simple to do, avoid the modifications phase, but can’t be seen in a visualization such or removed in some way. it matches the light and weath- in a single shot case the extra work as this application. Visualization er — because the building is seen done was probably smaller than models only need to contain data Some processing approaches for through camera — and there is training the architect to make visu- that can be seen, and even some diminished reality no manual work needed. alization-optimized models. minor details that would be visible • Clean plate compositing, which may be either simplified or removed means that most if not all of the to make the model run better in a surroundings would have been

65 4. CASES

Image 1: House Casagrande app distorts and fades the current house in order to highlight the historical, virtual buildings.

66 4. CASES 4.6 CASE MOBILIA I — THE ROAD GRADER SIMULATOR

The goal of the pilot was to provide capturing realistic material. A cam- good experience and the imple- also tilts so that in fast speed an embodied experience to muse- era setup capable of capturing full mentation was prone to fail. Other the camera looks more down- um visitors, and to study, how to spherical video was attached on the major changes made for the second wards, displaying more road support the feeling of immersion in roof of an actual grader, a micro- version were to expand video view near the vehicle, further af- a simulator environment by using phone was placed inside the cabin from one screen onto three screens, fecting the experienced speed. novel input and output technolo- and the grader was driven around and to provide a clearer visual re- • Provides some feeling of steering gies. Inspired by the theme of the small roads. sponse to accelerating and braking. by installation museum — automobile The current system setup of the · Camera heading adjustment and road museum Mobilia (Kan- simulator can be seen in image 1. It To create interactivity, the software matching steering wheel ro- gasala, Finland) — a vintage road consists of a seat, a steering wheel • Plays back in so-called small tation. grader was selected as the theme of and pedals, a computer, three pro- planet projection until a user sits · Virtual camera sideways posi- the simulator. The system has been jectors, an active audio speaker, and down and the video animates to tion varies to small degree per available for museum visitors since a set of sensors. The sensors are a view directly ahead. steering as well. June 2015, and subjective feedback 3D accelerometer and gyroscope • Communicates speed controlled from the users has been collected. for steering wheel (Yocto 3D), lin- with pedals in three ways: The steering feedback has not re- The simulator has been improved ear potentiometers for pedals (Yoc- · Video playback speed is full or sulted in a good enough feeling of based on the feedback, and the to Knob), and light sensor for the half speed. Only two speeds control, and we are currently build- main changes can be divided into a seat (Yocto Light) to detect wheth- are used to ensure reliable ing a solution, where more realistic first and second version of the sim- er someone is sitting on it. The seat video playback. driving simulation is implemented. ulator. also has a device called Buttkicker · Audio matches the speed. A The 360° video allows projection ef- to provide a strong vibration feed- selection of recorded sound SYSTEM DESCRIPTION back. clips with varying the play- The steering wheel movement is back speed is used both for The content of the installation is limited to roughly 65 degrees so a audio and vibration via the based on 4K resolution 360° vid- single sensor without moving parts Buttkicker. eo. While it does not allow users to is able to detect the steering. In the · Video image field of view var- drive around freely, 360° video was first version, the steering wheel ro- ies so that high speed has nar- chosen because it is more cost-ef- tation was unlimited. This was re- rower field of view than slow ficient to produce than 3D models, placed because the user feedback speed to emphasize the ap- and a real vehicle was available for indicated that it did not provide pearance of motion. The view Image 1. The road grader simulator setup.

67 4. CASES fects which can emulate actual driv- statements with a five-step rating respondents were very interested in hicle. ing within small area. This solution scale, which can be seen in image technology and history (medians Installing the Buttkicker during needs a video without a visible ve- 2. In addition, the overall liking of 5/5), and somewhat interested in the evaluation of the first version hicle. Thus, a 3D model will be used the system is inquired with a five- playing (median (4/5). of the road grader simulator was a instead. By limiting the steering to step smiley face scale ranging from good choice. This change improved the width of a narrow road, we hope extremely unhappy to extremely Results the experiences statistically signif- to create good experience with the happy. To support and explain the Considering the first version of the icantly (Mann-Whitney U, p<0.05) limitations of the video material. quantitative results, the question- road grader simulator, the over- considering the irritatingness and naire includes open-ended sen- all feedback (n=92–104) was rath- the funniness of driving with the EVALUATION tences inquiring the best and worst er positive. For the question “How road grader, the willingness to features of driving with the road much did you like the road grader drive again, and the overall liking In order to study museum visitors’ grader, as well as a possibility to simulator overall”, the respondents of the road grader simulator. experiences about the road grader provide other comments. The ques- rated their experiences as 4/5 as a Considering the second version simulator’s different versions, sub- tionnaire ends with a background median on the smiley face scale. of the road grader simulator, a sum- jective feedback from the users has information section asking the re- The respondents felt that the driv- mary of the statement results as re- been collected with a questionnaire. spondent’s age, gender, personal ing with the road grader was rath- ported by the respondents can be interest towards technology (e.g., er fun (median=4), and they total- seen in image 2. Subjective data collection cars), playing and history. ly agreed with the statement that Similarly as with the first version, Museum visitors’ user experienc- these kinds of simulators would the second version of the road grad- es about the road grader simula- Respondents increase their interest towards mu- er simulator was received rather tor have been collected with a pa- Overall, i.e., concerning the first seum visits (median=5). However, positively by the users. The median per-form questionnaire. Because version (from mid-June 2015 until the respondents did not really feel for the overall liking of the simula- a suitable questionnaire was not early-January 2016) and the second as if they were actually steering a tor was again 4/5 on the smiley face readily available, one was created version (from late-May until the road grader (median=2), and they scale. The experiences regarding to correspond with the context, and end of 2016), we received feedback were rather neutral considering the the immersiveness-related state- the objectives of the system, and from altogether 187 respondents statement “I felt like I was actually ments “I would like to drive again more broadly also the project. Thus, (84 boy/male, 58 girl/female, 4 other, aboard a road grader” (median=3). with the road grader”, “I would like in addition to more general user ex- 41 did not answer). The ages of the One obvious reason for the re- to tell my friend about the road perience aspects, the questionnaire respondents varied between 1.5–68 spondents not feeling to be aboard a grader”, “I felt like I was actually had to inquire about immersion years (mean=23.1, SD=17.8), the road grader might be that the cam- steering a road grader”, and “I felt and appeal of the system in the con- age group of 0–10-year-olds cover- era was situated on top of the roof I was actually aboard a road grad- text of a (car) museum. ing about 37% and 11–20-year-olds instead of the cabin when shooting er” as medians compared to the The questionnaire includes ten about 20% of the responses. The the 360° video with the actual ve- first version showed an improving

68 4. CASES trend. However, statistically significant (Mann-Whitney U, p<0.05) differences between the experiences about the first and the second version of the road grader simulator were found only considering the overall liking. Also, despite our ef- forts to improve the simulator, the lack of steering kept on being mentioned in the answers for the open questions. Thus, our work contin- ues by implementing a version with a more realistic driving simulation.

Image 2. Reported user experiences about the road grader simulator’s second version. Again, the whiskers indicate the minimum and maximum responses, the gray boxes indicate the interquartile range, and the black balls represent the median values (n=66–71).

69 4. CASES 4.7 CASE MOBILIA II — THE 360° RALLY SIMULATOR The goal of the pilot was to provide SYSTEM DESCRIPTION and separate headphones. See im- first deployment in June 2016. a pleasant, embodied experience age 1 for the physical setup in the to museum visitors by utilizing The 360° rally simulator is a system museum. Subjective data collection novel technologies. In line with built for Samsung Gear VR head- The application consists of sever- Similarly as with the road grader the theme of the installation mu- set and Samsung mobile phones. It al omnidirectional videos and au- simulator, museum visitors’ user seum — the rally museum within utilizes Oculus Mobile VR SDK for dio recorded inside a rally car dur- experiences about the rally sim- the automobile and road museum displaying the visual material on ing a winter practice stage by many ulator have been collected with a Mobilia (Kangasala, Finland) — a Gear VR. The simulator is built in- times Finnish rally champion Juha paper-form questionnaire. For con- simulator based on omnidirection- side an actual rally car which is part Salo and his co-driver Marko Salm- sistency and comparability reasons, al video and audio from real-world of the Mobilia Rally Museum col- inen. Some of the videos are interac- the questionnaire created for the rally driving was created. The sys- lection. Inside the car, the users are tive and some are just static videos road grader simulator was taken as tem has been available for museum able to sit either on the driver’s or which the user can experience. On the basis for rally simulator’s evalu- visitors since June 2016, and sub- the co-driver’s seat, and while sit- top of the videos is an audio overlay, ation as well. Because the objectives jective feedback from the users has ting on the driver’s seat, also keep which is important in enhancing for both simulators were the same, been collected. their hands on the steering wheel. the experience, because it allows only rather small modifications The simulator application is used the user to listen to the pacenotes and additions were required. by putting on the Gear VR headset read by the co-driver during the For the rally simulator ques- rally driving. This combination of tionnaire, in addition to naturally video, audio and actual physical car changing the wording to refer to environment is designed to provide the rally experience, an item in- a novel and exciting experience for quiring the respondent’s physical the user. seat within the car (driver’s seat or co-driver’s seat) and a statement EVALUATION inquiring whether the simulator caused the respondent nausea were In order to study museum visitors’ added. Also, the inquiry about the experiences about the rally sim- respondent’s personal interest in ulator, subjective feedback from history was replaced with interest Image 1. In the 360° rally simulator installation, the user is able to sit in a real car and expe- the users has been collected with a in rallying. The final 11 statements rience the rally stage through VR glasses and headphones. questionnaire since the simulator’s rated on a disagree–agree scale for

70 4. CASES the rally simulator can be seen in smiley face scale. The users have re- image 2. ported the rally experience to be extremely fun (median=5), and they Respondents would like to get aboard the rally Between mid-June 2016 and simulator again (median=5). As the mid-January 2017, we received feed- simulator did not attempt to mimic back from altogether 256 respond- the situation of one actually driving ents (134 boy/male, 84 girl/female, 2 a rally car, and the camera was sit- other, 36 did not answer). The ages uated between the driver’s and the of the respondents varied between co-driver’s seat, the respondents’ 4–70 years (mean=32.1, SD=18.1), experiences about the statement the age group of 36–50-year-olds “I felt like I was actually driving a covering about 32% of the respons- rally car” remained neutral (medi- es. A clear majority of 94% of those an=3). However, the respondents who reported their physical seat, sat somewhat agreed with the state- on the driver’s seat. The respond- ment that they felt like they were ents were very interested in tech- actually aboard a real, moving rally nique (median 5/5), and somewhat car (median=4). As with the road Image 2. Reported user experiences about the rally simulator. The whiskers indicate the interested in playing as well as rally grader simulator, the respondents minimum and maximum responses, the gray boxes indicate the interquartile range, and racing (median 4/5). strongly agreed that these kinds of the black balls represent the median values (n=227–241). simulators would increase their interest towards museum visits (me- inity, i.e., as if they would have been and thus, the environment outside RESULTS dian=5). An interesting finding is aboard a real car, the possibility to the car, and the road and its sur- also that the respondents totally sit in a real car, and the possibilities roundings, were not visible. Based A summary of the statement results disagree with the statement that provided by the omnidirectional on this feedback received mainly as reported by the respondents can the simulator would cause them video, i.e., the ability to look around. during 2016, new material record- be seen in image 2. nausea (median=1), although it is Also the speed factor, and the audio ed in summer scenery was made The overall feedback about the commonly known that 360-degree and the pacenotes, were mentioned available to the museum visitors in simulator has been very positive. videos experienced with virtual many times. The negative com- mid-January 2017. For the question “How much did glasses may cause nausea. ments about the rally experience you like the rally simulator overall”, Considering the open-ended concentrated on the issues raised by the respondents rated their expe- questions, the respondents admired the winter scenery throughout the riences as 4/5 as a median on the the experience in general, the genu- responses: the video was too bright,

71 4. CASES 4.8 CASE VAPRIIKKI — THE OPTICAL TELEGRAPH

The goal of the pilot was to study The installation is presented as switches and buttons utilized for presented on the screen where us- how novel input and output tech- an optical telegraph tower printed user interaction. The monitor shows ers can see the messages they have nologies can be used to enhance on a veneer plate. However, in prac- a virtual representation of the local sent and received, allowing them to museum goers’ experience, and tice, it simulates two towers, a local tower, which can be controlled by keep track of the conversation. how gamified content works in a tower where users can send messag- using multiple switches to change fixed augmented reality installa- es to a remote tower and a remote the states (ON or OFF) of each shut- TECHNICAL SOLUTION tion. We decided to create an aug- tower that communicates back with ter. In addition, the screen displays mented-reality, interactive version the user. Image 1 shows a schema of some additional information such The system comprises three main of the shutter telegraph invented the final installation and its differ- as a biography from the telegraph’s components: by Abraham Niklas Edelcrantz in ent components. inventor and the local tower’s code- The Core Module: A Node.js, the 1800s. The interactive replica The upper part of the veneer book. Moreover, a chat history is HTTP server that hosts the web ap- would then become part of the new plate is augmented using a projec- plication. This module handles the Rupriikki media exhibition inside tor and it represents the remote switch interaction from users and the Vapriikki museum (Tampere, tower section. The purpose of this the communication with the Chat Finland). tower is to display the incoming bot module. The intention behind the instal- communication from the remote The Chat bot Module: A Node.js lation was to demonstrate to users interlocutor. The communication is module that synchronizes the com- the operation of telegraphs through shown as moving shutters that turn munication between the chat bot a hands-on experience. In this to create different kinds of message service and the core module. way, museumgoers could obtain a codes, in the same manner as the The Chat bot Service: A Py- deeper understanding on how the original telegraph would. Addition- thon-based artificially intelligent communication worked with such ally, each shutter contains a numer- agent that generates language re- apparatuses and enhance their visit ical value — used for codification sponses according to the messages beyond the mere learning historical purposes — to help users to deduce received from the user. In addition, facts. However, when chatting with the logic behind the message codes the service invites museumgoers “von Edelcrantz”, the history of tel- they receive. to interact with the system by con- egraph as well as Edelcrantz’s biog- On the lower part of the veneer stantly greeting them while there is raphy and his background close to installation, there is a representa- no one using the exhibition. royal court in Stockholm revealed tion of the local tower section; it to the visitor with a playful manner. comprises a built-in monitor, some Image 1. Schema of the installation.

72 4. CASES

USER TESTING AND to fill an experience questionnaire • These kinds of applications and the code-solving difficulty FEEDBACK and answer the interview questions. would increase my interest to- (median=2). The experience questionnaire wards museum visits. Participants also stated that they The installation was evaluated comprised a series of statements re- • I would like to tell my friend would like to communicate with during observation sessions, ex- garding the interaction and experi- about the telegraph. the telegraph again (median=4) perience questionnaires and inter- ence using the telegraph as well as • I would like to share my coding and that they would recommend views. Three observation sessions some other background informa- experience to my friends in so- the installation to friends (medi- were held during the museum night tion concerning the users’ interests. cial media (e.g., Facebook, Twit- an=4). Furthermore, they consid- 2016 and on the museum’s birthday Each statement was assessed using ter or Instagram). ered that the application helped weekend in October 2016. In these a five-point Likert scale (1=Totally • I am interested in playing. them understand the telegraph’s sessions, users were approached af- disagree, 3=Neither agree nor disa- • I am interested in problem-solv- operation (median=4) and history ter using the system and prompted gree and 5=Totally agree). The over- ing tasks. (median=4). Additionally, visitors all impression of the system was • I am interested in technique. strongly agreed with the statement rated using a 5-step smiley scale. • I am interested in history. that installations such as the tele- • How much did you like the telegraph raise their interest in muse- The questionnaire statements were graph installation as a whole? um visits (median=5). • The first impression about the telegraph was interesting. The answers from a total of 54 re- OBSERVATION RESULTS • The idea of the telegraph was spondents were analyzed. Muse- clear. umgoers had a good impression of During a weekend observation, • The telegraph was irritating. the system, which can be evidenced museumgoers who passed and in- • Communicating with the tele- in a high general score of the interacted with the installation were graph was fun. stallation (median=4). Additionally, observed and tallied. A total of • I would like to communicate users expressed that the communi- 267 passersby were tallied amid with the telegraph again. cation was fun (median=4.5) and the observation weekend of which • Sending the messages was easy. indicated that the telegraph was 49% noticed the telegraph and 27% • Figuring out the coding princi- not irritating (median=2). More- started interacting with the tele- ple was difficult. over, users found the idea behind graph installation. • The application helped me to un- the telegraph clear (median=4) and derstand the principle of the op- their ratings for the level of difficul- Users’ information tical telegraph. ty ranged in the middle, as can be Besides the passersby information, • The application taught me about seen in the answers regarding mes- additional details were noted down Image 2. Final installation. the telegraph’s history. sage emission difficulty (median=3) about users who interacted with the

73 4. CASES installation. Such details include nice element of the installation. their estimated age, gender, and Switches can provide limita- the type of user. From a total of 178 tions in the system design. Because observed users, 44% were male and of their physical constraints, they 56% female. The age distribution of cannot be reset to an initial status, 176 of the users shows that the larg- which means that users will start est number of users corresponds to using the switches in the state the young adults and adults with ages last user left them. In the Vapriikki ranging from 19–29 (29%) and installation, it may affect the prob- 39–59 (29%), respectively. Followed lem-solving tasks that users had to by teenagers from 12–18 years old go through since the initial state of (21%) and children under 12 years the system could give hints or make old (20%). Finally, only 1% of the it more difficult. users were 60 years or over.

LESSONS LEARNED The remote tower projection acts as a nice point of attraction to catch the attention of people passing by. However, it is rarely used while users are interacting. The reason for this is possibly that the projection is located right above users, which makes it uncomfortable for them to look up to see the remote tower. The switches were also successful in catching attention and attracting users. Additionally, some users expressed that it was their favorite parts of the telegraph. They mentioned that the sounds and the feeling of moving the switches was a

74 4. CASES 4.9 AMOS ANDERSON OVERVIEW nical implementation were carried discussion directly with the viewer. stallation triggers content clips ac- out in an iterative manner brain- Installation of the final pi- cording to the presence of viewers. During the Miracle project, build- storming content and testing of lot system begun on-site began Approximated location of viewers ing of new facilities for Amos An- technical feasibility in collaboration at the end of September 2016. estimated from the captured cam- derson Art Museum began at the with the museum staff. Planning of Pilot remained operational from era images is used for controlling center of Helsinki. Building site is the pilot system implementation as end of September to mid-Novem- the gaze direction of the eyes that located in an area with one of the a public installation within the con- ber, during which time also user area rendered on the display and highest density of daily pedestrian struction site required also coordi- feedback and statistics of the use overlaid on top of the statue. traffic in the city. The interest from nation between construction site were collected. Bust of the museum director Kaj the museum staff to investigate how management, builder of the pilot Kartio used as a statue in the instal- mixed reality could be used directly installation structure and PR agen- IMPLEMENTATION lation is produced by 3D modeling at the building site to increase peo- cy designing the overall look of the the shape of the head initially cap- ple’s interest and awareness of the visual elements used at the building Implementation consists of the tured as a 3D scan. 3D model is 3D building project was the main goal site. physical construction of the display printed with a plaster printer and set for the pilot. In the final pilot, the director of setup, physical statue, computing processed to withstand the outdoor Based on discussion between Amos Anderson Art Museum, Kaj hardware and software. Optical environment conditions. Content museum staff and VTT researchers Kartio, is present at the building see-through display is created with clips are video recordings of per- on various approaches how mixed site as a statue that is brought to life a combination of a high luminosity formance by Kaj Kartio. Post-pro- reality technology could be com- with the help of augmented reali- display and a half silvered mirror cessed and masked video clips bined with the museum building ty. The pilot system is aware of the which operates as a beam combiner are displayed with the optical see- project, an approach of using op- viewers looking at the construction overlaying the image on the display through display so that the loca- tical see-through display setup to site with the statue through a view- to the view the user has through the tion and accommodation distance augment a view to the construction ing hole in the construction site half silvered mirror to the physical matches with the statue as seen by site through a construction site fence. Based on the detected view- statue placed into the construction the viewer. Aligning of video cap- fence was chosen as the main tech- er presence, the pilot system drives site. ture of live performance together nical approach to be used in the pi- the content clips where museum di- Installation uses several with the statue causes video image lot implementation. rector introduces various aspects of USB-cameras to inspect the area in to fuse together with the view the After the initial decision of tech- the building project. Goal is to cre- front of the installation in order to user has to the statue, thus creat- nical approach to be used, the de- ate a feeling of a living status that is detect people using computer vision ing an illusion of statue being alive velopment of the content and tech- aware of the viewer and engages a analysis. Software used by the in- rather than seeing statue and over-

75 4. CASES laid video image separately. COMMERCIALIZATION Reaction to viewer presence and use of gaze direction control to cre- Introduction of the pilot case re- ate an eye contact between viewer sults led to a design of a simplified and statue used by the installation ‘Talking head’ installation in dis- enhance the experience, creating cussions together with the staff of illusion of personal conversation Museum of Kymenlaakso. With and intimacy. This personal feeling the simplified installation version, of communication increases the at- a human character is printed to a tention and engagement of viewers scale on a cardboard material inte- and improves the distribution of grated with a tablet device. Tablet Image 1: Prototype of the installation being built and tested message contained in the augment- device display is used for displaying ed reality content. eyes for which holes are cut to the cardboard character. Tablet device USER EXPERIENCE is monitoring the area in front of the printed character with the em- User experience about the usage bedded camera of the tablet device. of the installation was collected by From the captured camera images UTA with a paper-form question- people are detected and audio con- naire. Collection was done as a one tent and eye gaze direction are con- session, during which time the in- trolled accordingly. First iteration stallation was in normal operation of the simplified installations were Image 2: Building of the installation at the final location in construction site mode and users were passing by manufactured and installed at the the installation with no prior in- Museum of Kymenlaakso at the formation given. In summary, the end of 2016 as part of the ‘Kasvoja feedback collected from the users’ Kymenlaaksosta ‘exhibition. displays a positive evaluation of the experience, 87,5% of all users grad- ing experience to be either good or very good. Image 3: Installation integrated with the Image 4: Augmented statue as seen in with construction site fence as a viewing hole the installation

76 4. CASES 4.10 MARITIME CENTRE VELLAMO / SELFIEWALL OVERVIEW based on depth camera technology. This enables the visitors to be A new exhibition of Museum of Ky- augmented into the photos as be- menlaakso was opened in Maritime ing part of it. Some dozen different Centre Vellamo, Kotka, in fall 2015. historical photos were chosen as the Together with the new exhibition, content, displayed as a continuous the museum expressed their wish reel. to create a high impact interactive The photos are segmented in dif- installation which would allow mu- ferent layers so that the users can seum visitors to “become part of see themselves e.g. behind differ- the history”. As result of workshops ent photo elements. The users are with the museum personnel, VTT transformed in the same perspec- Image 1: SelfieWall system with 2x2 screen. produced a concept of implement- tive with the photos, and also the ing an interactive large scale dis- color characteristics of the users are play that would augment viewers as tuned to match the photo appear- seamlessly as possible to historical ance, e.g. black & white, graining, photographs, and sharing the pho- lighting etc. Several visitors can be tos in social media. simultaneously augmented in the photos. Altogether, the application IMPLEMENTATION creates a surprising and delightful experience when entering the mu- The first implementation was com- seum space, and encourages playful pleted by time of the new exhibition interaction and personalized expe- opening. The system consists of a rience with the layered photos. large 2x2 screen display, installed at the entrance of the museum space. Visitors approaching the scene are separated from the background, Image 2: SelfieWall at AWE2016 exhibition, Silicon Valley, USA.

77 4. CASES

USER EXPERIENCE Kansallisooppera / Kirjamessut. As the first international custom- User studies of the visitors were er, Autodesk Inc. ordered two Self- conducted by Helsinki Universi- ieWall licenses in January 2017, one ty. In summary, the study revealed as permanent installation at their that over 50% of visitors spent over technology museum in San Fran- two minutes interacting with the sisco and another one for exhibi- system. Overall the installation tion tours around the continent. can be considered as a big success, Patent application of SelfieWall fulfilling very well the museum’s was filed in spring 2016. original expectations of engaging the visitors with the museum con- Video of press release by VTT: tent. Later on a second system was https://www.kauppalehti.fi/uutiset/ installed at another exhibition at selfie-seinalla-voi-sukeltaa-keskelle-men- Vellamo. neisyytta/kjnAPDBw.

COMMERCIALIZATION The system was coined by the museum as “MiracleWall”, or “Ih- meSeinä” in Finnish. Subsequent development of the system has taken place in an EIT Digital funded project StreetSmart, under name “SelfieWall”. The system has thereby gained lot of commercial attention and new customer segments also in new applications such as adver- tising, by way of immersing people into brand related photos. SelfieWall customers in Finland include ClearChannel / Itis, Kult- tuuritalo, HOK / Ateneum and Image 3: SelfieWall and Kullervo painting at Ateneum, Helsinki.

78 4. CASES 4.11 MUSEUM OF TECHNOLOGY / AUGMENTED RADIO WAVES OVERVIEW IMPLEMENTATION A new exhibition “Etäunel- The pilot system is now available for mia” — history of electronic com- general museum visitors on the up- munication — was opened in Tek- per museum floor. It is implement- niikan museo, Helsinki, in fall 2016. ed on a large screen on a fixed po- As general goal for the MIRACLE dium, with possibility to move the pilot, the visitors should be provid- screen around to augment different ed with new kinds of memorable devices at the exhibition area. Also experiences and insight into the ex- a mobile version of the system was hibition, including ways of under- implemented on a tablet device, but standing how the technology works, it was not made available to general enhancing the learning experience audiences. by “seeing the invisible”, and link- The application is designed to Image 1: Pilot system at Museum of Technology. ing the exhibition items together be equally interesting to all visitors, under a general visual theme. however school students being a The museum people came up special target group. The motiva- with the idea of augmented visual- tion is to extend from traditional izations of electromagnetic waves wave theory presentations to more being transmitted and/or received sophisticated and easily memorable by different devices in the exhibi- visual methods, and link different tion. The user may view the mu- waves to real physical objects at the seum exhibit devices and see the exhibition. This will support under- related electromagnetic waves si- standing and learning of the wave multaneously augmented, or select theory and its relations to different certain devices or wave frequencies devices. from the system’s user interface. The visualized data is pre-de- The waves are visualized in an in- fined based on the devices in the teresting and instructive fashion. exhibition. The target devices (e.g. mobile phone, radio, television) Image 2: User interface and augmented radio waves.

79 4. CASES have their own wave lengths and tion at similar exhibitions around properties. These waves are mapped the world. in visual light range and shown in augmented reality on the screen. Video of the application is available The user may change the visual- at: ization by selecting desired wave https://youtu.be/YhbbiJ972UY. spectrum or only wave source. The user interface to select the devices and wave lengths was designed to be as self-explanatory as possible. A student should be able to use the system with only minimal guides at start of the application.

RESEARCH CHALLENGES AND RESULTS The application field is novel and the implementation provided several design choices on user interface and usability. A special research item was the rendering, how to visualize the invisible? The nature of electromagnetic waves makes the visualization very challenging. The visualization must support the text- book theory but should also visualize the waves in real environment in an interesting fashion. The final pilot implementation seems to solve these research questions as well as the original requirements well, having potential also for wider adop-

80 4. CASES 4.12 ILMAILUMUSEO / HOW TO FLY OVERVIEW IMPLEMENTATION The Finnish Aviation Museum“Mixed reality” was the key word of (“Suomen ilmailumuseo”) is reno- MIRACLE. The “How to fly” –ap- vating its’ exhibitions and planning plication was really putting togeth- a brand new museum building in er a) Augmented Reality; b) the Vantaa Airport area. The key issue mechanical equipment (real science is the development of the audience test model The Wind Tunnel), and relations and especially coopera- c) the learning and use took place in tion with the schools, teachers, and the exhibition context surrounded school administrations. The main by real objects as mainly airplanes. goal was “bridging the gap between How to fly Air wing applica- formal education and informal tion demonstrates how air moves Image 1. Augmented reality demonstration of airflow and resulting forces on a model air- learning”; e.g. giving opportunities around the airplane wing and what craft. The location and orientation of the model are observed from the graphics on the for out-of-school education. kind of forces it creates to the wing. model aircraft’s body. The content of the “How to fly” It also shows how air moves around AR-solution was strongly support- different shapes such as cones and ing the curriculum of the Finnish spheres. It was implemented on An- comprehensive school. Especially droid platform using Unity game the New Curriculum (OPS 2016) engine. Several augmented reality is demanding the use to apply toolkits were used during the pro- “PBL-Phenomenon Based Learning” ject, including ARToolkit and Vu- (“Ilmiöoppiminen”), and this How foria. First implementations relied to fly –case gave fruitful opportuni- on Unity physics engine but it had ties for the teachers. performance limitations. A Unity The application was also pre- editor plugin was implemented so sented for greater audiences in the that we could easily create dynamic “Suuri Lentonäytös 100 v Suomi”, three dimensional trajectories for June 9th 2017, the biggest air show the air particle flows. Image 2. An AR demonstration consisting of a fan and an adjustable miniature wing was in Finland. Augmented reality toolkits were used as pre-material before a museum visit. Airflow is visualized on the computer screen.

81 4. CASES

used to find the angle of the air- um. The data, which consisted of REFERENCES plane wing so we could calculate both cognitive tasks and self-report the airflow trajectories. A hidden questionnaires, were collected us- 1. Salmi, Thuneberg & Vainikainen (2016). three dimensional wing model with ing a pre- post-test design and were Making the invisible observable by depth mask shader was used to cre- analysed by SEM path-analysis. Augmented Reality in informal science ate an illusion where airflow could The results showed that AR-tech- education context. International Jour- be hidden behind the wing seen in nology experience was beneficial nal of Science Education, the camera feed. for all, but especially for the low- http://dx.doi.org/10.1080/21548455.20 The mini-wing demonstration est-achieving group and for the 16.1254358 was used in the Science Show, which girls. In general, pre-knowledge Image 3: The real scientific research equip- was given to all pupils who visited skills predicted post-knowledge ment, The Wind Tunnel, created by the von the exhibition. The demonstration test results. As expected, school Karman Institute, was used in the demon- lasted 30 minutes in the beginning achievement had an effect on strations of the visit. Also the wind tunnel pre-knowledge results. In addition, was utilized in this presentation. motivation turned out to be an alter- How to fly –demonstration was native key route for learning. Being utilized also as a tablet version. Pu- a boy predicted directly or indirect- pils could use it while roaming in ly all other motivational variables, the exhibition among tens of differ- enjoyment and interest, but girls ent real airplanes. had a higher relative autonomy experience (RAI). Situation motiva- RESEARCH CHALLENGES tion and attitude towards learning AND RESULTS in the science exhibition were much Image 4. The portable wind tunnel can be more strongly inter-connected packed into cases for transport. The hand- The research project related to “How among boys than girls, and attitude held tools on the foreground can be used to to fly” was analysed in order to get predicted post-knowledge only for demonstrate the effects of airflow on differ- evidence about learning using Aug- boys. AR seems to be a promising ent shapes. mented Reality (AR) technology method by which to learn abstract and the motivational and cognitive phenomena using a concrete man- aspects related to it in an informal ner (see also[1]). learning context. The 146 participants were 11–13 year old Finnish pupils visiting the Aviation Muse-

82 4. CASES 4.13 MOLECULE MOVEMENTS OVERVIEW Toolkit and Vuforia. A native An- RESEARCH CHALLENGES droid Java plugin was written for AND RESULTS The pupils used the AR-application Unity to communicate with the “Molecule Movement in Gas”, which wireless ETI BlueTherm tempera- The role of the digitalization in enabled them to see something ture probe. The biggest technolog- teaching is rapidly growing. Along more than is possible by ordinary ical challenge was not related to with that, adapting of the augment- experiments. While investigat- augmented reality at all: It was to ed reality (AR) is becoming more ing thermal motion the velocity of overcome the bluetooth connection and more common at school and molecular nitrogen for example in errors and to create a robust appli- therefore its study within the for- a refrigerator or on a hotplate can cation with a good user experience. mal and informal learning context be compared both by following the is important. Participants of the motion of augmented molecules in AR-study were 416 sixth-graders. different places and also by com- They used the AR Molecule Move- paring the different velocities of the ments at the science center. By the molecules on a temperature-veloci- SOM-clustering analysis the aim ty graph. was to identify subgroups of the students and supplement earlier re- IMPLEMENTATION sults. The students using AR in science learning were clustered based Image 1: Digital AR-thermometers — first Molecule Movements application on reasoning, motivation and sci- and second version — developed to show displays real time temperature ence knowledge results. Earlier it the molecule movement’s speed in different measurements using bluetooth con- had been noticed that after the temperatures nected wireless thermometer. The AR-experience the science test re- application was implemented on sults generally improved, but most Android platform with Unity game of all gained the students with low- engine. Unity physics engine was est achievement. The cluster analy- used to create the particle move- sis supplemented this by identifying ment. Several different augmented a boy majority group in which the reality toolkits were investigated students were especially interested during the project, including AR- in science learning both at school

83 4. CASES and at the AR-applying science REFERENCES center. In spite of the low school achievement their high motivation 1. Salmi, H., Kaasinen, A., & Kallunki, V. led to good science learning results (2012). Towards an open learning subsequent to the exhibition. The environment via Augmented Reality earlier results, according to which (AR): Visualising the invisible in science the girls closed the science knowl- centres and schools for teacher educa- edge gap between boys after using tion. Procedia — Social and Behavioral AR, became more relative, as two Sciences, 45, 284–295. girl-dominated subgroups were 2. Salmi, H., Sotiriou, S., & Bogner, F. identified. In the other the students (2010). Visualising the invisible in were motivated, however the wrong science centres and science museums: answers increased; in the other the Augmented Reality (AR) technology students were highly uncertain and application and science teaching. In N. after the AR-experience there was Karacapilidis (Ed.), Web-based learning no change. Possible reasons were solutions for communities of practice. considered on the basis of motiva- (s. 85–208). New York, NY: Information tion and concept formation theo- Science Reference. Hershey. ISBN 978-1- ries. The clustering results comple- 60566-711-9. mented earlier findings of AR-gains in learning.

84 5. SUMMARY 5. SUMMARY

This chapter crystallizes the results of the MIRACLE project into a compact set of observations and guidelines.

85 5. SUMMARY

Effective attractions • Offer alternative stories and and immediate as possible. The Get the audience involved Many people find services and ap- interpretations. In many cas- more the people can communi- • Communication technolo- plications based on mixed reality es there’s something about the cate the better, and the better the gy, and especially social media technologies very interesting. Over- history that we can’t be abso- people know each other the bet- channels, can be used to make all, the pilot applications developed lutely sure — it might have been ter the outcome. the general people do things to- within the project got mostly pos- this way or that way. Would it • Experts of one discipline should gether and in cooperation with itive comments from the audience. improve the visitor’s experience know enough about the require- the professionals. To maintain the interest, the sub- if you described more than one ments related to other disci- • When the general audience is ject and content must be interesting possible interpretation? This is plines in order to provide results supposed to have valuable sto- in itself. New technical gimmicks easy to do with virtual content. that will advance the production. ries or other information to be only attract for a moment, so great • Allow access to a lot of informa- Historians should know some recorded and shared, you could attention should be paid to the sto- tion, no matter what the physi- technology, and technology ex- get them involved by means of ry and the actions the user can do cal space limitations are. With perts should understand the es- social media or tailored services. within the app. the help of a portable device, all sentials of the content that is to The application created to be a the information the visitor will be presented. guide for a site can also contain To make the most of the possibilities get needs not to be put physi- • Consider what tasks can be done functions enabling communi- of mixed reality, consider these cally into the exhibition room. by the “in-house” personnel, and cation between people. Visitors potential advantages of MR and Give the visitor more info about what must be purchased from may want to share their experi- think how they could be realized in things that are of interest to just outside. Appropriate tools allow ences with others, and they may your case him/her. Offer guidance in more some tasks that would otherwise be able to bring in something • MR can be used to explain and languages than just the usual be done by technical profession- that can be added to the general visualize many phenomenons, three. Let the visitor get access to als to be done by the local per- offering at the site. even complex ones. Are there all kinds of related information sonnel. • Communicating and forming such in your case? MR can result in the web. • Be prepared for an iterative pro- communities around various in efficient learning, especially cess that may take time. Make topics can also have societal ef- for persons with a low achieve- Many experts, good cooperation the schedules so that there is fects. Sometimes this may be the ment level. • MR project often require coop- time for solving some unantic- main target you’re after. • Add life, interactivity, action to eration between experts of dif- ipated problems or making re- the usually still museum envi- ferent areas. How well it goes has designs. Especially when people ronment. Think what could be a great effect on the outcome of are working together for the done to make the experience the project. first time there may be things to with your case more active and • Make the cooperation between clarify to each other, and all that lifelike. experts of different areas as easy takes time.

86 5. SUMMARY

What could go wrong? turb others around. Paths may Technical and usability problems Keep it going It isn’t easy to guess in advance be blocked by app users, and can spoil the experience. Generally Making an app may be fairly easy, where the problems will be. But noises coming from apps may be it can be said that the experience is but keeping it alive for an extended some potential risks can be predict- disturbing. as good as its weakest part. period of time probably isn’t. Soft- ed and avoided — like these ones. • The locations where the app is • With cultural travel applications ware platforms and operating sys- • Using an app the visitor may for- supposed to be used should be tailored for individual sites, a tems are frequently updated, new get the reality around. The result away from main pathways. Aug- large portion of the usage ses- product generations are launched may be that the visit isn’t as full mented reality tracking and ren- sion will be by first-time users with new features, and even new an experience as it could be. Or dering may run into problems with the app. Therefore it is im- product categories appear on the worse: the visitor might stum- when there are a lot of people portant to assist the user getting market. All this may cause some ble on things and cause harm in the area where virtual con- started with the app. It may be functions and products to become to him/herself, other visitors or tent should be, so the locations done in many ways, like tutorial obsolete — either the app may not exhibits. To avoid this, the ap- and directions of AR activities screens, videos, printed instruc- work any more, or something in it plications should be designed so should be where heavy visitor tions on the site or even personal is considered out-of date. that they do not require constant traffic is not expected. assistance. It should just be there This is less of a problem for limit- concentration on the device. • Sound design should be done to make sure the users won’t get ed time cases, like apps for a specif- • Holding a device up front is also considering the possible distur- stuck with the app when they ic exhibition that will be over after tiring. Even though the phone or bances. In some cases the sounds give it a try. a few months. But often museums tablet would be light, just holding could be offered only via head- • It is also a good idea for the ap- plan their exhibitions to go on with- hands up will be difficult after a phones, and where speakers are plication to give hints whenever out major updates for several years, while. To avoid this, the appli- used so that the sounds will be it seems the user has problems and in such cases the need for soft- cation should not require con- heard by other visitors, the audio proceeding. Tell what to look for, ware updates may become evident. stantly “looking through” the should preferably be subtle. make the essential control func- This should be taken into account device. Scene durations should tions obvious and easy to notice. already in the planning phase of an not exceed some tens of seconds • Tracking problems may occur application. to keep usage comfortable. In fairly often, at least with current • If the plan is to have the app some cases it is possible to let the technologies. The app should be running for at least a few years, user freeze the view and then use able to detect situations where there should be a plan about fi- the device in a more comfortable tracking doesn’t work properly, nancing the software updates position, continuing to observe and let the user choose an alter- that could be needed. Even for the details on screen and getting native solution. For example, a shorter usage periods an update more information. 360 degree video clip can be of- plan would be a good idea, as it • Visitors using an app may dis- fered instead of the AR view. is common to find something to

87 5. SUMMARY

fix in the app after it has been taken into wide use. The plan may also contain more or less continuous content updates, which don’t require changes in the actual application. • When it seems probable that there will be new kinds of products on which the app could be used in the future, it is worthwhile to design the application concept and architecture so that it is possible to make new variants without writing everything from the scratch. • To keep users’ interest up, the content of the application should be flexible: people will have a better experience of a revisit if the application shows something new instead of repeating everything exactly the same way as the first time.

88 6. TOOLKIT 6. TOOLKIT

The Toolkit is a collection of software applications and components for mixed reality production. All these tools have been tested in the MIRACLE project; some are written in the project and published as open source software. Everything chosen in the Toolkit are low budget, free or open source tools suitable for production by multidisciplinary teams with limited resources.

Shown costs are dated 7/2017.

89 6. TOOLKIT

UNITY ALVAR BLENDER ARTICY:DRAFT https://unity3d.com http://virtual.vtt.fi/virtual/proj2/multime- https://www.blender.org https://www.nevigo.com/en/articydraft/ dia/alvar/ overview/

Game engine, a development envi- ALVAR (A Library for Virtual and Widely used 3D modeling and an- Articy:draft is a visual environment ronment for real-time applications Augmented Reality) is a suite of imation package with a variety of for the creation and organization originally targeted for game devel- products and libraries for creating features. As a general 3D applica- of game and story content. It is opers but recent additions in func- virtual and augmented reality ap- tion it has steeper learning curve designed for creation of non-line- tionality target more wider user plications, developed by the VTT than specific beginner-friendlyar storylines, branching dialogues, groups such as animated movies. Technical Research Center of Fin- software (e.g. Marvelous Design- level layouts or character and item As it is made to handle varied 2D land. Library includes a high-level er). Fast to use once learned and databases. Content can be export- and 3D content, it is a platform well application programming inter- has plenty of learning material ed into various formats — including suited to mixed reality applications. face (API) and several tools to cre- freely available, including tutorial JSON, XML and Microsoft Office. Deploys applications for all popular ate augmented reality applications. videos. Contains tools for model- Content can be integrated into Uni- mobile, desktop and console plat- Low-level interfaces also make it ling, rigging, animating, sculpting, ty with the comprehensive arti- forms. possible to develop custom solu- texturing and more. Comes with a cy:draft to Unity plugin. tions. built-in Python interpreter for tools Cost ALVAR has been used as the programming and extending func- Cost Free for non-commercial use and tracking technology for many pilot tionality. Free trial, different license models, limited commercial use, reasonable applications in MIRACLE. It offers also educational discounts have subscription-type licenses available both 2D marker or image based Cost been available. when used for commercial applica- tracking and markerless, point Free and open source. tions. cloud based tracking solutions.

Cost Reasonable per developer licensing model.

90 6. TOOLKIT

MARVELOUS DESIGNER SUBSTANCE PAINTER SUBSTANCE DESIGNER AGISOFT PHOTOSCAN https://www.marvelousdesigner.com/ https://www.allegorithmic.com/products/ https://www.allegorithmic.com/products/ http://www.agisoft.com/ substance-painter substance-designer

A tool for designing for 3D virtual Texturing software with support Node-based texture creation soft- A photogrammetry tool for cre- characters and objects by familiar for physically based rendering pipe- ware with support for physically ating 3D models and point clouds concepts of cutting and sewing. 2D lines common in current game en- based rendering pipelines. Techni- from photographs taken of a target clothing patterns are simulated on gines like Unity and Unreal. As a cally oriented, but allows creation from different angles. Supports as top of 3D objects for realistic folds plus the model can be updated after of procedural tiling textures which well small objects as entire environ- and looks. Basics are learnable in a texturing and the software repro- can be used straight-away in sup- ments. Standard and professional few days allowing quick start in the jects textures to the updated model. ported game engines (e.g. Unity) version with different capability clothing creation; does not require Has baking functionality for creat- and in Substance Painter to gener- profiles. The standard version suits skills in software development.ing different maps for masking and ate traditional texture sets. Textures AR productions well while the Pro- Data can be exported and import- texturing purposes. Can export created in Substance Designer for- fessional license is targeted for high ed between MD and other common and package the created textures mat can have variables exposed by quality aerial imaging and manipu- 3D tools. for many different game engines the texture artist. This allows quick lating pointclouds of very large en- and renderers. changes of texture look in game vironments. Cost engine, like color and smoothness 30 day free trial. Monthly and year- Cost or more complex variables like wet- Cost ly subscription licenses available, Depends of license: free 30-day ness and weathering. Professional and standard versions one-time fee USD 550. trial, indie license $149 (under have very different pricing. Educa- $100K revenue), pro license $590 Cost tional licenses available. ($100K-$100M revenue). Depends of license: free 30-day trial, indie license $149 (under $100K revenue), pro license $590 ($100K-$100M revenue).

91 6. TOOLKIT

MIRACLE OPEN SOURCE COMPONENTS https://github.com/ututrc/ https://github.com/thomaspark/bootswatch/blob/gh-pages/LICENSE

A number of software components MuseumExplorer Content supporting implementation of MR Management System applications was created in the MeteorJS based web content man- MuseumExplorer Unity SDK UX Lab MIRACLE project. The main com- agement system (CMS) to create Unity based software development Meteor based data logging web ser- ponents are described below. and upkeep data for augmented kit built with C# to create immer- vice targeted for collecting user ex- All these components are pub- reality applications targeted to cul- sive augmented reality applications perience measurement data. Mod- lished as open source, under the tural heritage sector. The system in connection with the MuseumEx- els data into user definable streams MIT license. provides technical developers, con- plorer CMS. The SDK provides all that can be used to feed in data via tent experts and eventually even basic logic and functionality build- simple REST API. Provides rudi- Cost casual users interfaces to access and ing blocks and allows any 3rd party mentary web UI to define the data Free and open source. modify the recorded data via inter- tracker to be connected to provide streams as well as view and export faces targeted for each target group. the tracking data. the collected data. The system employs a ARML 2.0 like XML data format to exchange data back and forth with client applications. Additionally, the system provides a 3D editor for use in web browsers to create and edit MR content.

92 GLOSSARY

360 video/photo Marker (fiducial marker) A video or still image the view of which covers the full sphere seen from the An object placed in the field of view, for use as a point of reference for posi- location of the camera that recorded the image. tioning and tracking of an AR device. Figures consisting of black and white rectangular elements are typical AR markers. AR Augmented Reality Mixed Reality An environment consisting of real and virtual elements. According to a classic ARML definition, the term mixed reality covers everything that falls between reality Augmented Reality Markup Language (no virtual elements) and virtual reality (completely virtual environment).

Augmented Reality Mocap The real world view is augmented with additional, artificially generated infor- Motion capture. Recording the movement of objects or people. Motion mation, for example computer-generated 2D and 3D images that are super- capture can be used to animate 3D characters e.g. in animated movies or imposed on the real world view. augmented reality applications.

HMD MR Head-Mounted Display. VR glasses like Oculus Rift and HTC Vive are examples Mixed Reality of HMDs. Optical see-through display HUD Eyeglasses with transparent display so that the user can see the actual envi- Head-Up Display. A transparent display that presents data without requiring ronment through the glasses while virtual content can be added to the view users to look away from the usual field of view, typically using a reflecting by a display element. (see Video see-through display) glass that combines a monitor’s image to the view that is seen behind the glass. Photogrammetry Vertex A process of making measurements from photographs, especially for recover- A single point in a coordinate system. 3D models are made of nets of vertices ing the exact positions of surface points. 3D models of objects can be made connected by surfaces. using photogrammetric methods on a large number of photographs taken of the object from different angles. Video see-through display Head-mounted display where the actual environment is recorded via a video Point cloud camera, virtual content may be added to the camera view, and the user sees A set of data points in a coordinate system. In AR, point clouds can be used as the resulting video on the display. tracking targets to define the device’s location and orientation. Virtual Reality SDK An artificial environment generated by a computer and displayed to the user Software Development Kit via displays, speakers and possibly other perception-generating devices.

SLAM VR Simultaneous Localization and Mapping. An augmented reality tracking Virtual Reality principle, where the device creates and updates a model of the environment while the application runs.

Tracking Keeping the exact location and orientation of an augmented reality display device updated during its use.

UV UV coordinates are 2D coordinates used to define texture mapping for 3D objects. Called UV because XY were already taken. UV-unwrapping is the process of projecting a 3D object on a 2D plane.