Non Visuals Material Exploration of Non Visual Interaction Design

NON VISUALS MATERIAL EXPLORATION OF NON VISUAL INTERACTION DESIGN

MFA MASTER THESIS 2020 SEBASTIAN DE CABO PORTUGAL NON VISUALS Sebastian Cabo Portugal de

“To understand something is not to be able to define it or describe it. Instead taking something that we think we already know and making it unknown thrills us afresh with its reality and deepens our understanding of it” – Kenya Hara MFA Thesis Project 2020 Thesis Project MFA

- 2 - NON VISUALS Forward This project really scares me, and that’s why I had chosen to do it, I didn’t want to do something that I had done before, I wanted to try new technologies and new design methods that were completely novel for me.

This is why I chose to work with VR and AR technologies, something I have barely touched before. I wanted to really understand how to build things for these kinds of technologies, and more importantly find ways to make prototypes and lo-fi iterations without having to do massive amounts of code, or even no coding at all. Sebastian Cabo Portugal de

This is also why I worked with audio, something I’ve done sporadically during filmmaking but has always been my weak spot due to my heavy focus on the visual side of things. Ironic I know.

Working with visually impaired people is a user group I haven’t collaborated with yet and this also scared me, from an ethical and empathetic point of view and from a pragmatic one, for example how do I show people concepts or initial sketches if they can’t see a sketch. Finding non visual ways to communicate my ideas.

“This is not a problem driven project or a technology driven project, but an experiment driven one”

Normally the projects we’ve done in the past dealt with a specific problem or user group and in the end we could evaluate against these points of contact to see if the outcome had been successful or not. - if we had "solved" the problem at hand. But on this project I wanted to try a research through design method, a more experiential process, one where you can have many different starting points, even opposing ones, and you develop the research and concepts through making - its use will come out after. MFA Thesis Project 2020 Thesis Project MFA

Even though everything about this project scares me, and I can’t imagine what the outcome will be, I feel confident that something good will be produced, I have to learn to trust the process, and myself as a designer, and that is the main reason I wanted to put myself in this situation, to prove myself I can get out of any situation successfully even if I don’t know anything about the subject at all at the beginning of it. In the future I won’t be able to choose exactly what I’ll be working on so I like to train myself to be prepared for any situation and find ways to improvise my way out. I just couldn’t predict how the world would change in the short time this thesis was being produced, and how challenging that was going to turn out to be.

This is why I chose to work on a topic that scared me most as my final project in this master’s degree. Sebastian de Cabo Portugal - 20/01/2020

- 3 - NON VISUALS Abstract Design is all about visuals, or that is what I have found out during this thesis, from the process materials to the outcome our main entry point to any problem is how will we solve it visually so it’s understandable for the general user. This aspect is problematic in itself due to the fact that we, as humans, understand the world and the things around using all our senses continuously, even though we can forget as visuals are so overpowering.

There is a huge opportunity area in exploring our other senses and bringing them back to technology, and this can be seen in works in the past like Tangible Interactions [1] or Sebastian Cabo Portugal de Natural User Interfaces [2]. But in this moment in time, where all these new technologies like VR/AR and IoT are about to enter our lives and change them forever, this topic is more important than ever. We have already seen what happens when we turn humans into mere machines with some fingers as interactive inputs, and barely any senses to process all the information given to us. Now that these technologies are still young and malleable, we can direct the future to where we want it instead of being guided by the technology itself.

To do this we need to reimagine the design process, not reinvent the wheel, but add experts which we currently leave behind and I argue are key to unlock these technologies, experts not only of the technological side of things but on the human side too, like physiotherapists and dancers. Add also people who we never think about when we think of VR like visually impaired users, which could make these technologies inclusive since early on, insteadof as an afterthought like we usually do. Not only people, but we also need to add new materials to understand how we use our senses and explore ways that we can understand and explore them differently; like bodystorming and improv theatre because when things aren’t visual, how do you sketch it? A sketch turns into a video about movement. MFA Thesis Project 2020 Thesis Project MFA

The end result provides a wide breadth of examples of the types of innovations that can come out of using these new design materials, and to open new frontiers. From a VR game with no visuals whatsoever to an AR location based story game, to a home sized multimodal operating system containing several different apps controlled through physical movement. The examples open up the space instead of closing into a single solution. This is just the tip of the iceberg, a hope that others will be inspired by it and continue with this journey that has just started, to guide the future into one that is more technological and at the same time more human than ever before. What we know is that VR does not equate Visual Reality. Keywords – immersive environments, virtual reality, mixed reality, wearable computing, virtuality continuum, internet of things

- 4 - NON VISUALS Contents

FOREWARD -3- 4.0 MATERIAL EXPLORATIONS -27- ABSTRACT -4- - Working in Sprints based on Design Principles -28- CONTENTS & ACKNOWLEDGMENTS -5/6- - How a Loop looks from the inside -29- EXPECTATIONS & ASSUMPTIONS -7- - My Activities -30-

1.0 INTRODUCTION -8- INTERVIEWS Sebastian Cabo Portugal de - Uniting digital and physical worlds -9- - Interview - Luca Contato -31- - Ethical Considerations -10- - Interview - Christy Smith -32- - Philosophical Considerations -11- - Interview - Christy Smith (cont.) -33- - Interview - M Eifler -34- 2.0 BACKGROUND -12- - M Eifler Studio Metaphor -35- - The world we come from/the world to come -13/14- - M Eifler 3 levels/scales of interaction inside VR -36- - Focusing on visually impaired users -15- - Interview - Arianna Ortelli -37/38- - Audiogame or be prepared to grind hard -16- - Interview - Enrique Sanchez -39- - How are we adding senses to VR? Add more tech -17- - Interview- Pablo Gody and Nelson Sanchez -40- - Why read a book when you have audiobooks -18- - Inspiration -19- EXPERIMENTS - Inspiration from Microsoft Research -20- - Playing with existing tech - Amazon Alexa -41- - Games, an excuse to improve the whole tech -21- - Playing with existing tech - VR games -42- - Playing with existing tech - VR games -43- 3.0 APPROACH + METHOD -22- - Playing with existing tech - Torball -44-

MFA Thesis Project 2020 Thesis Project MFA - Insights from background research -23- - Experiment 1 - Spatial awareness through music -45/46- - Flip the norms - "make the known unkown" -24- - Experiment 2 - Playing fussball blindfolded -47- - DESIGN PRINCIPLES/Perfect Timing -25- - Experiment 3 - Controlling Logic Pro with moves -48- - Research through design -26- WORKSHOPS - Workshop 1 - Music Maker with Toby & Julien -49- - Workshop 2 - Physical Game Jam/Bodystorming -50/51- - Suddenly the world changed -52- - Workshop 3 - Communicating through movement -53/54-

- 5 - NON VISUALS Acknowledgments 5.0 CONCEPT EXAMPLES -55- This thesis wouldn’t have been possible without the help and support from: - Introduction to Provocation & Examples -56- - My parents, for always being close even when I’m so far away - Playing games without vision - VR Audiogame -57- - My supervising tutor Monica Lindh-Karlsson - Playing games without vision - AR location based -58- - My thesis supervisor Stoffel Kuenen - Using our own bodies/movements/senses/objects - The best class anyone could ask for, my IXD colleagues - The rest of UID for always being up to try one of my weird experiments as controllers/input- Home OS -59- - Amanda Stendahl and the Torball group in Umeå

- Using our own bodies/movements/senses/objects - RISE Interactive studio Umeå Sebastian Cabo Portugal de as controllers/input- Physicality apps -60- - Luca Contato from Rising Pixel - Physicality apps - Music Maker through movement -61- - Christy Smith from caniplaythat.com - Physicality apps - Walk + Talk -62- - M Eifler from Microsoft research - Physicality apps - Home Physical Rehabilitation -63- - Arianna Ortelli from the Blind Console - Laura Vendrell from VRave 6.0 ANALYSIS -64- - Enrique Sanchez from Orbisnauta - Insights from explorations -65- - Pablo Godoy Physiotherapist at NHS Trust - How to develop a concept -66- - Nelson Sanchez Physiotherapist at Fisiotec - Example in use: Music Maker -67- - Julian Loretz from APD - Julien Desvaux from IDI - Design methodology insights -68- - Rachel Cheung from Goldsmiths University Design values in the Concept Examples -69- - Javier Viera aka MakerFly - VR Audiogame values -70- - Teenage engineering for their sick products - AR location based game values -71- - Humble bundle for the amazing set of VR games - Music Maker app values -72/73- - HTC for their Vive headset - Walk + Write app values -74- MFA Thesis Project 2020 Thesis Project MFA - Home rehab app values -75- - Home OS values -76/77-

7.0 DISCUSSION -78- - Is this better than whatwe had before? -79- - What’s changed in the design process? -80- - Conclusions -81-

8.0 REFLECTIONS -82- - Postface -83-

REFERENCES -84/85- - 6 - NON VISUALS Sebastian Cabo Portugal de

Expectations Assumptions The final goal would be to end up with two or I’ll start by stating my assumptions before the start of the project after carrying three games or experiences to be used as exam- out just a tiny bit of research in order to produce my brief: ples, they would include many of the experiments done and tested throughout the process. It would - Developers don’t think this is an actual issue - What’s the point of making also include a menu to choose between the sev- something in VR if the user can’t see - Is the most common answer I’ll get from eral options of the games. This would be instead developers when I explain the thesis topic, and I feel that is exactly the point why of making one single solution that encompasses I’m trying to make. everything as I think it would be too confusing, and it would enable me to show the many possibilities - Visually Impaired people are so forgotten that there’s no options out there in and ways this problem could be approached by VR/AR for them. others in the future. MFA Thesis Project 2020 Thesis Project MFA - Is it possible to show an idea if the user can’t see a video or a sketch, how do you My wish for this project would be to create some- explain an idea especially when it’s about 3D and about physicality? thing no one has ever tried before, and be at the forefront of what I hope will be a more important - Is it possible to create lo-fi prototypes of games that involve moving in space aspect of the VR and AR community in the near fu- with your own body and listening to sounds in your environment without coding? ture, or at least to raise awareness of these issues early on so they can be taken into account and im- - Does VR equate Visual Reality? Can we consider walking around a 3D Sound- proved as the rest of the technology is improved scape or Hapticscape VR? and experimented with. - There is plenty of guidelines to design for VR/AR/MR but they are all about the visuals and not much or none at all about sound, movement and physicality.

- 7 - NON VISUALS Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

1.0 INTRODUCTION

- 8 - NON VISUALS Uniting digital & physical worlds This topic could be called many things, VR with no visuals, soundscapes in virtual realities, exploring other senses in the virtual realm... The main idea is that I don’t want to focus just on VR or on AR, but on the physical and the digital realities; to more importantly, expand the ways we use our senses in these virtual worlds, explore ways we could understand them differently and thus more in depth and open it up for new opportunities that arise through new approaches or by using new design materials.

We give a higher priority to visuals in our lives than anything else, even though there are at least 4 other ways to sense our world (there are more), digitally we have been concentrating on exploiting our visual side, relegating sound to a secondary position and making touch, smell or taste disappear. This is not only problematic for the general population, but Sebastian Cabo Portugal de it can be a real impediment for people who are visually impaired. We choose who to include and who to exclude from early on via our decision to use multimodal interactions or not.

This is why I chose to tackle this topic, VR and AR are such young technologies, unbound by all the constraints inherent in screen-based digital technologies, with a language that has been developed and matured over the past decades, that we should really try and explore all its new capabilities. Everything in VR is an experiment right now but we’re already falling into our usual traps by concentrating on improving graphics to try and make the most realistic representation of our world possible. This sounds a bit counterproductive, if you want the highest representation of our reality, you already have it in the real world, why make a copy in a virtual one. We are stuck in the same old paradigms because we’re using the same old design materials.

On the other hand, if the digital reality and the real one, which were once separated by screens, gets blended into a single unifying reality, where both physical and digital coexist, we should be interacting with them accordingly to the way we do things in the real world, using our full capabilities of our bodies and senses so as to not get lost or overwhelmed in them by focusing on just one sense.

MFA Thesis Project 2020 Thesis Project MFA This new technology can be developed differently, we can develop it to include our other senses too, from walking around and touching objects through AR or haptics in VR, to aural soundscapes and voice control, so technologies can start to feel more natural as they adapt to us and not the other way round. I see the future designers on these jobs in teams that might includemore architects than graphic designers as it will be all about the space we move in, and dancers instead of UI experts, as they will be able to guide our bodies through movements that make sense for that application.

To create these future innovative experiences, especially in games, it would be impossible to get to if we keep looking "Engage our imaginations at sight as the sense with the highest grade of importance, and better visuals as our main source of inspiration. through what we hear and My anchor point to start the project was blind gaming in VR, because exploring a whole technology is too vast without feel rather than what we a clear starting point, so my research began focused on accessibility in games, especially for blind people which then see” evolved into how we can introduce more senses with less devices into our lives. - 9 - NON VISUALS Ethical Considerations

Who do we design for? Right now the user we have collectively created in our minds and for whom we work for is an incomplete version of a human being with huge eyes, tiny mouths, small ears, no nose or tongue to taste, and with no body, just one hand with a couple of fingers that can touch but not feel. This is a far cry from what we actually look like and how our bodies work and understand the world around us. Closer to E.T than to a human being.

Even though I’ll be championing inclusive design in this project, no process or product is perfect, and Sebastian Cabo Portugal de everything can’t be solved in one go, but I hope this will be a first step into considering more types of people than the normative to push our technologies to places we wouldn’t be able to imagine without them. I am fully aware that some of my solutions will be inclusive for visually impaired users, but probably not for people with other disabilities like motion restrictions; but as I said, due to limited time and resources I’ve decided to just focus on including this group, as well as users with full vision.

I’m also worried about creating a feeling of "using" people, going to blind associations and asking for their help might make them weary about my intentions and I’m still not fully sure how this process can be carried out, everything is still experimental. Although in our current process we are already the de- ciders on what goes into a design and what doesn’t - we don’t carry out everything the user says, but we should have a critical eye on what is important and what is not. Hopefully by always having this user group in mind, everything produced in this project will be inclusive for them no matter how much visuals it ends up having. The important thing should be that the visuals are not giving key information which if missed would make the user unable to use the product fully.

One of my objectives with this is to empower every user: to provide the visually impaired with heroes and games that represent who they are in a powerful way, having more diverse representations is quite MFA Thesis Project 2020 Thesis Project MFA a hot topic in the gaming and tech industry nowadays so I want to keep pushing that agenda forward; and to provide visual users with novel experiences which will not only be enjoyable for them, but also to train their senses and let them rediscover the power of their other (sometimes forgotten) senses whilst changing perceptions about what VI people can do, referencing the title of a paper I read during this project "I can do everything but see" [3]

So what are we going to do? are we going to continue doing things the same way we’ve always done it, or are we going to embrace the change others can bring to the process like we already did with different professionals in the design teams. My hope is that this thesis will point everyone into a new direction for the benefit of all. Innovation with a reason. E.T. the Extra-Terrestrial (1982)

- 10 - NON VISUALS Philosophical Considerations

Working on this project gave me a lot of questions to ponder about the future we are creating and where we want to be living in a near and far away future. The first thing that comes to mind while I develop my concepts is - "Am I doing right for my users? Is this the way to move forward?" As with everything, there is no simple yes or no answer and maybe "better" is not even the right question right now to ponder. We should have the freedom to explore areas and then see what kind of solutions can fit better or be developed from there, but sometimes we can’t know what we’re creating or how someone else will carry our findings forward into the future, like investigating nuclear energy to provide a better energy solution for Sebastian Cabo Portugal de the world and ending up creating nuclear bombs.

"These are my principles; if you don’t like them, I have others" - Groucho Marx

Is every step I make with this project taking us into a future like the one seen in The Matrix? That’s obviously not my goal, but I already see how including more of our senses into these types of technologies will create more confusion about what is real and what isn’t and will suck people into the promise of unbound possibilities and a life lived inside the machine. The more sense we include,, the harder it will be to distinguish between what’s real and what is digital and that worries me a bit. The line between what’s real or digital will blur even more, but that seems to be the direction we’re going anyways...

Berkeley argued that our only knowledge of this world is what comes through our senses, so will these technologies add or remove our sense of belonging to the planet we live in, will we care less about the environment because we have another environment to go to where everything is possible?

I’m not trying to provide a framework or even dare to propose "truths" about how these technologies

MFA Thesis Project 2020 Thesis Project MFA should be made, there’s not enough time for it. What I wish is to provide inspiration and examples of how to do things differently, so other designers can come up with their own conclusions, and opens up a conversation on the current design approach and how we want to develop further. That’s the best part of designers and the design industry, that we are constantly looking at ourselves and our process in the same way we do any problem we encounter and we keep iterating and refining it until it gets as close to perfect as it can, and when it doesn’t fulfil our goals anymore, we change it again. This might be the next step in its chain of iterations.

The Matrix (1999)

- 11 - NON VISUALS Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

2.0 BACKGROUND

- 12 - NON VISUALS The world we come from The world to come

We live in a visually dominant culture, so it’s normal that most of our en- If the world we live in is one of shooters and eye tracking to use our sight tertainment, including games, are heavily focused on the visual part of as a kind of desktop mouse, then the future is continuing down this line. things, and we have assumed that there is a hierarchy in the importance The biggest immediate advances in VR apart from bringing costs down; of our sense, namely sight, hearing, touching, taste and smell; but is this are better field of view/screen resolution, higher PPI, eye tracking for fo- true for every culture? veated rendering (rendering highly what you look at instead of the surroundings) and better/higher refresh rates - basically everything deal- A study on the matter revealed “that cultures which placed particular ing with improving the quality of the images seen in VR, still focused on Sebastian Cabo Portugal de value on their specialist musical heritage were able to communicate gaming. [Youtube-Future of VR by Virtual Insider] more efficiently on describing sounds, even when non-musicians were tested. Similarly, living in a culture that produces patterned pottery Why is VR so immersive? Because of the fact that you can move and look made people better able to talk about shapes.” [4] around naturally in perfect 3D "physical crouching behind cover feels completely natural because we act how we would in real life...the more Even in Aristotle’s time, the hierarchy was sight, hearing, smell, touch you have to do with your hands, the more immersive it is...(about periph- and taste; so with time, smell has been demoted to the bottom of the erals) Holding on to something you can use as a real life counterpart can list. So our presumption that vision and audio are more objective than really add to the experience and immersion" [Youtube - Why is VR so im- the other senses, which in turn help us learn and understand better the mersive by Virtual Insider] world than our other senses, which are more crude, is wrong. [5] Which is proof to the fact that we could live in a more inclusive world without We will move beyond games into more common uses for the main- losing any value for “the people in the norm”, and that we should start stream public, office or creative applications like the ones found nowa- focusing on our other senses, especially in terms of our modern digi- days on desktop or mobile computing. The examples by Keiichi Matsuda tal-led world if we don’t want to leave a huge amount of the population exemplify the vision of the future most of us have relating VR at work or out of the future, especially as the world is getting progressively older, AR on the street - a never-ending overflow of visual stimulus which you and with age comes the loss of senses. can’t look away from and which is not only collecting all the data you can about you, but is then using it to continuously sell things back to you. MFA Thesis Project 2020 Thesis Project MFA "The visually dominant culture emphasizes fashion and style over func- Looks like stressful times. tion; physical beauty over intelligence, culture and empathy; and shiny objects over Nature’s sensory cornucopia. The quality of sound, smell, Everything still feels screen based, like being inside a VR environment taste and touch are degraded by bad acoustics, noise pollution (includ- with an infinite amount of screens and navigating menus by physically ing audio devices); stale indoor air and water borne sanitation; indus- tapping digital buttons, or by waving our hands in thin air like in Minority trial food chain; and synthetic materials, particularly shoes and pave- Report (2002) ment which keep us from feeling the Earth under our feet." - Lawrence de Martin, What senses do we use most and least. [6] This is the kind of future I’m opposing to throughout this thesis and my proposals works on the exact opposite direction, more connectivity but also more calmness and everything controlled in a more humane way.

- 13 - NON VISUALS The world we live in The world to come Sebastian Cabo Portugal de

Half Life Alyx by Valve Merger video by Keiichi Matsuda MFA Thesis Project 2020 Thesis Project MFA

Beat Saber by Beat Games HYPER-REALITY video by Keiichi Matsuda

- 14 - NON VISUALS Focusing on Visually Impaired users

The power of the non-normative Also checking Reddit like r/blind and r/audiog- I’ve ever heard anyone reaching including people Making products accessible means that even ames was very useful because again people de- who have vision. “This just shows that people with people beyond target users can benefit. Disabili- scribe their experiences in detail and explain what disabilities can still play video games just as good ties have too often stayed hidden and taboo. I be- the best games for blind people are and why. as people without disabilities” [11] lieve we are entering a new age where disabilities can serve as a precursor to improving the world Fighting games: These are the most accessi- I’ve even read stories about VI people playing for others. [7] ble games for blind gamers due to their incredi- World of Warcraft, which is a super huge and com- Sebastian Cabo Portugal de ble sound design, every move has its own sounds plicated game with loads of mechanics and sys- The World Health Organization (WHO) estimates so you can memorize the combos through the tems through the use of add-ons. [12] that roughly 285 million people are visually im- sounds it creates, you can also know what the paired worldwide, of whom 39 million are blind [8] other player is doing through sounds. Gamers re- Accessibility in games has been a hot topic in the These numbers continue to increase as the global ally study the game by trying every single charac- developer community lately, as more and more population ages, as right now one in three people ter and memorizing all their moves which is quite companies start adding accessibility options or over the age of 65 have vision-reducing eye dis- an achievement in itself. more levels of difficulty in order to accommo- ease. [9] This implies that more and more users date more people with different kinds of abilities. will encounter these barriers in the future, espe- Diablo 3: This RPG game has narrated menus There’s still a long way to go but at least now this cially when dealing with technology. so you can learn where everything is positioned, topic is an issue developers are trying to tackle. and great voice acting for every character in the Companies like Microsoft have also worked on I’ve contacted Synskadades Riksförbund Väster- game, so you can follow the story. Also if you play accessible hardware which was a huge develop- botten to get expertise on visual impairment and with friends and you leave your character alone ment as now players with different abilities can also to see if I can get in contact with some visual- for a while it will automatically start following the map the controls to where it’s easier from them ly impaired gamers to help with the concept ide- rest of the group so it’s easy to stay all together. to reach and let them play any game available for ation stages and the testing too. The other way I To move around you can hear if your character any console. tried to get collaborators was through YouTubers. moves to your left or right and the audio changes MFA Thesis Project 2020 Thesis Project MFA Luckily there are quite a few blind YouTubers out a bit if you get stuck in a corner, there’s also audio there who create videos explaining their day to indicators if you’re going the right way in a quest. day lives and how to they deal with everything, as Still it’s very impressive to see someone playing well as some gamer channels in Twitch where they such a complex game without being able to see play fighting games, Diablo 3, Nintendo games it. [10] and audiogames and commenting on them just as any other streamer would, which was a great help Call of Duty: There has even been a blind player in these initial stages to get an overview of how that made it to level 28 in the zombie mini game in things are done nowadays. Call of Duty Black Ops 3, this is the highest round EXIT in Braille

- 15 - NON VISUALS Audiogames or be prepared to grind hard

If someone is completely blind then their best option is classic games like dungeons and drag- ons where all the action happens in their imaginations, or audiogames, which are games where the only feedback you get is through audio, no visuals, mainly a black screen or some abstract colourful screen. There’s several repositories for audiogames like audiogamehub.com/games/ which has archery, memory, labyrinth or blind cricket amongst others and they’re mostly free; or audiogame.store which has more commercial games. Sebastian Cabo Portugal de Audiogames play mainly with directionality, for example the Lworks shooting game Supershot, where you shoot sounds coming from the left, middle and right, and you have to shoot them using left, up and right arrow before they hit the ground. It starts easy but get fast the higher the level. There’s also a game by L-works called Lockpick where you’re a bank robber trying to find Mortal Kombat 11 the combination to open a safe before the police arrive and catch you. This game is a lot harder than Supershot, harder to understand. Lworks keep the same menu structure so once you’ve played one of their games you understand how the rest work. They always make it clear whether you’re moving or not through the appropriate sound design, like grass rustling in the Egg hunting game, where the egg make a ringing sound, getting louder if you’re near them, and the pitch is higher if it’s in front of you or lower if it’s behind as well as coming from one speaker or the other.

A game from the audiogame jam that has been highlighted by the community is EscapeBeat, where you have to escape a maze just by getting close to the walls around the game area and try to find the exit which is clearly demarcated by a specific sound. It’s a great example of spatial awareness through music, it had a great comedic voice over commenting on how well you’re doing and you even get to fight creatures by following an order of sounds like “Simon Says”. The Diablo 3

MFA Thesis Project 2020 Thesis Project MFA sounds get louder the closer you get to the walls and you hear a “thump” when you hit a wall, if you’re in the top right section of the map the sounds are higher pitch and the lower sections are lower pitch so you can know the direction you’re walking through those sound cues.

Even though there is a wide variety of topics in these games they feel quite similar between each other, they’re mostly about timing and choosing sides, like pressing left, up or down to hit or avoid some object coming at you; and they’re also quite simple and short, more like experiments than actual games.

The Blind Swordsman gameplay by caramida9 - 16 - NON VISUALS How are we adding senses to VR? Add more tech

Haptic Suits and Gloves or how to run around in VR There has been development of devices which give players the feeling of haptics inside a VR game through gloves like HaptX or Dexmo, which is like an exoskeleton glove around your hand that constraint your movements so when you pick something like an apple in VR, your hand stops at that point and feels like it’s also holding an apple, minus the feeling of weight. Sebastian Cabo Portugal de

There are also haptic suits in development like the Teslasuit which has HaptX Gloves Teslasuit 60 haptics points all around the body and lets you feel things like getting shot inside a game. It’s an interesting approach for shooters which could be expanded into other games but I feel it’s taking everything into a Ready Player One future which might not be the right path.

Finally there’s the Virtuix Omni, which is just one of the several products in development which enable users to run around physically but statically, in order to move around in VR. It’s a bulky product which is connected to your hips to keep you in place and either has an omnidi- rectional treadmill, or a slippery surface where you’d run in your socks.

To be honest I disagree with these approaches because I feel the more separate products you need to buy to get the full experience in VR, the less mainstream it will become, it can be a good approach for special- FEELREAL Virtuix Omni

MFA Thesis Project 2020 Thesis Project MFA ised VR arcades, but for the general public it lmits access and makes the technology more excluding even than before, this time in monetary terms.

On the other hand, big players like Facebook’s Oculus are going in an opposite direction by allowing hand tracking in their Oculus Quest system. Even though this approach will make everything even more intangible than with controllers, it also opens great possibilities: how could we use objects that we already own in our house to play different games, for example a banana could be a gun or it could be the joystick for an airplane. Oculus reveal Hand Tracking update Dexmo Gloves

- 17 - NON VISUALS Why read a book when you have audiobooks

Some people might think that using technologies like VR and AR without Meditation is another powerful experience that happens with your eyes visuals defeats their point, but I completely disagree. Not being able to see closed, as meditation is about clearing ones’ mind from any mental activity anything, just darkness, gives you a completely unique experience, a more and thought and concentrating for example on the breath, it’s about resting focused experience. This could create a new format/type of game inside your attention inwardly; keeping your eyes open can consequently distract VR or AR experiences. you from achieving the state of meditation.

For example listening to audiobooks, if you look at your surroundings, even There’s even restaurants like Dans le Noir, where you have dinner blindfold- Sebastian Cabo Portugal de if you’re not concentrating on them makes your imagination less vivid than ed and in the dark, which lets the customers focus just on the tastes, smells, if you had your eyes closed, this is because our visual senses normally over- sounds and tactile feelings natural in food...even the waiters are blind. power the rest of the senses even in situations where it shouldn’t be an in- tegral part of the experience. An audiobook is a completely different experience than reading a book, and some people even prefer it this way, it’s so engrained in us that listening to our parents read books is probably one of our earliest memories. I can be more immersive or interesting when different characters have different voices, and it opens the possibility of multitasking, although this probably reduces the concentration on the book and could be distracting. MFA Thesis Project 2020 Thesis Project MFA

Meditating at the beach Dans le Noir - 18 - NON VISUALS Inspiration

The company working hardest, maybe the only one who really is working on Microsoft Canetroller [14] this issue is Microsoft Research who have produced most of the concepts Microsoft Research’s Canetroller demonstrated how rendering virtual ob- some still in experimental mode but great inspiration nonetheless of what jects haptically, including simulating materials’ properties and textures, others are trying. could enable users who were completely blind to successfully navigate and understand virtual scenes when paired with a novel haptic controller that Microsoft Project Torino – Code Jumper [13] mimicked the interaction of a white cane. A great example in innovation for all through design for the visually impaired Sebastian Cabo Portugal de is in Project Torino by Microsoft Research. Coding has stayed the same ever Canetroller provides three types of feedback: (1) physical resistance gen- since it started, screen-based where you type the commands into the com- erated by a wearable programmable brake mechanism that physically im- puter to then run the code; but Project Torino thought of a new way to teach pedes the controller when the virtual cane comes in contact with a virtual coding basics to visually impaired students – Code Jumper, a physical pro- object; (2) vibrotactile feedback that simulates the vibrations when a cane gramming language designed to be inclusive for children with all ranges of hits an object or touches and drags across various surfaces; and (3) spatial vision. 3D auditory feedback simulating the sound of real-world cane interactions [15] “What I like about Project Torino is that you can actually touch, physically, the program,” said Victoria, 14. Touching a program can seem inconceivable, but Microsoft Seeing VR with code jumper kids could immediately experiment and build programs. Microsoft Research found a great solution to solve many problem for VI users with the focus on the visual side of things. Not a bad option and very use- The way they included experts into the process, in this case the VI students, ful for most of the population - so I’d want to try a different approach, one team members began meeting regularly with a small group of kids and get- that completely forgets about making the visuals easier to understand. ting their ideas. Based on the kids’ feedback and ideas, they switched to bigger plastic shapes that fit easily into kids’ hands, and they created surfaces Playing through sounds by Andreas Refsgaard [16] the kids could rub or squeeze in order to recognize and interact with them. Digital artist and designer Andreas Refsgaard created a program which Through this approach they started seeing the technology from the kids’ would enable gamers to play the old school game Wolfenstein 3D 1992 just MFA Thesis Project 2020 Thesis Project MFA ( ) perspective. For example, the kids with some vision benefited from bright, using sounds like claps to open doors, whistles to move forward contrasting colors. They also found that kids like to work together, guiding each other’s hands, so they built the pods to be about the size of two kids’ Audio in AR Space by Zach Lieberman [17] hands. [20] It really opened the eyes of the researchers to different per- Artist and educator Zach Lieberman uploaded a sketch created in open- spectives, different ways to see the world. frameworks which mapped audio in 3D space and this was shown through the sound waves revolved into spiky cylinders. If you moved towards the line “These students are taking out brightly colored plastic pods, connecting of audio and went through it you’d hear the message backwards, and once them together with thick white wires and then adjusting the pod’s buttons you followed the line in the direction it was created you’d hear the saved au- and knobs. These physical components will be used to create computer pro- dio message or sound saved in the space it was created. Mezmerizing stuff. grams that can tell stories, make music and even crack jokes.”

- 19 - NON VISUALS Inspiration from Microsoft Research Sebastian Cabo Portugal de

Project Torino - Code Jumper Canetroller MFA Thesis Project 2020 Thesis Project MFA

"I just felt very independent, and I liked that" said Daniel "To just do coding, it was a fun experience" [13]

Seeing VR

- 20 - NON VISUALS Games, an excuse to improve the whole technology

Games are the perfect ground for exploration and experimentation as it sic, and even a full robot suit connected through strings so the movements has no boundaries of what you can do with it, as long as it’s fun and excit- the player makes get translated into the moves the robot in-game makes. ing. Compared to working with utilitarian applications which are more con- I feel this approach would be great for any kind of player but especially blind strained because you have an actual problem to solve, games are so differ- players as they would have a much more tactile feel to the controls and the ent between each other while still falling under the same category of game, games they are playing, for example while riding a bike and turning to one there’s much more diversity in games than even in movies or music, and side or the other with the vehicle kit. This approach has also been introduced every year, even though it feels like everything is invented, new ways to play into VR/MR scenarios to give it a new tactility that would be impossible to Sebastian Cabo Portugal de keep coming up. get with the available controllers.

Games are also important for another reason, they improve people’s lives even though most of the time they’re underestimated in the entertainment and design industries.

"Games do so much for me in managing my anxiety on the metro and stress about nearly getting run over and just general ability to function… I really wanted to play games. Games help me even out drastic emotions by taking away my attention for a little bit until I’m not as activated." [18]

But this is just the starting point, ideally everything that is learnt from this project would be then translated into other apps or even full operating systems in VR or AR. How to guide someone through instructions, move around in the environment and attract their attention when needed are all things that games do really well. Especially due to the fact that games have in- corporated 3D environments much more prominently than apps, which are MFA Thesis Project 2020 Thesis Project MFA mostly 2D screens. How would the internet look if it was more inspired by gaming rather than by classic Human Computer Interaction?

Nintendo is one of the biggest players in this area, in every new game they start with gameplay first, and then they create the characters and stories around that to keep everything consistent and give an extra meaning to the gameplay. Their latest innovation apart from the Switch console was the Nintendo Labo, where they concentrated on creating more tactile controllers through a set of cardboard pieces which you can f old into shape and put together without any glue or screws to make your games more immersive and fun, for example by creating fishing rods, a small keyboard to create mu- Nintendo Labo Variety Kit and VR Kit - 21 - NON VISUALS Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

3.0 APPROACH + METHOD

- 22 - NON VISUALS Insights from Background Research

What I call Virtual realms is known as The virtuality continuum: which is It has been established many times throughout this report that "VISION a continuous scale ranging between the completely virtual, a virtuality, rules everything around me" to paraphrase the Wu Tang Clan, but the and the completely real, reality. The reality–virtuality continuum therefore problem is not only shown in the products released, but the materials and encompasses all possible variations and compositions of real and virtual design methods used that create these products; and the visual culture in objects... The area between the two extremes, where both the real and general that has gained importance throughout the decades, from audio the virtual are mixed, is called mixed reality [19] in the radio to video with TV, and from reading posts to watching video in the internet era, flooding us with content that we can only interact with Sebastian Cabo Portugal de Right now we are still separating them through screens where we choose through our eyes. when to enter the digital world or leave it, even though it’s getting harder and harder due to the addictive design of the apps we use everyday. But Because the design materials are heavily focused on vision too, those will it will also enter our lives through our Smart Homes or Internet of Things have to be modified to fit a non visual approach, how to brainstorm, save (IoT); which is why I’ll also take it into account in this triad of technologies sketches and develop movement or sounds were some of the expected which could benefit from using the same modalities like sound, move- challenges. ment,etc. which I’ll be exploring. MFA Thesis Project 2020 Thesis Project MFA

- 23 - NON VISUALS Flip the norms - “make the known unknown”

From the Background section it’s clear there is a whole research area that These are the key factors I’ll go against in my own Design Principles to use is being barely exploited and which could be used to great benefit for the as a guideline and not loose track inside the huge worlds of these techno- technologies we deal with in this project. Focusing on the senses not only logical advances and realities. creates innovation, but this innovation is more inclusive, and even if later on you would add visuals, it would still be accessible enough for all.

The way this project is being approached is through finding a series of Sebastian Cabo Portugal de what we believe is common knowledge about VR technologies and flip it on its head - as an experiment to see what could be produced through opposing values than the current ones. Like the quote that opened this report, to make the known unknown and understand it deeper due to having to question it in other terms. To see if a different future can be achieved.

The users we design for are based on an incomplete form of ourselves as humanbeings illustrated beneath. Huge eyes, small ears and mouth to talk, no nose or sense of taste, and just one or a couple of fingers to interact with most of the tech products nowadays, no body at all, just floating ghosts.

Not only this will be challenged but also the current user centered design methods, as there is much talk about it being user-centered but doesn’t seem to see its users as human beings with physical and other sensory capabilities more than a brain.

MFA Thesis Project 2020 Thesis Project MFA “Everything in my world is 3 dimensional, everything I touch or experience, so it’s very difficult for me to imagine putting a 3D thing onto a flat piece of paper...like it doesn’t make sense to me” - quote from Tommy Edison in his Youtube video - Can Blind People Draw? Leaving behind: We could now include: - Visually Impaired Users - Visually Impaired Users or other This new focus will have to be shown to users who like the one quoted - People who think the tech is sensory capabilities above might find inconcievable that something can be translated from too complex for them - People who know how to interact 3D into a 2D shape. So the post-its will have to go, in exchange for audio - People who prefer to use their with day to day objections and actions and physical sketches which would be recorded through video, and most bodies than just their minds - - Other types of learners that are importantly as experience prototypes one would be able to try and feel manual learners not just visual - manual or emotional themselves, because some things have to be felt to be understood, no learners amount of talking would equate it. - 24 - NON VISUALS Design Principles Perfect timing

Strengthen other senses to overcome reliance on vision The design principles came from grabbing everything I disagreed with in the - Sound, touch, movement (body & space), balance, passing of time previous sections and the exclusive approaches that had been tried before - Not just a problem for the blind community, but for everyone to provide a new framework which would produce innovative work and not walk on treaded ground. These DESIGN PRINCIPLES were used to guide Make tech inclusive from its inception me through the project and inform my methodology and the experiments to come. - VR has affordances that uses the natural capabilities of VI users Sebastian Cabo Portugal de - Use less tech devices by using own objects/body as input People think tech like VR is more mature than it actually is, defending this opinion using the Gartner Hype Cycle. [20] My approach is to bring it back Unpack the technology to discover the affordances of the tech to a new starting point to open it up and reimagine how things could be done - By exploring form an expression of non-visuals in a different way while still be considered VR because it’s already getting - Breaking the boundaries of what we think is VR stale. I’m proposing that this is the perfect time that we should be exploring - Thinking differently through new design methods everything about these technologies in crazyand opposite ways to see and - Designing for a rich sensory experience expand what the technology really can do. MFA Thesis Project 2020 Thesis Project MFA vs what I’ll be focusing on

REFERENCE https://www.gartner.com/en/research/ methodologies/gartner-hype-cycle

- 25 - NON VISUALS Research through design - trying things out

After the desktop research weeks were done, the next step was to go deep Focusing mainly on sound and touch, as well as balance, movement into understanding our senses in more general terms, our connections with and the feeling of time passing. These would be easier to implement and our bodies and between movement and our mind, as well as trying things less disruptive as a whole, imagine users wearing a mask that controls the out there firsthand like the audiogames which were described previous- air surrounding them to give a bit more immersion or to "smell" the internet, ly, and playing with VR and AR apps to understand the state of the current or electrodes in their tongue so they can taste virtual plates, not good. technology. Sebastian Cabo Portugal de When you start designing for something other than visual, the design materials we usually use suddenly seem very limiting, or it downright doesn’t work at all, this means that we have to explore new ways of creating prototypes for physicality and movement and find ways to introduce sound into the process. The importance of Experience Prototyping [21] also shows up in this case because it’s very hard to explain these types of experiences through talking or through drawing although storyboards and journey mapping are still useful.

These questions are relevant because the more we move away from flat screens and into 3D environments, the more we will all have to make use of these new materials for idea generation and prototyping. Especially in a project like this one where maybe you can’t even show a quick sketch to your target user because they won’t be able to see it, so what do we do when we can’t use sketches and post its? What’s the next step in the evolution of the design process? MFA Thesis Project 2020 Thesis Project MFA For this project I decided to leave out some of our main senses, apart from vision, because I feel they wouldn’t be as interesting or plausible to play with in the future; for example, smell. There’s products being developed to include smell in VR like FEELREAL, and in 1999 there was even a quite successful company, in terms of earning investors, called iSmell which tried to "bring the sense of smell to your computer and the internet" but obviously flopped as soon as everyone understood they were trying to solve a non-existing problem, the tech was nearly there, but the need wasn’t. Taste, tem- perature and pain are the other senses left out on purpose.

- 26 - NON VISUALS Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

4.0 MATERIAL EXPLORATION

- 27 - NON VISUALS Working in Sprints based on Design Principles Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

Working in Sprints is very common in digital product and games development, so a similar approach seemed the most reasonable, also to produce as many experiments and learning as possible to be able to input them into the next loop/sprint. The design Principles would always be used as a guideline through the fog of uncharted territory, to make sure the direction kept the values it started with and the objective it was looking for could be achieved.

The concepts came out since early in the process as concept embryos which would be fed all the new insights gathered through each loop in order to become fully formed examples which could be then used as an example of both the methodology and new ways to imagine VR.

- 28 - NON VISUALS How a Loop looks from the inside Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

This was the experiential part of the process, 1st hand trying tech added to interviews with experts and past research would inform what experiments to conduct, these would then be analysed to get out the key findings from each and introduce these into the concept embryos, as many insights would fit in more than one concept/provocation. Everything added and learnt would be again sent into the next loop for validation and to gather further insights.

- 29 - NON VISUALS My Activities These activities aren’t showed in chronological order in the report but grouped by method to make things clearer to the reader, whilst during the process it was much more chaotic and mixed, interviews happening before or after experiments and workshops could be used to validate them and then inform the next experiment/workshop for the next loop. In all honesty the interviews were as insightful as the experiments and the workshops, it really helped understand how people percieve things differently and to know what others inside this tech world thought of the provocations being proposed, if the old guard didn’t seem too convinced then I was moving in the right direction because nothing that is really innovative happens unquestioned by the previous generations. Sebastian Cabo Portugal de

Interviews Experiments Workshops

Interviews are common methods in user cen- The experiments were one of the key parts of Workshops were also like experiments but in- tered design, although the current way of do- the methodology, they started as improvisa- volved maybe more people at the same time or ing things is to include people that are relevant tions to see what directions they could take, just went deeper into development mode for a to the issue at hand i.e. the users, or sociologist but once it started opening some interesting specific idea and physicologists who have a broader view of areas linked to the rest of the research, they any issues the users could have. started being more focused on things which They would happen in two phases, when pos- could be used in the concepts. sible, where the first workshop would be more My approach will not only include people from improvised just to check if the idea would work the game and tech development worlds, like In general the experiments aims were to and something could come out of it; and the game developers, MR experts and audio entry and give me and the project an under- next workshop would be more concentrated gineers; but also people currently considered standing about our bodies and our senses, - with a clearer objective - even though it was MFA Thesis Project 2020 Thesis Project MFA outside of it, but which I feel should and will be what it can do well or not, and how it can be still all about improvisation and brainstorming. included more prominently during the process tricked. of these immersive technologies: people con- Some workshops were moreinsightful than nected to the human body and movement like These experiments were all very lo-fi and others due to a lack ofproper boundaries for physiotherapists and dancers as well as our that was on purpose to see if it was possible the concepts to come out shaped in a use- key “extreme“ user - Visually Impaired gamers to mimic VR or AR experiences without hav- ful form or because a second part couldn’t to have a holistic inclusive view since the start. ing to go into Unity and code a game or app. be done after we all had to go into quarantine Just using speakers across a room or my own mode. The last one was tried in Zoom and it Include experts in the tech side of things voice as a narrator in improv theatre style ac- half-worked due to the limitations of the video and the human side of things. tions to test and validate ideas, or to scratch calling platform. them alltogether.

- 30 - NON VISUALS Interview - Luca Contato from Rising Pixel

Rising Pixel is a video game company based in Gran Canaria which I had are currently in the market, if you can move around a space with headphones the chance to meet in a game jam they organised a couple of years ago. The that can track the space then this is also a Virtual reality, VR doesn’t mean game jam they organised this year was an accessibility one and they have Visual Reality. also done what I believe is the only Audio Game Jam done in the world. They make audiogames for commercial purposes and they are quite successful, He also thought some of my ideas were a bit dangerous, like controlling an although “success in the audiogame world means selling 5,000 units”. They AR Operating System by using objects you have in your house or the location

have also explored many ways of controlling games like through voice or based AR game which is only audio, because it would not let them hear their Sebastian Cabo Portugal de through a midi controller. surroundings which is their main sense to know what’s going on around. The solution we could find was using Bone Conduction headphones. I thought they would be great people to talk to as they have plenty of expertise working with games specifically made for blind people and involving blind game testers into their process. He was interested in the concept buy also quite averse to it from a commercially viable point of view, which I explained was not my objective in this project as I have the chance to explore something which could or could not be viable in the future but that’s exactly what I want to prove. The main points I got out of the conversation with him were:

“When we go to a games conference with our audiogames, no one is interested because they say I can see, so this game is not for me/ not interesting”- Visuals are so engrained into what we expect from games that the lack of it is a disadvantage, and it’s true there’s a lot of people who care about the graphics most, but in general I think if it has fun gameplay then graphics

MFA Thesis Project 2020 Thesis Project MFA come second.

“No blind person would buy a VR set”/”Why do blind people need VR if they can’t see?” – I think this is a valid point from his side as he is thinking commercially viable products, but I also believe he has a constrained view about what VR is and could be. Blind people might never buy a VR set because there’s nothing for them but if it was more inclusive maybe they would; it’s like a snake biting its own tail – no one makes anything inclusive, so the blind don’t buy it cause there’s nothing for them, and there’s nothing for them because blind people don’t buy it... Also VR doesn’t have to be like the sets that Blind Arena Tournament - blind multiplayer game developed by Rising Pixel

- 31 - NON VISUALS Interview - Christy Smith, gamer & reviewer

Christy Smith lives in New York and her side job is as an accessibility game a mat to land on or there’s a barrel to jump over you need to space things reviewer for the website caniplaythat.com, although she started two months appropriately. All beginners (no matter their vision) use chalk marks to know ago she’s done several reviews on Nintendo games already. She said she has where the jump should start or your hands should hit for a cartwheel. a lot of vision in the spectrum of visual impairment. We had a really long a fruitful conversation where we touched on every concept I was planning on We spoke about how there’s blind gamers who memorize the button inputs to making and gave me invaluable feedback and directions for them. get through the menus and spend hours, weeks, months of their lives learn-

ing videogames back to front so they can play them without vision at all “I Sebastian Cabo Portugal de She mainly plays with the Nintendo Switch “because I can put it really close don’t have the patience or desire to do that though - Grinding for the sake of to my face and carry it around in my commute”. Mario Odyssey is her favour- being able to grind”. When she gives recommendations for what visual skills ite game because of the exploration part “It makes the exploration a lot more someone needs she is intentionally very generous because she doesn’t want accessible than most games do, and more accessible than the real world, it’s people to think they can’t play this game because some lady online told them all on the screen and there’s no danger in the same way, freedom of move- they can’t, but she doesn’t want people to have to memorize these sequenc- ment that’s not hindered – you can run and jump and swim and you can go to es to go through menus for example, because that is not the standard game- a random alley which you wouldn’t do in real life”; in terms of shooters, Spla- play that she would like. Moving through a menu is not the game. “When I toon was her favourite. do reviews I talk about a very normative way of playing the video games, not talking about people who find workarounds very exhaustively because I’m a In the realm of sports, she does ballet and artistic gymnastics, and also ice more casual gamer, I play on my commute so I’m thinking of children or very skating but not as much these days. She did these sports since she was re- casual gamers who just wanna pick this game up and play” ally little and her vision has gotten worst over time, so when she was young- er she could learn the technique of ballet and get more physical corrections She has tried some travelling carnival VR game like shooting ducks, and a “they’d come and move my foot or stand directly in front of me and show me hand gliding one “and the graphics looked like 10 years ago, still fun though”. how my feet were supposed to be.” She also did a bit of AR when reviewing the Harry Potter location-based game but didn’t like it because “you’d have to hold your phone up to things

MFA Thesis Project 2020 Thesis Project MFA She voluntarily chose not to do parts of gymnastics like the vault, “I’m not and it didn’t seem smart enough to realise when it was putting objects on running full speed at a stationary metal thing” or the balance beam “but like top of people, so it looked like I was taking pictures of people which made with the floor…the floor is not moving, you don’t need to see the floor to know me violently uncomfortable” the other issue is that your phone dies instantly where it is, and we have all sorts of mats, so it can be just as safe for a blind which is a great accessibility issue. You have to move through actual streets person as it would be for a sighted person”, “mostly what we do as adults is with actual dangers and you have to focus on your phone the whole time, just tumbling; cartwheels, handsprings, summersaults in mid-air… it’s body so even if you can normally navigate the space you’re staring at your phone movement and you don’t really need to see. On the trampoline track you just “even if I’m not playing and someone else is they are not paying attention so need to see the general direction of where you’re going and the muscle con- they become hazards”. She thinks it’s an innovation in gaming but should be trol to stay in a relatively straight line then you’re fine, you also get the feed- better controlled or provide spaces to play these games instead of going all back from the spring in the trampoline” She comments on how if you need over the place.

- 32 - NON VISUALS Interview - Christy Smith (cont.)

Her feedback on my location based game idea was that it would be better to the framing is really important” She thought it would be a really useful video have more physical feedback, like a concept she knows where a GPS guides game that is helping your brain to open up, be more aware and have different you through vibration on your shoes because through sound it’s harder, bone types of thoughts. conduction headphones would be better than earbuds that would block out too many environmental sounds. “Even people who are sighted don’t realise She commented on the idea that your own house would be a dangerous how much they rely on sound when moving around or when in a crowd to place to play or move around in a videogame like Luca Contato had pointed

know where people are, when they have earbuds they walk terrible” out. “No one would have any issues navigating their own home, if you have a Sebastian Cabo Portugal de setting of not going downstairs” She said she really wished she could play “You always have one hand that’s occupied either by your cane or your dog the Harry Potter game inside her apartment or building “I’m still walking!” when walking around, so if you also need your GPS, which you do, then that occupies the other hand, so what happens when you reach a crosswalk and She liked the idea of using your own objects from home to play games but you need to push the button, so if the shoes could take the place of the phone also commented on a game called RingFit Adventure which will be further to take the place of the phone and give you another hand that would be very explained in another section of the report, because the controller in this useful” game is very flexible and could be used in many different ways for different games whilst providing real physical exercise when in use. As a dancer she really liked the idea of the music maker through movement - “That would be really awesome, there’s tons of styles of movement, fluidly, She had a contrasting perspective with Luca from Rising Pixel and to be fair sharply, with a good sense of beat or an erratic and messy way; so that would I trust her judgment more than his because she has lived with this situation be really interesting. It wouldn’t work for everyone but it would still be a really all her life, and in her mind all my concepts were really positive, possible and cool way to explore for kids and adults cause and effect”. She did music pro- interesting which gave me a lot of energy and validation to continue forward fessionally for a while, and the sense of fun gets lost when you work like this with them. “so if there’s a way that the music isn’t necessarily the primary focus or even for dancers the dance wouldn’t be the primary, it’s just an exploration, then In terms of including visually impaired people in the design process she gave

MFA Thesis Project 2020 Thesis Project MFA that would be a lot more fun for everyone and just a creative outlet” me some pointers on places where I could find people interested in helping like guide dog schools where there’s a lot of downtime and different groups She doesn’t like ideas like the existing game Blind VR or some experiences come to do activities to keep the people entertained, or even in schools for like world cane awareness day where they blindfold people and make them blind kids “Kids would love it if you went there and let them play videogames use a cane and they’re like “isn’t this the hardest thing ever, don’t you feel for an hour”, as well as contacting people on Twitch. sorry for blind people” and she feels it’s weird to call out the differences like that, because for them it doesn’t work like that. On the other hand, she does Finally she commented on how Xbox has the best accessibility features in- like my approach of the non-visual VR game because it encourages people grained in the system and also created the adaptive controller that was a to expand their capabilities and learn a new skill “really empower them huge success because it solved issues for all types of different users “they and not making seem like blind people are completely incapable” “no- care about this more” but the games are too violent and uninteresting for her. tice how you can focus on your other senses and still get a lot of information,

- 33 - NON VISUALS Interview - M Eifler from Microsoft Research

M Eifler is an XR designer and Senior Researcher at Microsoft Research in One example they gave me was spatial programming in the game rec room San Francisco, I found about them when I read an article titled “You’re do- in VR, an in-world spatial, embodied, dataflow programming language. [23] ing mixed reality wrong” on Medium. Their approach was exactly the kind Even though this session was a quick 30-minute chat, a lot of interesting in- of thing I was looking for, instead of adding screens into a 3D environment, sights came about as we were talking about possibilities and philosophical they’re starting to use the potential of a 3D environment to break software approaches to VR. down into SPATIAL INTERACTIONS.

One combined reality Sebastian Cabo Portugal de They call this leaving the desktop metaphor for a studio metaphor; by break- In Microsoft Research they also see how VR/AR/IoT will be all intertwined ing down the convention that we should be sitting down and looking at a spe- into the same environment in the future, and how apps will be switching from cific screen in a specific location (fixed viewpoint computing) and making a one mode to another depending on the capabilities you need at a specific new way of working with computers and our bodies: an all-brain, all-body, moment. “Right now they’re hard to mix because of hardware, everyone is all-bodies tool. [22] going their own way”

"We can steer computation, one of the most powerful tool humans have New starting point ever invented, toward a future in which it engages with and empowers They never start development from an audio or movement point of view but a wider range of human capacities allowing us to create entirely new they see the value of my approach for exploration. Even though I haven’t forms of thought and action we cannot currently imagine." [22] found much experimentation in these areas, they say “there’s a lot of exploration being done behind closed doors even though we’re still using past par- When I asked them about not knowing if what I was developing was better adigms” than we currently have they responded “Better is not the right question at this moment, this is a moment for exploration and the applications for it will How to move forward with the physical apps concept come up later.” After explaining some of my provocations they had an awesome response to this by pushing me to think beyond just games, “Can you use the same mo-

MFA Thesis Project 2020 Thesis Project MFA “The future of these technologies will go both ways, more haptic/physical, dality/tool in a different context?” – from creative mode to communication and also more Minority Report/waving hands in space” Which is something I mode or even utility mode, to see if the insights I’m gaining can be univer- completely agree with. People like to think of one unified future where there sal or can only be used for creative applications and games. This is why the is one technology used in one way that feels like the only in existence; but in project turned from game development to the develpoment of applica- reality it never happens that way, there’s always a huge variety of technolo- tions and an OS. gies and approaches so we can’t say the future will be one way or another but all the possibilities at the same time.

- 34 - NON VISUALS

M Eifler Studio Metaphor [22]

Desktop metaphor is Fixed Viewpoint Computing. It keeps our bodies locked in fixed, forward facing, industri- alized relationships to screens and keyboards and mice and chairs and desks and on and on. Even our precious phone, which gets to come with freedom of movement, is still haunted by the hunched back of fixed viewpoint.

By creating a system which is more based on a studio metaphor than a desktop metaphor you can sit or lay down in any way you want and still be able to traverse the virtual environment, it is designed to offload interaction

demands from the prefrontal cortex onto the motor cortex and hippocampus thus distributing load more evenly Sebastian Cabo Portugal de across brain and body systems making the use of computation tools less fatiguing over all. This can be done by creating a safe, padded space outside the headset, for example by covering the floor with padded mats and remove chairs - Variable Viewpoint Computing.

The images below show the unlimited variety of positions which could be used to be more comfortable by using VR and breaking the convention of having to sit on a chair in front of a screen to do work. To the left a portrait of M Eifler. MFA Thesis Project 2020 Thesis Project MFA

- 35 - NON VISUALS

M Eifler 3 scales of Interaction inside VR [22] An example project is this podcast app which has three distinct levels/scales of interaction: The wearable scale is small and usually kept on the avatar. It gives you glance-level access to information, quick access storage; The holdable scale is comprised of tool-sized objects, body scale interactions, and the highest level of interactivity; and the habitable scale which comprises the functionalities sorted into the navigable environment. Thinking of UX as an ecosystem.

Obviously this solution is all visual, which doesn’t make it less interesting, as there still is movement in space but it would be cool to develop more how this space could sound or feel with haptics,or even be blended more into the real world as AR. This was the main inspiration for Home OS and it’s 3 distinct levels of interaction. Sebastian Cabo Portugal de

Images from Walkthrough of Malleable-scale VR podcast app. Link to full video - https://www.youtube.com/watch?v=o4vRumkDKG4

Door behind Holdable scale to exit Habitable scale Habitable scale has same form factor as Holdable scale Holdable scale: timeline, volume, pocasts as sculptural icons MFA Thesis Project 2020 Thesis Project MFA

Wearable scale means you can holdit and move it around You can add new podcast icons to the Holdable scale Wearable scale ring just used to turn on/off audio

- 36 - NON VISUALS Interview - Arianna Ortelli, CFO of Novis

Luca Contato from Rising Pixel recommended me to research a project they also want it to be universal and not branded for only blind gamers, the called the blind console, a device that has been a year in development, in col- product name and the fact that this is its own device will make it harder for it laboration with different associations for the blind in Italy. Snowballing really to become mainstream outside the blind community. This is the main reason helped me find the right people to collaborate with. why I chose to use technologies like VR and AR in its current form, because with this approach a lot of gamers would be able to find these games in the “The idea was to make a game 100% accessible for blind and visually im- environment they already own and might have a higher chance of trying

paired people, not using the screen but concentrating the VR on the audio them than if they had to buy a complete set of devices to play a limited num- Sebastian Cabo Portugal de and the tactile feedback” The device is just some headphones and a hand- ber of games. It’s also the reason why I want to introduce everyday objects as held controller that connects to your smartphone for processing power “Us- controllers or even one’s own movements, they also see this as a very crucial ing 3D audio and stereophonic sounds to introduce 3D space in the audio to point “It’s important to have a solution that’s as inclusive as possible without have a more complex game and a more intense reality”. having to buy other stuff.”

They are still testing the device because “it’s hard for a blind user to use many The blind console is still in development and there is only one game out objects at the same time, the important part is to give them a controller that which is “a simple Table Tennis game to allow people to understand the con- could help them to not use the phone in the same way they use it, they will be cept, but there’s not many features in this game”, although she assures me able to forget about the phone, which would be in their pocket, and use only there’s 3 more games in development for the beginning of 2021 and by the the joystick” This is their proposition, although they’re still testing the ap- end they could have 5-10 games (depending on the funding they gain). These proach and it could change in the future. “The hardware part is an important will be more complex than ping-pong, like “a complex tennis game with the value because you can work on the tactile feedback and the vibration on the possibility to play with two players”. They just secured funding to produce 50 end and makes it more interesting” units to test the market in three or four months with the associations and test the market by selling the units. The conversation went super well because we both felt we were exactly on the same wavelength regarding the issue at hand, we had the same vi- "There is a software part and a hardware part so it’s really complex, there is a

MFA Thesis Project 2020 Thesis Project MFA sion, we wanted to create technologies that are fully inclusive and to enable lot of R&D that we have to develop" crossplay between VI and visual players. “No one is including VI players in their experiences or they make an experience only for the blind so they don’t Their main collaborator is a 17-year-old VI gamer called Dennis “very pas- have the possibility to play with other people, even sighted players, this is sionate about audio games” who is helping with the game ideation stages as our value proposition”. It would be a great advantage for everyone to include well as testing “we do brainstorming for the other games but we listen a lot to them throughout the design and development process as opposed to just for what he thinks”. It’s very important for them to have the co-design approach accessibility or not at all. and “have direct contact with users” since the very beginning of the ideation; because even though they can try and understand their position, it’s not A difference in approach the same, and VI gamers have “different sensibilities” to blindfolded gamers The difference between their approach and mine is that I tried to move away who are used to seeing “A perfect game for a blind person doesn’t mean is from making something that is distinctively for blind people; and even though perfect for sighted people”

- 37 - NON VISUALS Interview - Arianna Ortelli, CFO of Novis (cont.)

“We are focused on families, so we want to make a game for a father and his son and it should be challenging, interesting and thrilling for both”

A big issue that popped up was how hard it is to make it so visual people understand this is also for them. The key for this is “the right concept with the right tutorial and customizable for both types of customers, but it’s not so

easy. it has to be challenging for everyone, and it’s a new way of playing for Sebastian Cabo Portugal de everyone”

“For sighted users it’s simpler to close your eyes because you can focus on the sound. In the future, a mask could be included and used to close your eyes while playing”

“It’s useful for people to try the idea out physically, even if they don’t know nothing about the idea” which is an approach that I’ve been trying personally and will continue to try in the development of this project (although it was Usert trying Ping Pong game in Blind Console very hard to do online due to quarantine).

She also pointed out how hard it is to collaborate with associations for the blind “They ask why are you here? They’re weary about your objectives/interest”. We include these collectives so little in our lives that when you try and do something with them they intuitively don’t trust you. “It’s better to find individuals who are interested, to try and think about the game or test the

MFA Thesis Project 2020 Thesis Project MFA idea, and when the idea is more developed, then you can bring it to the associations”. This is something I’ve personally seen, as the hardest challenge in this project has been finding partners to collaborate with within these types of associations.

Right now they are developing their own SDK but they would like to use Unity or other engines so other developers can make their own games for the console, they even offered me the chance to make my game for their hardware in the future! We will also try to have another session with the whole team to try and ideate or develop further some of the concepts and even some design Arianna teaching a player how to use the controller guidelines.

- 38 - NON VISUALS Interview - Enrique Sanchez, VR and game designer

One project made by a close friend of mine, Enrique, during an accessibility The hardest part of the project was to create a game that was challenging for game jam in Las Palmas de Gran Canaria was my first and main inspiration to everyone in equal parts, “some very competitive gamers thought the game start this project. This project was the first thing I ever saw tackling this area was super easy and finished it in no time” whilst others thought it was too of VR for the blind and it opened my eyes to an area I had never thought about confusing and difficult from both sighted and VI players side. This is a com- or questioned before. Trying out his game was a great experience for me and mon issue with games, that’s why most of them have difficulty levels, so it’s already showed me some of the limitations inherent in having a completely something I won’t address in my games due to lack of time. It should be chal-

dark VR environment. lenging enough, and a new way of playing for everyone – which is something Sebastian Cabo Portugal de I feel I can make happen. The game jam was carried out in 48 hours and in a group of 3 so it’s obvious that the product is a short experience, very experimental and not very pol- ished, but the feedback he got from sighted and VI players is still super useful for this project and a good first learning step for me to continue it forward and make the version 2.0.

He created a small VR game with no visuals where robots would come from 360 degrees and you would have to spot them and shoot them before they reach you. Talking to him was great feedback as right now I’m still not at the point of making a working prototype in Unity and wouldn’t be able to test it with users.

“The robots would all make the same sounds which made it confusing when several of them were around you at the same time, hard to know where the Enrique teaching a blind user how to play sounds where coming from” So it’s better to give them different sounds to

MFA Thesis Project 2020 Thesis Project MFA make them easier to distinguish. “They sounds also got louder the closer the robots were to you and would make a feedback sound if they got to you and hit you” The different sounds from the previous point could also be done by giving the robots different sounds depending on if they’re about to attack, if they’re on their way and if they’re about to hit you.

He used 3D stereophonic sound and made it bounce different depending if the player was inside the metal hatch where the game starts or when they left to the outdoors to make it clear for the player they were moving in different scenes/levels.

Images from TV Canaria about an Audio Game Jam done in Gran Canaria - 39 - NON VISUALS

Interview - Pablo Godoy, Physiotherapist Interview - Nelson Sanchez, Physiotherapist One area which could be very interesting to explore and where these ad- The way to develop this app should start simple, for example, trying with foot vances in senses can be included is in physiotherapy, not to remove the mobility which if it goes wrong it won’t do as much damage as a neck injury. role of the therapist, but as a compliment which can be carried out at home. The process would be a long one with several group of testers which would “Many parents don’t know how to treat it once they’re home”, especially now come regularly over the months to see the improvements. So even though that video consultation is becoming a highly used option for many people this idea has a lot of potential, it falls out of the timeframe of this thesis. during the quarantine due to COVID-19. Seeing someone make a move and trying to copy it doesn’t work in his line of Dementia patients or people with Parkinson’s Disease could greatly benefit work because this is “mirroring”, the patient would be copying and the brain Sebastian Cabo Portugal de from solutions which enable them to continue with rehabilitation more than is not teaching itself how to make the move they need to do. He only gives just when they see a therapist “There’s thousands of opportunities regard- instructions through speech, not by grabbing the patients arm and moving ing this theme”. Making it a game can also be very useful for families with it himself, even the fact of removing vision would help because the patient kids with cerebral palsy, it takes their mind off the fact that they’re receiving concentrates more on the move and that feeling – which is all points in favour treatment and can even turn into an enjoyable experience. The key to this is for the tech approach I’m exploring. that it should be easy access and understandable not only for the patient, but also for the parents/family members for them to use it regularly. We spoke about two approaches, one using VR and another using AR and machine vision with a camera pointing at the patient to track their move- Music can be used for patients to follow a rhythm, completing the movements, he preferred because when using VR and creating scenarios like ments as the song advances, tracking can be done more accurately through showing them bubbles and asking them to reach the bubbles, once they go sensors like the ones included in VR controllers, or through stickers with back to reality there’s a dissonance which makes it more confusing for the accelerometers or even advanced machine vision. Compared to now where patient and could even work against them. Using sounds instead of visuals families have to follow video instructions or photos which doesn’t give them in e blind VR setup would possibly work well too, so I’d like to try develop this feedback if they’re doing the exercise right or not. It can also be closely mon- application during summer after the project is presented. itored by the therapists through live data which is super accurate and instantly recorded to be analysed after each session and see if there’s any real

MFA Thesis Project 2020 Thesis Project MFA improvement.

“Artificially produced acoustic information and real-time auditory information have been implemented in sports and rehabilitation to improve motor performance in athletes, healthy individuals, and patients affected by neu- rological or movement disorders” [24]

These are the kinds of insights which are incredible to find because you realise that the best way to carry out the rehab excercises follows the same values as the ones described in the design principles and which have been explored throughout the thesis.

Home rehabilitation right now, especially in times of COVID-19

- 40 - NON VISUALS Playing with existing tech - Amazon Alexa

I started playing with the Amazon Alexa, an object which we never thought of as suitable for playing games but which was surprisingly fun, Alexa has at least a hundred games, and all are using just the audio and speech recognition features. Most of them are games we usually play by talking: 21 questions, guess my name, true or false, Jeopardy... There’s also quizzes like the song quiz or some story-based games; some games are specifically made for the device like for example, The Wayne Investigation, which was made as promotional material for the movie Batman vs Superman some years ago, it’s a choose your own adventure game where you have to solve a murder by looking for clues in the crime scene and then choose what clues you’ll follow and what questions you’ll ask the suspects, it has amazing voice acting and with your eyes closed it’s also very immersive, the only thing is that if you choose the wrong path, it lets you

go back one step and try the other one in a similar way that the interactive Black Mirror episode “Bandersnatch” did in Netflix. Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

Toby playing an Escape Room wth Alexa

- 41 - NON VISUALS Playing with existing tech - VR games

At some point in the thesis I had to try out some VR games by myself to really up to date, our natural movements as game input – if you saw a lever you’d experience it first-hand, I couldn’t carry out my whole research just watching pull it in the same way as you would in real life and after finishing with a wave other people play, so I carried out some VR game nights at school where I of enemies you had to salute the king watching from the stands to continue got a group of students to try some games I bought on Steam, using the HTC playing (which was a super cool touch). VIVE from school. I’ve been looking forward to this section of the research ever since the project started and it couldn’t have been more fun. Skeuomorphic VR

This is exactly what the future of VR will be like, we’ll start introducing our Sebastian Cabo Portugal de The games I tried are all on the top of every list for best VR games and are normal day to day moves in a way that will feel skeuomorphic at first, until we quite varied from one another, we could say they’re already classics in the VR get used to the environment and its possibilities and move beyond that; but I space. Superhot VR and Space Pirate Trainer are two shooter games which feel skeuomorphism is a good way to start as with any technology that we’re follow the classic gameplay of look somewhere, point and shoot, although not used to, especially one as complex as VR. The hardest part of this will be Superhot VR is quite an innovation as time stops if the player stops mov- to know what can or can’t be done in each instance (although easily solved ing, so physicality is a huge part of this game and makes you feel like you’re with good tutorials). When you play on a console you first learn the controls, in the Matrix, and that’s why it has become an instant classic. Budget Cuts which is a limited amount of buttons or a huge amount of button combina- is a kind of shooter game as well, but in this one you can move around and tions like in fighting games, but even without tutorials you could just press all choose how to attack your opponents, it’s more about stealth than about the the buttons and see what each does. This approach won’t be possible with shooting and it used the teleportation mode of movement which is normally natural movements because the possibilities are limitless and there can be a used to avoid motion sickness, but I personally don’t like this way of moving huge number of variations on the same move, like swinging your arm around around, it gets really annoying fast and breaks the immersion, we don’t move standing, sitting, crouching, laying on the floor… you get the point. It will also like that in real life and the game has no good connection between the story be a challenge to program everything to be interactive in a realistic way, al- and the movement mechanic, so it makes it obvious that you’re in a game. We though some games like Half-Life Alyx are already succeeding at this. also played a game called Moss, which is a platformer game where you play the role of a forest god that has to guide a little mouse throughout a fantasy Currently the best-selling VR game is Beat Saber, and although I couldn’t try

MFA Thesis Project 2020 Thesis Project MFA world, controlling its moves and fights while helping him with puzzles that it personally, I can understand why it’s been such a success; this is because he wouldn’t be able to solve because he’s too small or has the wrong point of it’s one of the first games to really use the potential of body movement that view; a very interesting approach that differs from every other game in this VR affords, it’s simple to understand because you control two swords, one list as it’s not a first person view in the classic sense. per hand, and we all inherently understand how swords work even if we’ve never really used one before, so it’s quite intuitive. It’s a fast, exciting game We also tried a game called Gorn where you are in the position of a gladiator with great music and challenging moves, and the fact that is in VR makes it in a coliseum and have to brutally murder every fighter in that arena coming closer to the old dance arcade games which are still popular today. For some in waves. The way you moved in the game was by swinging your arms as if reason these types of games always hit first when new technologies devel- it were your feet while walking. This approach isn’t ideal and I haven’t seen op, I remember playing a similar game in my smartphone once they became many more games using this mechanic, but at least they’re trying something more mainstream and being amazed by the affordances a touch screen gave different which I feel is exactly the right move. The reason why I like to high- us, even if it’s super limited compared to what VR gives us. This is just a fu- light this game is because it brought something that I hadn’t seen in games ture iteration of that same idea but feels like a huge leap forward.

- 42 - NON VISUALS Playing with existing tech - VR games

Problems that happen while playing VR Something I noticed is that even if you don’t move much around the space, just moving your arms and body on the spot is quite tiring and also you get quite sweaty, which makes the visor steam up. I feel this is a problem I would have to address when I say we should be moving more in VR, there really needs to be a hardware solution for this problem or if not we’ll have to stop playing constantly

to wipe off our sweat. It’s also a problem with the rest of the body, so maybe we’ll Sebastian Cabo Portugal de all end up playing VR games in sporting clothes.

Even though right now any game in VR will be a novel experience; like the excite- ment we all felt during the game night trying Adventure Climb VR, a very sim- Moss ple climbing game with only one level which we could never get past, the fear of looking down and while falling feels so real, even if the graphics aren’t top notch. This novelty will wear off as soon as people try more VR experiences in their life, just like with any other technology or activity, so the developers shouldn’t be complacent with how exciting things are now because they won’t last for long if we stick to the same path. I feel some games are going one step further and they’re finally trying to break these moulds they already got stuck in, but this is mostly shown in the visuals more than in gameplay.

The first VR masterpiece has just launched as I’m writing this; Half-life Alyx which is surely going to become the biggest VR game of this generation is already making waves throughout the corona quarantined world due to it’s amazing graph- Gorn MFA Thesis Project 2020 Thesis Project MFA ics, huge scale environments, exquisite sound design and in-game physics (a staple in half-life games) where objects react to your movements like they would in real-life keeping everything intuitive and realistic. It would be great to know if this level of detail in sound design and movement would enable visually impaired gamers to play but there are no reviews of this kind to this date.

Superhot VR

- 43 - NON VISUALS Playing with existing tech - Torball (Goalball)

I participated with Julian from APD in a training session of Torball organised at Dragonskolan for blind players. It was the third session ever done in Umea and it’s still not that popular as only three other people were attending the session, but there’s a league in Sweden so there’s potential for it to expand. It’s quite a popular sport played in 30 countries around the world and even included in the Paralympics in Beijing 2008. It’s an acoustic game as the ball has metal pieces inside which make a sound when the ball is thrown.

Torball is played on a rectangular court 16 metres in length and 7 metres wide. On the court are six players from two teams - three players per team. Goals are located at either end of the rectangular court. The game is played with a bell ball, which must be thrown underneath three cords tightened across the court. Sebastian Cabo Portugal de The object of the game is for each team to throw the ball across the opponent´s goal line while the other team attemps to prevent this from happening. Then the defending team takes on the attacking play and former attackers in turn defend their goal. [25]

It was super fun to play and quite interesting as you’re crawling around which not a position you usually find yourself in while doing sports, we could keep track of where we were positioned through feeling the tape on the floor. Even though they weren’t throwing the ball too hard, it was scary because when the ball bounces and is a bit in the air, the sounds stop and you don’t know exactly where it is; and even though it’s clear if the ball is coming from the right or left side is harder to know exactly with what you’ll block it, hands, face or body. Maybe it would get easier the more you trained as it didn’t feel like there was a difference between the visually impaired players we played with and ourselves.

The kids that played with us didn’t play any videogames but they liked to do sports like running and skiing when accompanied; they also tried climbing and spinning, as the bike is static and the instructor is good as giving instructions which is one of the most important aspects they commented on, same as with fitness. Torball and football are two team sports they enjoyed playing, and they had also tried gymnastics but said they didn’t like it because the other kids weren’t very understanding of their condition. MFA Thesis Project 2020 Thesis Project MFA

- 44 - NON VISUALS Experiment 1 - Spatial Awareness through music

First I wanted to test if our other senses are as good as our sight in getting a click that sounds like it is coming from one meter away on the left. If the information and helping us memorise stuff, in this case it was about know- system recognizes the person’s face, it will play a bump sound, and if that ing what objects were in a room. There are products like “Microsoft Seeing person is also known to the system, it will announce their name. [26] AI” which can read out what the device sees in a similar way as voice over in our phones. The problem I see with this approach is that it creates a pret- Facing emotions by Huawei also uses musical sounds to convey the emotion ty strange world for anyone using it, where objects are read out loud to the in the face of whoever is in front of the camera. The sounds were created

user when they look, imagine going into a room and hearing “chair, sofa, tv, by a musician and they’re each inspired by the emotion it’s trying to repre- Sebastian Cabo Portugal de chair…” Even though it’s super useful, the aesthetics of it are not very ap- sent. The app translates seven universal human emotions into seven unique pealing in my opinion. I wanted to try a similar approach but instead of voice sounds. Through the phone’s camera, the app scans the face of the person over through the use of music. Like going into a room and listening to a sym- the blind person is talking to, identifying their eyes, nose, brows and mouth, phony of objects to know what each piece is and where they are positioned along with their position in relation to each other. AI then processes the emo- in space. tion into a defined sound heard on the phone. All this happens in real time and in offline mode. [27] So this experiment consisted on putting four pieces of furniture in a room - a chair, a tall table, a short long table and a TV - each one with its own sound Huawei collaborated with the Polish organization for the blind, and “it was track, blindfold people and get them into the room and let them walk around through this collaboration that we learned app users would love a hands-free and touch the objects so they can figure out what each thing is and get closer option. Visually-impaired people often need their hands to hold a cane or to to the sounds so they can link the two together; I then take them out of the help them navigate”. [27] room and remove a sound track, so when they come back in the room they have to figure out what piece of furniture is missing. The results were really good, the four people that came into the test completed successfully; when they came back to the room they didn’t have much chance to walk around, all the sounds where coming from a similar direction but they still knew what

MFA Thesis Project 2020 Thesis Project MFA thing was missing pretty fast, then I would let them walk around the room again and then they would really spot what was out due to the directions the sounds where coming from. One piece of feedback that did surprise me was how some people were using the vibrations created by the speakers to remember better the rhythms and sounds each piece of furniture was generating “I was isolating the sounds by feeling the vibrations in the table, that way I could remember the patterns better”.

Microsoft Research have been working on a similar approach in Project To- kyo by using AI to help blind people find familiar faces in a room, but instead of directly reading out the names of every face in the room if the device de- tects a person one meter away on the user’s left side, the system will play Facing emotions promotional image by Huawei - 45 - NON VISUALS Experiment 1 - Spatial Awareness through music Still images from a video recorded with a couple of the participants in the experiment, two out of the six that took part. Showing how they get led into the room blindfolded, they each take their own path to choose an object to touch and then tell me what they think it is before memorizing its sound, then taking them out of the room and removing one of the sounds, so when they come back into the room they have to spot which object is missing. Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

There was another variation of this experiment with two other users, where they wouldn’t be able to touch the objects but I would describe what each thing was as they walked through the room, this proved a bit more tricky than the first approach as out of the two test subjects, one of them got the answer wrong and the other one right. I’m quite confident that using sound and touch can be as memorable as sight, or even more, as sight is quite easy to trick, I mean how many times have we been looking for objects all around us to finally discover they were right in front of us the whole time.

In the next version of this experiment I will try and create sounds that make sense for each piece of furniture and are related to each other, for example tables would sound in a similar way, chair and sofas would change a bit more but still be recognisable as seating and a TV would have a completely different sound than the other objects.

- 46 - NON VISUALS Experiment 2 - Playing fussball blindfolded

This was a very quick experiment where I participated with three other colleagues, we all got blindfolded and tried to play a full game of fussball like this. The interesting thing was that it was quite possible to play like this because you can hear the ball going up and down

the field, hitting the walls or other players. Sebastian Cabo Portugal de

The tactile affordances made it really nice to play, the only issue was when the ball came to a halt and didn’t make any noise, then we would all be moving our players everywhere to see if someone would hit it. A stopped ball made it feel like some kind of sensory depra- vation chamber where you would be confused and not know what is happening at all - some kind of sensory feedback is continuously required for this not to happen.

If the ball had integrated sounds, like real full- sized footballs for blind players, then it would be a lot easier to keep track of where the ball is at all times. MFA Thesis Project 2020 Thesis Project MFA

Goal getting scored while everyone is blindfolded

- 47 - NON VISUALS Experiment 3 – Controlling Logic Pro with movement

This experiment was done in collaboration with the musician and audio en- when it was in the middle grounds, the software would tell you the exact gineer Andreas Estensen in his studio. We tried improvising a quick setup distance in meters but it was hard to gauge oneself. It was also impossi- where he could control different parameters for some beats and melodies ble to know if a sound was in front of you or behind you unless it swished like tempo, filters and effects, and I would move/dance while he would passed your head or it was moving in other ways like rotating around your change these parameters depending on what I did with my movements. head. We then flipped it around and he would play sounds while I tried to react to them. Two ways of developig the movements Sebastian Cabo Portugal de There can be two ways to develop this which were looked at during the con- Even though it was a quick improvisation we figured out quite a lot of things; cept exploration phase: on one side is free movements that feels more like for example, that the whole system should be constrained to only musical play, and on the other side of the spectrum is choreographed moves which stuff, instead of unbound sounds which would be harder to control or be you would have to learn in order to produce what you want, in a similar way precise about, everything should be in steps and on beat so you feel like that the art project Body.Coding by Alvin Arthur. In this project the user you’re creating music while playing and not just random sounds. The con- learns a predetermined set of dance moves to create things like void setup cept will follow the steps of things like the Nintendo LABO keyboard, which () {, ellipseMode (), fill ()... It can be done alone or in teams, and even though is fairly simple to use as its marketed for young kids. When I tried this prod- it’s a super creative idea it feels more like a provocation than something uct during my time at LEGO, it was the first time I really felt that I could play that could actually be used for coding, as programming is so complex and music straight out of the box, without having to learn any theory or practise boundless, the amount of moves you’d have to learn would be too many; for a long time, and this is the feeling I want to convey with my own music but the choreographies would look nice. [28] maker. There’s two sides to this, the human side and the technological one. The We played around with software that creates 3D spatial audio called human side is concentrated on the experience and play and that their in- DearVR, where you could grab a sound and move it around the head of the tentions are being understood, from the tech side it gets harder because listener to see how they would react. It was really easy to distinguish if the they software would have to know what the participant is inferring with

MFA Thesis Project 2020 Thesis Project MFA sound was on one side of your head or the other as well as if it was really far their movements which can be uncountable iterations if they are allowed to away or it was close by. It got trickier knowing exactly where the sound was move freely, this is also explored during the zoom workshop.

- 48 - NON VISUALS Workshop 1 – Music Maker with Toby and Julien

Music stimulates the pleasure and reward areas of the brain, and it also ac- From the chat with M Eifler I got the motivation to try and explore other tivates the cerebellum which is involved in the coordination and timing of kinds of applications using the same tool in a different context, from a crea- movement; this means that sensory experiences are also motor experienc- tive mode into a communication mode and a utility mode. My mind instantly es. [29] goes into appls like Excel (which for me is the most boring application of them all) to try and see how could this be made more fun or physical. In the I’m not much of a musician so as soon as I had the idea to design a music end I chose to reinvent Word and a home rehabilitiation app to show exam- maker through movement and physical objects I thought I needed to do a ples of the exploration of utility/day-to-day apps. I’ve had this thought in Sebastian Cabo Portugal de workshop with people who are already making music. I already did an initial my mind for a while on how would the world look if we could do our jobs and improvised workshop with Andreas Estensen where we spoke about what exercise at the same time. and this is the chance to find out. kinds of parameters could be tweaked while keeping the software simple without an endless amount of options.

“Interactive music succeeds when it encourages spontaneity while resid- ing within the boundaries of a dynamic artistic context that is whole and engaging” [30]

Music and dance are inextricably linked together, once you hear the right beat you don’t know why but you have to move to its rhythm. This is one of the reasons I instantly thought of this application of physical or body control.

The workshop was super fruitful and lots of different options came out of it, from creating music through playing with LEGO or through drawing on a

MFA Thesis Project 2020 Thesis Project MFA paper all the way to using a whole room as if it was a synthesizer. Sequenc- ing sounds was the toughest nut to crack but we found some plausible ways in which this could be done.

I spent at least some weekends and even a whole week exploring two devices by teenage engineering, the pocket operator PO-33 and the OP-Z. I did this firstly because they are one of my favourite brands in terms of design and interaction, but secondly to learn more about what instruments like synths can do by themselves, especially simple ones like the PO, and use this as a basis to develop my own music making software.

- 49 - NON VISUALS Workshop 2 – Physical game jam/Bodystorming

This project pushes me to find new ways to prototype and generate ideas What happened in the workshop? that aren’t commonly used in the current design process. One type of work- The participants were divided into groups of three and each group given shop I love is improv sessions. Focusing on things other than vision means one sense and 15 minutes to ideate, then every group would present their we have to introduce new materials that fall away from visual ideation like ideas which would be recorded on video. Filming the ideas is key because sketching and writing down in post-it notes in order to break the mould and it’s the only way to keep them as sketches, the same way as you would col- think differently. lect all the post-its in a workshop to analyse later. Sebastian Cabo Portugal de How do you ideate for something that is so emotional and connected to our We then did a second round where each team could choose to change the bodies like movement? Do you start by talking about it? Drawing it in post- sense they were working with or keep the same one as before, but this time its? I’ve tried these methods before and it’s really hard to talk about move- with more focus on having no vision, so one person in the group would be ment, let alone imagine new ways of playing with movement without try- blindfolded to play, and after 15 minutes we filmed the new results again. ing them first; it’s completely different than creating anything visual, when dealing with sound you need to create sound bites to guide your ideas, and The ideas that came out from this workshop were generally multiplayer (as when it comes to physicality you have to create physical sketches. The only I didn’t put in place any rule against this) and many of them still used vi- thing close to this that we generate in our current workshops are user jour- sion as part of gameplay even though I asked them not to (especially during neys, but they are still more about the bird’s-eye view of the journey instead the first round), but it’s hard not to use vision when you have it and it’s so of the detailed version of how they would actually move and act inside it. overpowering; for example, by copying someone else’s moves, or getting triggered through some kind of move the player sees, that’s all still vision. I The objective I gave every participant at the beginning of the workshop hoped they would use sounds and tactile or body feelings like sensing wind was really hard – imagine a game or story which can only be told through blowing (which one team did explore) to create these gameplay triggers, so one sense. I then gave them a list of three senses to play with – movement, that’s a big change that needs to happen in the next version of the work- touch and balance. As hard as I tried to imagine how a story would look shop. through just balance, I couldn’t come up with more than a couple without

MFA Thesis Project 2020 Thesis Project MFA trying anything physically, and I was expecting the teams to really struggle This was a first version of the workshop, mainly to try out if anything would in my workshop but I was gladly surprised that it was the opposite, they all come out, not even looking for great ideas to be used in the games I was came up with many ideas, really fast. ideating, and in that sense it was a success because everyone came up with a bunch of ideas, some which could even be introduced into concepts To begin the workshop, I did a simple warmup exercise to get everyone like the VR Audiogame or even the location based AR game. moving and thinking fast, get them on their toes. It’s good to start a workshop like this with energy, compared to typical workshops where everyone The idea was then to do a second workshop more focused on the stories is sitting down around a table writing or drawing, because they will be mov- and concepts I was creating for my games but due to the closing down of ing a lot throughout, even in a typical workshop it’s a good idea to start by the school after the COVID crisis, this could not be done. That second work- moving a bit or stretching, getting people minds and bodies loose. shop would’ve really been the insightful one and key in generating useful “physical sketches”.

- 50 - NON VISUALS Workshop 2 – Physical game jam/Bodystorming Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

What I would change in this future workshop would be to add more rules: to make sure the team would concentrate only on single player gaming, one acting as the player and the others would be the triggers in the environment. I would also blindfold the player since the beginning until the end, so the teams would have to put more effort into doing things that can’t be shown and copied through visual cues.

The most important aspect learnt from this workshop is that it’s key to introduce Bodystorming into any process that deals with applications for VR/AR/ IoT. The only real way to generate concepts for these types of tech is by trying them physically as soon as possible, and then record them through video so they can be pointable and shareable with the rest of the team and in future parts of the process. You can track the development of the idea by comparing videos. - 51 - NON VISUALS Suddenly the world changed

Right after I had finished all these interviews and done a couple of workshops my laptop decided to die completely, this was the first hard blow as I spent 4 days trying to fix it only to see it die completely with all my software and projects inside. I found a way to take the last bits of the thesis I hadn’t backed up yet from the hard drive so I didn’t lose any work from that side. The school gave me a Microsoft Surface which will enable me to do simple work like write my report, and make presentations but it’s not powerful enough to do proper design work or run anything to do with VR. Sebastian Cabo Portugal de Then when I thought everything was sorted and I’d go back to normal came the second and biggest blow of all, COVID-19. This was a huge disruption to my project as I spent some days wondering if I should stay in Sweden or go back to Spain, where the virus was hitting hard. I finally decided to go back home as my parents were worried that I wouldn’t be able to go back home later due to closing of air space, and I was also worried about their health and being so far away.

This disruption is going to affect every thesis project this year, but I feel I will be able to overcome it without having to change my plan much. I knew it would be a struggle to carry out interviews or tests with visually impaired users anyways, but now every quick workshop or test which I could carry out will be also affected, compared to the ease I had on doing quick workshops with people at school at any given time. It’s also a shame because I had finally found a group of visually impaired users, within the Torball group in Umeå, who wanted to test my prototypes but now that opportunity was gone. This will be the real loss for the project, especially for my topic which is so experiential and physical, it’s going to be close to impossible to make concepts which users will be able to try, more than creating videos which they can watch and give their opinion on; but in my case, the real useful feedback would come from experiencing the concepts oneself.

On the brighter side it’s a good opportunity for everyone to develop methods of doing design and research MFA Thesis Project 2020 Thesis Project MFA while being stuck in one’s home. How do we carry out user-centred design when we can’t meet our users? And what does it mean to be user-centred in this new context? This is a question I’ve been asking myself for a long time; what does user-centred mean anyways, how many users do you need to qualify as user-centred… so now it’ll be the perfect time to question these approaches. The answer to all these questions is…we’ll have to find out.

Another struggle will be the isolation of working on one’s own all the time without seeing others, even though the thesis is individual, the energy that others give me in school, the ability to have breaks together is something which will be hard to match through video chat. This is going to be a wakeup call or a good opportunity to develop better ways of working and meeting friends/co-workers from home, and the most immersive way would obviously be with VR and AR, so maybe this is the beginning of the VR revolution that has been talked CNBC and LA Times talking about COVID-19 about since the resurgence of the technology in 2012.

- 52 - NON VISUALS Workshop 3 – Communicating through movement

Even though we were all stuck in our homes we still had to find ways to con- In theory the format was perfect for the experiments I wanted to carry out tinue with the design process as normal as possible, so I decided to do a because it would show a grid of participants all acting out at the same time workshop through zoom. which would make it easy for comparison. But again due to the limitations of internet connections and video call technology I didn’t get the quality of The first experiment was to see how people moved through specific sounds results I wanted. and beats. Everyone would close their eyes, so they wouldn’t be self-con- scious and so they wouldn’t get influenced by the rest and then differ- The second part of the experiment was playing a game of charades, where Sebastian Cabo Portugal de ent soundbites would play and they would let themselves express freely one person has to act out a word or phrase by just using movement, then if through movement/dance. no one got the correct answer I would encourage them to use just sound, and if they still didn’t get it I would ask them to combine movement and It started simple enough, with just a recurring beat, like in techno music, sound. Props could also be used if they felt like they needed them. which I was expecting everyone to nod their heads or do fist pumps. Then I moved to more complex beats and melodies. This part of the workshop was Insights not as successful due to the limitations of video calling through zoom, one In this experiment I did get some interesting findings; mainly that some ac- problem was with sound, the mic would pick up the music I was blasting tions are really easy to communicate because they are very iconic, for ex- through some external speakers and it probably thought it was background ample sports like rock climbing, playing football; there are also characters noise so it turned it down and the participants couldn’t hear properly. The like clowns which seemed to be instantly recognisable through the simple other problem was with screen freezing and lagging, most times, some of act of pressing your imaginary fake red nose. One action that surprised me the participants would freeze for me, but not for others, so I couldn’t get the was landing a plane, everyone instantly got that it was a plane flying, but image I wanted of everyone moving in unison live. The final problem was acting out like it was going down made people instantly think of landing it. that because we are used to sitting down in front of the camera for video Some situations that we don’t see that often or they don’t have an iconic calls, we couldn’t see the whole body of the participants and they would action behind it were more confusing for the participants to spot like food just move the upper part of the body or mainly arms and heads. I expected fighting or celebrating a birthday, in these cases the use of external objects

MFA Thesis Project 2020 Thesis Project MFA them to show bigger movements and use their whole body. as props was required for people to really get the words they were trying to act out, although it could also make the actions more confusing, for ex- I will make another version of this experiment where I’ll make a sound track ample when one of the participants tried to blow some imaginary candles with many different sounds and let them film themselves while listening to out of a mug and the rest thought he was spitting on coffee. Adding sounds it and move however they feel it would portray that sound best. Then I will was also quite useful for words like coffee, where the simple act of drinking manually join all the videos together to see if there are any visible patterns from a mug could mean drinking anything, but the sound of coffee brewing or not, and what types of sounds have movements linked to them and which made it clear what it was specifically. ones have greater variability from one person to the other.

- 53 - NON VISUALS Workshop 3 – Communicating through movement

Some situations that we don’t see that often or they don’t have an iconic If we were to communicate with our computers through movement maybe action behind it were more confusing for the participants to spot like food a good way would be that the machine gives you sound feedback so you fighting or celebrating a birthday, in these cases the use of external objects can be sure what it’s understanding what you’re trying to say. It would still as props was required for people to really get the words they were trying be hard because of what this experiment shows, the way we understand to act out, although it could also make the actions more confusing, for ex- and portray things vary person to person so people would be saying the ample when one of the participants tried to blow some imaginary candles same things in different ways, and it might be hard to calculate all the dif- out of a mug and the rest thought he was spitting on coffee. Adding sounds ferent possibilities or understand the small differences between saying Sebastian Cabo Portugal de was also quite useful for words like coffee, where the simple act of drinking one thing and understanding something different. There are infinite num- from a mug could mean drinking anything, but the sound of coffee brewing ber of variations with unbound movement so maybe it would be better to made it clear what it was specifically. provide the appropriate choreography needed to fulfil an action, and give tutorials in similar ways as yoga lessons are done to teach these moves to I also tried giving them common actions that we do with our computers, the users beforehand for example sending an email, which was as simple as acting out typing and then pressing a button to its side, everyone understood that action instantly, even though I expected them to move more into the physical realm of sending letters, but for that I should’ve asked to describe “send mail” – which one of the participants showed by acting like she was licking an en- velope. It’s funny to think that we might have to continue using these paradigms, at least for a start, because we are so used to the tech world the way it is now that we equate every action to using a laptop instead of its counterpart in the real world, what we know as skeumorphism.

Another suprising thing that came out of the experiment was when one

MFA Thesis Project 2020 Thesis Project MFA participant described excel via sounds, he would move his head from one side to the other mimicking column, then choose one and move down the rows until he reached the right position he would make a typing sound as if you were inputting data into that space. For me, knowing beforehand what he was trying to portray, it was really easy to understand and was quite

When asked to describe “add to calendar” things got really tricky, it’s very hard to explain complex themes like time passing and organization; when Zoom workshop - in this situation Tao (in the middle) was trying to communicate climbing you don’t have any object to point at, first you try and create the calendar through mimicking it’s shape

- 54 - NON VISUALS Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

5.0 CONCEPT EXAMPLES

- 55 - NON VISUALS Introduction to Provocations & Examples

This project wasn’t a problem/solution oriented one but an exploratory one, The main aim of everything created in this project is not only inclusivity but so the following concepts aren’t solutions to prescribed problems but more also innovation. By flipping around every “generally accepted convention” examples of the kinds of innovations that could come out if we changed of how the future of VR/AR and IoT will look like, we can open up the spec- our methods a bit and focused more on our humanity instead of our search trum of possibilities of these technologies, and move beyond a vision-cen- for ultimate efficiency and focus on visual stimulus. Trying to design stuff tric approach into a body and sensory-centric one, which some have al- is another way of getting input, this was the aim of the concepts, to ready tried through products like Dopper - the first computer interfaced continue opening the exploration done in the research and material mainly through audio, or which could even be seen in movies like “Her” Sebastian Cabo Portugal de exploration sections. (2013), even companies like Bose have just started including 3D and spatial audio recognition in their latest updates for their newest headphones. So This research project is not about finding THE ONE SOLUTION, but is trying this space has been and is being explored currently, showing it’s a key area to explore the design space – and all of the examples produced are equal- of interest for the future. ly valid in opening this design space. Even though is not about finding the right solution, we can find many advantages that these examples bring over The examples where developed from two initial provocation areas: playing what we currently have or the current direction of our imaginations towards games without using vision and using our own bodies/movements/ these technologies. senses and objects as controllers/input. These where then broken down into two different concept examples inside each provocation area, We’re bringing our physical reality and the virtual reality we created with the physicality apps ending up as three examples to show a wide va- closer together than ever, not limited to a screen any more, blending riety of different real-life applications. Each project is trying to break down into one through mixed reality. This shift calls for a new way of think- something from the principles I found and through the research of current ing and designing for these technologies and applications; bringing technology and inspirations: them closer to how we interact as humans with our physical reality, so instead of being E.T like beings with a glowing finger that inter- - VR and AR Games are all about visuals - then take the visuals out acts with the virtual world, we could use our full human capabilities of - Our current OS only let’s us communicate with it using a couple of fingers

MFA Thesis Project 2020 Thesis Project MFA movement and feeling senses to interact with this new reality. and everything is show in one screen - then create an OS where you use the tactility of real objects to trigger the applications and the use of space in How to develop the concepts your home to separate the programs, contextually aware. Apart from all the findings from the previous sections which informed - Our softwares are controlled mainly through our fingertips and visualised the directions for the things to solve and inspiration for the concepts that through our eyes and other senses are added through the use of more tech- can come out of that; each provocation/example tackles a different area I then create pgrograms that are controlled through physical movement and wanted to address from all the ones found during research and that took the use of already owned objects like cutlery or fruit. shape by following my own prescribed design principles generated during the first loop. Detailed analysis in the next chapter.

- 56 - NON VISUALS

Playing games without vision VR Audiogame - PRIMAL SOUNDS This concept exploits the best capabilities Audiogames can have, mainly The game (Primal Sounds) consists on a blind hero living in prehistoric 3D audio, in a much more immersive way than its current form - playing on times who has to survive using his wits, sound cues and the help of his dog 2D screens or with keyboard and mouse in a 3D environment. You can now companion. Your main tool is a stick which you carry with you all the time. move in real space to get close to the sounds and objects to interact with and even touch them and use your body movements to interact with these You would start lying on the floor inside a cave next to a fire crackling and on objects or creatures along your adventure. the other side of your head would be your dog. Once you’ve explored your cave by moving around the room you,d be able to go to the outside world

One of the aims that came from the research was - the visual overpowering with all its dangers lurking. Every scenario/level would be room sized. Sebastian Cabo Portugal de to a point where we have forgotten how much we use our other senses in our daily lives to move and understand the physical world. So this example Once outside you’d hear the nature around you and a series of key sounds is not only a direct provocation on all the graphics-centered games which which would lure you to different actionable points in the map, a waterfall need to re-create physical reality in the virtual world in the closest way pos- next to a small pond where you’d be able to fish by swinging the stick as if sible and fall short on usability and finding innovative ways to play within it was a fishing rod. There would also be out of limit zones which would be this medium. But also brings forth a new experience which can be enjoyed marked by dangers like the sounds of a snake pit, letting you know if you go by everyone as it would be available on all current platforms. there you’d die/ you’re out of the play area.

For visual users it would be a reconnection with their other senses, an op- Finally there would be a section where dire wolves would attack from dif- portunity to retrain them, and also to feel how much they can actually do ferent angles around you, with distinct sounds so you know when they’re without their vision which would ultimately help them empathize better about to attack, when they’re on the air jumping towards you, and finally with visually impaired users as more than a "disability" but people that "can when they’re close enough; and by moving your arms like hitting them with do everything but see" [3] a stick you’d protect yourself. If not you’d loose health which the player would notice by the heavier breathing from his character, until you die. For visually impaired users it not only helps them to get into VR content they can enjoy and fully explore, but also creates a hero, in this case the main MFA Thesis Project 2020 Thesis Project MFA game character which represents their situation, and we’ve seen from the research how important representation in all kinds of entertainment is for the people that are normally left out. Empowering for them and gives them “visibility” in other people’s eyes.

Extra advantages: It also avoids common problems with current VR like motion sickness as there is no visuals to cause it (more inclusive), and in the future when VR games are streamed just like Netflix or Google Stadia already provide, the amount of internet used will be negligible in comparison to games with amazing graphics.

Stella Jacob - Unsplash - 57 - NON VISUALS

Playing games without vision AR location-based game This was the trickiest topic to handle because it was trying to break the The game starts in-house and then moves outdoors during the second act, barrier of having visually impaired people stuck inside all the time, unable but not just anywhere outdoors, specific safe locations are chosen which to enjoy games like Pokemon Go for their lack of accessibility and dangers will provide the playing ground for the game for anyone who wants to play. when walking around a city. The specific locations can also be used to introduce some of the senses I had left out at the beginning of the project for being too complex, like smell The main point of the game was to find new ways of playing around a city - if you want your player to smell flowers at a specific part of the game, just that don’t involve playing through a phone that is constantly attracting at- locate that mission inside a park.

tention visually to play in the AR environment. Again challenging current Sebastian Cabo Portugal de conceptions that all AR is visual and that is its main appeal. New ways of The player would recieve a text message once they open the game app as a gameplay had to be explored, and through experimentation with sound, quick quiz to see if they’re in a safe space, then they would recieve a phone movement, real objects, location and the passing of time the game call with the story and mission to complete - meet the next day at 5pm in X reached the point it turned into a Transreality game - a type of video game location (the location would be based on the city of the player). or a mode of gameplay that combines playing a game in a virtual environment with game-related, physical experiences in the real world and vice They would then recieve another call from a different phone number which versa. In this approach a player evolves and moves seamlessly through would tell them not to trust "the agency" they got in contact with before- various physical and virtual stages, brought together in one unified game hand, that they’re "the resistance", and to meet in a different place at 5pm. space [31] This would already make you choose sides by going to the location of the agency or the resistance. They would also ask you to scan your house in The application of game mechanics to tasks that are not traditionally asso- case you’d been bugged and you’d do this by pointing the phone camera ciated with play, a transreality approach to gaming incorporates mechan- around and like a radiation scanner, following sounds until its peak where ics that extend over time and space, effectively playing through a players’ you’d find a bug and deactivate it. This ends Act 1. day-to-day interactions [32] Act 2 is outdoors. Once in the location at the specific time (where you’d ar- The problem with VI users is to take away another main sense for under- rive the same way you would in life outside of the game) the game would MFA Thesis Project 2020 Thesis Project MFA standing the world around them like audio feedback, so the solution would re-start, triggered by that location and you’d be advised that you’re being be either to play with Bone Conductor headphones which can be played at followed, so guided through voice and tense music you’d have to move to a level low enough to let them hear their surroundings but high enough to different spots around that area to loose the tracker. Then you’d have to enjoy the story and challenges of the game, or by using their current head- scan that location for alien forms and follow the sounds they produce to phones at a low volume. At home the game could use common objects as find these “invisible beings“ and either protect them if you’re in the resist- "spy tools" by scanning them with their smartphone camera. ance, or kill them if you’re with the agency.

The Game Act 3 would conclude the game and is still in development but it would This is an X-Files style Spy game that is about "seeing and not being seen" again use the passing of time and inclusion of real life, and in this case you’d - as a play on the fact that vision is the least important part of the gameplay be attacked by the contrary side from the one you chose to follow. and linking it to a theme that makes sense with the non visual gameplay.

- 58 - NON VISUALS Using our own bodies/movements/ senses & objects as controllers/input Home OS – system level non visual computing More than a product this is a UX Ecosystem which finally marries VR/AR The system has three distinct levels of interaction: and IoT into one, what is known as the virtuality continuum: a continuous - The first level is to access simple information or deal with simple

scale ranging between the completely virtual, a virtuality, and the com- actions via voice control, for example in the music maker to play a Sebastian Cabo Portugal de pletely real, reality. The reality/virtuality continuum therefore encompass- song, in a similarway to how Amazon Alexa works nowadays. This is es all possible variations and compositions of real and virtual objects, with the higghest level of the system and would work more like apersonal mixed/augmented reality as a midpoint in the continuum - augmented re- assistant that talks to you instead of overwhelming you with visuals ality, where the virtual augments the real, and augmented virtuality, where everywhere. the real augments the virtual. [19] - The second level is to access more important information or actions, it’s All your devices can be connected through the system and used with any of the middle ground, and this would be triggered by touching the object that the common devices which will be available in the market, from your smart contains the specific app you need. Here simple visuals could be intro- homes speakers to your wireless headphones. Like Android for your home. duced, for example cover albums or song lists show once you touch your speaker, or in case of the music maker you culd cut samples from the songs Used across different realities rather than across different media you’re listening using common objects like fruits. One key difference between this OS and the current ones is that is localized - The third and final level is where you cana cess the full capabilities of inside your house, which means that you can map your home and assign the application/software, here you would go into full VR immersion either room-to-folders and object-to-apps, so the cooking apps go in the kitchen, through visuals, like going into a 3D modelling studio that looks like a clay the workspace is separate from your sleeping room, and your toilet could studio instead of your room or in the case of the music maker it would track

MFA Thesis Project 2020 Thesis Project MFA ; contain all your social media apps (two birds one stone). In order to use your your movements and position in space and give you aural feedback to pro- work related apps you’d have to go to the specific work room. Your bed- duce a song and even mix the samples with the fruit you used in the level room would probably contain the entertainment apps although something above (deeper explanation of how this works in pg.66). like Spotify could be more generic and be mapped to speakers around the house or be used anywhere. This approach for an operating system cntradicts everything we expect from an OS nowadays, where you’d physically have to move from room to This breaks the current convention where we carry all our apps and utilities room and touch objects to trigger apps instead of sitting down and have with us wherever we go on our phones or we have a centralized screen like everything in the palm of your hand - this improves concentration, reduces our desktop, where everything happens and can get distracting and over- the stress of multitasking and distracting notifications, and improves your whelming. Breaking the desktop metaphor for a studio metaphor. [22] health by making you move constantly.

Inverse Skeuomorphism from the digital world back into the physical

- 59 - NON VISUALS Using our own bodies/movements/ senses & objects as controllers/input

Physicality apps – example of apps that can live inside Home OS Sebastian Cabo Portugal de

These examples blur the virtuality continuum where in augmented virtuality the real augments the virtual, so by touching real objects you feel better the virtual app; while the virtual app gives extra uses to the physical static object.

All these applications are based against the all-brain way of working we have currently with our technology, all the stress happens in one place and it gets very tiring and produces secondary effects that affect our mental health, so in order to break this we put into practice the all-brain, all-body, all bodies design philosophy from M Eifler in order to balance this workload between your mind and your body; because the body is also a way for us to learn and to think, and so it can take some of that stress produced in your mind and by moving more, even get you physically more fit. Gone are the days of back and neck pains or dry eyes from staring at a screen all day long.

It was important to explore what is considered more utilitarian applications, to see

MFA Thesis Project 2020 Thesis Project MFA if these modalities could work not only in creative applications or games which are a lot more free to experiment with, but in the day to day life of most white collar workers.

The first one that got developed was the music maker one, as it was the lowest hanging fruit - sounds and moves connected together is obvious. The challenge then was to try and devlop app which initially didn’t seem connected in the same way, like writing and walking, but after some though you realise that walking and talking is very natural to all of us, and that we even think better or have our best ideas while we walk, again proof that our sensory experiences are also motor experiences.

- 60 - NON VISUALS Physicality apps - Music maker through movement Sebastian Cabo Portugal de

Music and dance are obviously heavily interconnected, as I’ve explained before, sensory experiences are also motor experiences in our brain. So this was the first application that came into fruition in terms of using the body or common objects as inputs, instead of the current model of adding more tech devices, that add more expenses, to be able to have the "full immersive experience” and are another form of exclusion.

This application creates a grid base on the floor, and depending on where you are in the 4x4 grid you control a different parameter, drums on the top row or synths on the bottom row. You can choose an array of There is a middle section which controls the effects, so by moving more erratically you add more/crazier effects MFA Thesis Project 2020 Thesis Project MFA for example.

Inside an earea of the grid you can explore the samples available by moving your arms around your head and if the arm is kept in a position for long enough, that’s the chosen sample you’ll play with and then you’d use it just by moving, or use several samples by assigning them to parts of your body, so you could use two arms to play two different snares at the same time.

You can mute or delete sounds by gesturing, and change some parameters like volume or punch in each part of the grid by twirling around.

Dancing to make music, who would’ve thought...

- 61 - NON VISUALS Physicality apps - Walk + Talk - Live caption through movement Sebastian Cabo Portugal de

Tired of being bound by our desktop centered lifestyle, always stuck in the same position, sat down even for the most mundane of actions like sending an email. We wanted to open the world that everyone has at their disposal surrounding them just outside of those screens.

This is why this application was created, now you can just walk and talk while the live caption does its job of capturing all the information in the format required (so you don’t have to worry about the ummss and ahhsss) stopping creates a full stop whilst a longer stop is used to skip paragraph, gestures or sounds to do actions like write in italic or quote “in commas” Just imagine starting your day shifting through emails

MFA Thesis Project 2020 Thesis Project MFA while you walk to work or across the park.

For the creatives out there - if sitting in front of a white page has always been your main creative block, the thing that keeps you from starting, walking provides the kin- esthetic push of inspiration you need to get creating right away.

This could be done through glasses or an earpiece which have cameras that can see your whole body or through an external camera like the one from your phone or computer - which tracks your movements live, making it ready for the current market and future proof.

- 62 - NON VISUALS Physicality apps - Home Physical Rehabilitation Program Sebastian Cabo Portugal de

Introducing the best home rehabilitation product in the market, all worked out with professional physiotherapists from Spain and the UK, which gave us the insights into the science and methods behind physical therapy.

This app turns rehabilitation into something close to meditation, body and mind connected like never before – eyes closed, simple instructions with MFA Thesis Project 2020 Thesis Project MFA no hand-holding and guided by the music. The volume of the song of your choosing gets louder as you move until you reach the right point for that exercise. Always closely followed by certified physicians that updates the exercises when needed and keeps a close tab on the data the user produces to see if they are improving and doing their homework for a faster recu- peration

This could be done through glasses or an earpiece which have cameras that can see your whole body or through an external camera like the one from your phone or computer - which tracks your movements live, making it ready for the current market and also future proof.

- 63 - NON VISUALS Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

6.0 ANALYSIS

- 64 - NON VISUALS Insights from the explorations Sensorial Exploration Tech & Content Exploration

LOOP 2 LOOP 2 We all use our senses (non visual) more than we realise Tech can be used to reconnect better with our senses

We can expand these capabilities by using them more We can move from fixed viewpoint computing to variable viewpoint computing

(training) Sebastian Cabo Portugal de Validation that VR/AR/IoT will merge into unique UX Ecosystem Sound & touch can be used for spatial localization/ feedback/ object memorization Immersive computing has affordances to become 1st fully inclusive technology Sensory deprivation is too confusing - continuous feedback of some sort needed Content is as much of a problem as tech itself: - More empowering representation of non-normative people LOOP 3 is important All-brain, all-body, all-bodies to balance workload be- - Inclusive content broadens the mainstream market tween brain and body - Better Physical health LOOP 3 - Less distractions Use of mainstream technology is better than creating a specific device - Better body/mind balance Enable VI and visual people to play together Sensory experiences are also motor experiences MFA Thesis Project 2020 Thesis Project MFA Efficiency based tech overstress our brains Iconic sounds/actions ease communication Machine needs to give continuous feedback on what you want to do

Free movement - better for human Choreography - better for machine

Non visual experience for visual users - help empathize better with VI people

- 65 - NON VISUALS How to develop the concepts Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

Workshops were developed from the insights gathered throughout the process, which would define the topic/provocation that would be explored in the workshop. Then a first improvised workshop would be made just to test the idea and see if it would lead to any kind of result. Once it did, a second workshop would be organized but this time with more focus on one or more of the concept examples that were being develpoed all the way through the process alongside the methods.

- 66 - NON VISUALS Example in use: Music Maker Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

- 67 - NON VISUALS Design Methodology Insights

Push beyond games and creative apps into utility apps (white collar apps)- to see if modality works in general, and is not just a gimmick or a new way of gameplay specific for that game.

"Starting from other senses is good for exploration phase" Sebastian Cabo Portugal de Lo-fi prototyping with existing tech (not actual VR/AR) can yield results - for example placing several speakers across a room like it was done in the first experiment or even by using improv theatre workshops, where everything is in the imagination and any object can be used as anything - letting our minds free for exploration.

Mixing experts and non-experts in workshops yield best results - Experts bring more focused results/idea - Non-experts bring freshness and early problems to be solved, for example simple things they might not understand and for an expert that’s like second nature - they can still bring huge amounts of ideas to complex questions as shown in the bodystorming workshop.

Talking to experts that have opposing points of view due to their backgrounds is also key when dealing with projects like this. MFA Thesis Project 2020 Thesis Project MFA

It’s hard to talk about movement, but easy to ideate by moving

Workshop evolution: - 1st iteration of workshop as proof of concept - 2nd iteration of workshop to define final concept

Fun, energizing workshops, especially the ones that make you move around or act, encourage ideas without judgment

- 68 - NON VISUALS Sebastian Cabo Portugal de

Design Values in the Concept Examples

The concept examples that come out of the process not only showcase what kinds of things can come up from working with more sensory focused methods and design materials, but also shows how expansive these immersive technologies are and how little we explore the world of multimodal interactions, because if we did, our technologies would look very different than today. Some are more developed than the others, but even these smaller examples can show how many things can be reinvented and made more open for a bigger chunk of the population which is now being excluded. All these examples try to show different aspects of designing with NON-VISUALS: what happens when you completely take vision off for an interactive experience, how could this tech work (for VI users) to play outside ones’ house in a location based app, how could you control the capabilities of the software through gestures or movements or touching already owned objects and feel the feedback through non visual cues, and how would you interact with a connected

MFA Thesis Project 2020 Thesis Project MFA system like an OS.

The concepts come imbued with a series of design principles which were created at the start as a guide throughout the process. These had huge im- plications in the methods and design materials used/created in order to develop designs with these kinds of values in them, focusing on the form and expression of non-visual solutions using workshops like bodystorming and improv in real life and online through zoom; as well as in many of the other experiments - to put users in a situation and see how they improvise their way through it, to get insight into movement and sound patterns that can be used as a tried and tested base when designing multimodal interactions.

The reason for the experiments and workshops was to ideate and create with others in a way that kept everything open ended. This project didn’t start problem driven, it was about exploration, and it was more important for me to show a breadth of different areas and applications in the concept examples, than to choose one and go deeper on just that idea, to find the ideal solution to a problem. This is why the experiments feel a bit scattered although they are all connected by the thesis main thread - our understanding of the human senses.

- 69 - NON VISUALS VR Audiogame values This was the first concept that was developed, which came up early dur- Advantages for VI of VR vs desktop ing the process and worked like a kicking off point for the project, the first VR Audiogames are instantly better and more intuitive than classic Audi- workshops were done in order to experiment and ideate more on this con- ogames, this is due to the spatial nature of VR which we immediately get cept, although the insights gathered can be used in any multimodal inter- because it’s the world we live in everyday; when you move your head the action concept because it’s exploring the base of the technology, what can noises are positioned relative to you (vs having static sounds in the envi- you do when you take vision out completely, just darkness. How do you in- ronment), you can move physically in 3D space and get closer to these nois- teract with what you can’t see and how can you spot where the feedback or es (vs moving with a keyboard), and if you have a series of different mats on

sensory cues are coming from. the floor you could know when you reach the border of the play area, and Sebastian Cabo Portugal de where the center of the area is, as well as knowing where north, south, east The idea of not seeing anything and guide yourself just via the noises and and west are by feeling with your feet. Objects can even be added in order haptic feedback from the surrounding environment has been explored in to give even more haptic feedback and immersion, for example using a ba- many multimodal projects, mainly wayfinding for VI users. But it hasn’t nana as a gun or a selection of varied fruits to control a plane. been explored much when it comes to VR or AR because most of the focus is clearly on the visual side of things. Microsoft has done some research on Inclusion and Diversity these matters which can be seen in (pg.19 – Inspiration) but it’s very focused It’s also important to develop games like this because the games and en- on producing things specifically for VI users, which I think is super neces- tertainment industry needs more diversity in its main characters if the story sary, but the way I’m working is different because I include VI users from is linked to the gameplay, a cool blind hero can be created which is not only the beginning of the process and they’re always on my mind, while at the empowering for blind users; but it can also teach visual users about their same time I’m trying to see if visual users would also benefit or enjoy the own senses and help them connect with them once again, to realize how concept. This is why I wanted to explore the current VR capabilities instead much information flows through these channels that we think we ignore, of creating or adding extra devices like the direction the Blind Console that we are only guided by vision. This would make anyone empathize bet- team took, because then it would reach mainstream users (who are mainly ter with VI players and see how they can live their lives without vision users with sight) and it would enable VI and visual players to play together, everyone on the same playing field. Insights can be used across all concepts MFA Thesis Project 2020 Thesis Project MFA This concept was really helpful when exploring how to trigger a reaction or Exploring form and expression of non-visuals interest from the player by just using sounds or vibrations, which is the only The bodystorming workshop was very insightful in terms of understanding two other interaction modalities that can be used in current VR systems better the human senses and triggers, although the ideas that came out (outside of vision). Many of the insights gathered during the exploration and of the method were’t directly usable because the groups generated many development of this idea were used in every other concept, especially the ideas to play with others but not many for playing solo, which is what I was physicality apps in order to imagine the gestures the user would have to concentrating on to start simpler than with multiplayer games. Another make and what kind of feedback and it what modality to share the workload bodystorming workshop iteration could’ve given even more useful insights with as many senses as possible. if I directed the groups to create solo games where the player is blindfolded and the rest of the group are triggers around the player.

- 70 - NON VISUALS AR location based game values The first thing that was tried out specifically for this idea was to try out Explore the affordances that uses the natural capabilities of humans some bone conducting headphones in the middle of the city to test if they The interesting thing about Transreality games is that it uses real time would completely block all surrounding sounds (which would be a proble passing as a gameplay modality. It also feels like a more inclusive approach for VI users) or not; and the results are that they don’t block much, you can because the player can reach the location in any way they prefer, using still hear music or instructions at low volume without loosing environmen- whatever mode of transport they want, so they aren’t limited by their abil- tal cues. ities. Once they are in the right location, the game triggers and continues with the mission around that same area. It also makes you feel like a real

The dangers of playing outdoors when blind spy as it’s not just a thing you turn on or off, but spomething that subtly fol- Sebastian Cabo Portugal de Moving around while playing is obviously not very safe, we have seen many lows you throughout your day. examples of accidents happening whilst playing Pokemon Go, but for VI playing like this is even more dangerous than for the people who can see Use less tech devices but look at their phones instead of the path, as making someone cross the The game starts at home because it’s a place that everyone knows well, road when they shouldn’t could end up in a tragedy. This is the reason why it even VI users know exactly where everything is, removing all the dangers was decided to place the activities in the game in locations that can be safe that can happen when playing outdoors as Christy explained to me. It also for anyone to play - parks, plazas, avenues... uses common objects like cutlery as “spy tools" so as to not include more devices in order to play whilst making sure everyone will have that object at Exploring form and expression of non-visuals home and can take a picture of it, if the game AI recognises the object then It’s hard to ideate for non visual gameplay but anything generated in this it continues the mission. realm feels instantly like an innovation just because most gameplay in games are based on visual cues, although sound is gaining more and more This concept was the least developed as quarantine made it impossible to importance due to its immersion capabilities. Trying to imagine new inputs go outside and test the idea with users, but I believe it would work quite to play I came up with several ideas: playing with your voice by recieving a well, there is a new type of game which is starting to gain tracktion, walking call and chatting, recieving puzzles via text which you have to resolve and simulators (which I really enjoy), well this would be the version 2.0 of that then text back, take a picture of something specific like a piece of cutlery, idea. Although most of the insights came from the same place as for the VR MFA Thesis Project 2020 Thesis Project MFA scan the room by moving the phone around and listening to feedback (sim- audiogame, with the main difference that in this one you can traverse real ilar to searching for gold) to find where the invisible aliens are hidden or big spaces compared to the room sized levels in the VR audiogame, making where the bug has been planted in your house. everything more realistic and immersive, as you’re really moving through this digital game world. The ones that moved forward and were finally pre- You can even create decision trees (like many RPG games do) by making sented were Home OS and the physciality apps as I thought it would be the player decide whether to go to a place at a specific time or to another better to explore utilitarian apps compared to games, because a game can place to meet a different team at the same time, so you can’t meet both have a unique way of playing but when things are for day-to-day use people because they happen simultaneously. expect some intuitiveness in the gestures or actions they do.

- 71 - NON VISUALS Music Maker app values This was the first idea to come out from the physical app side of the con- Use less tech devices cepts. Like it was said before this was the lowest hanging fruit to start with I then carried out another workshop with some amateur musicians which as the connection between dance and music is with us since the begin- helped me develop the idea of the room sized music maker in detail, from nings of humanity. So creating a machine that uses movements as inputs its user journey to the moves required to mute a sound, for example. It to create sounds was the initial idea for the concept and then it developed started with the idea of creating a music maker through commonly owned into a more refined beat making and sequencer. Out of the three physicality objects like cutlery or fruit, and during the workshop with started with piec- apps this one is the most developed, right down to its micro interactions; es of LEGO and finding a way to build songs through that. Then we came

although it never reached the depth of apps like Logic Pro obviously due up with the idea of drawing sounds, so the cameras would pick up what you Sebastian Cabo Portugal de to time constraints, I think that many of the things that app does could be draw on a piece of paper and interpret that to create the music. translated into a multimodal interaction, although too many moves and controls would make it difficult to track without any visual aids but could These ideas were quickly forgotten once we started developing the full be simplified to the use of an LED light like Teenage Engineering use in the body, room sized version of the music maker. We realized there was a way to OP-Z product. grab the most basic building blocks/instruments to make a song – drums, snares, bass, chords… and used that limited amount of different instru- Create new design materials to think differently ment areas to be broken down across the room in a 2x4 grid, so in a certain It all started with the first sensorial experiment with the idea of breaking space you would tweak the drums and in the space next to it in the grid you things from a screen and positioning them across a room, in the case of the would tweak snares or in the space behind it you would tweak the bassline. experiments it was different sounds which would be linked to an object in The next step was to figure out how to choose different samples inside that the room. Having the sounds separated in space made it easy to still recog- area, and we decided to also dive de sounds in the space so when you move nize and remember which objects were in the room even if you never saw your arm around your head you get all the different samples and if you keep the object, just heard it and touched it. This then evolved into the idea of a your hand for some seconds in the same spot then you choose that sound room sized musical instrument. sample.

Explore the affordances that uses the natural capabilities of humans Exploring form and expression of non-visuals MFA Thesis Project 2020 Thesis Project MFA The next step was to try the idea out in a quick and rough way, and that’s In order to control the instruments there are two ways we tackled it: a sim- where the first workshop with Andreas (sound engineer and musician) was plified way where the user could play the instruments in a similar way to key, because we could very quickly choose some parameters from Logic their real life counterpart, or just dance to the rhythm and the machine will Pro and connect them to specific pieces of equipment or knobs which he follow the moves and timing; and a more choreographed version where could then use to tweak the sounds whilst I was moving. He would try and specific moves made specific changes in the sound. The first approach was link the sounds to my moves, side to side or up and down. Even though this further developed during the workshop and the choreographed approach was just a quick test, a lot of good insights came from it. For example, the was done in collaboration with Rachel Cheung a UK-based dancer and art- idea that the sounds could be connected to your speed and that it would all ist. During the rest of the workshop we figured out some general gestures be quantized to be on beat, so everything the user did would sounds musi- which could be used all around the system; like muting or removing sounds, cal and not just random noise – a cross between the simplicity of the Nin- rewinding, play and stop. These ideas were captured in a one shot video in tendo Labo keys and the song building capabilities of something like Logic order to keep as much of the information as possible which post its couldn’t Pro. handle like movement or sounds.

- 72 - NON VISUALS Music Maker app values (cont.) A song was created in order to develop the first prototype, and then bro- Designing for a richer sensory experience ken down into individual instruments to try and build it up again using the It also works on the inclusivity level as it’s more confusing to have all the new system instead, and see how that would work. Mainly shown through depth and capabilities in one screen simplified through a sea of menus, storyboarding and video sketching. Rachel was then given the song bro- than to have the parameters spread around in space and triggered depend- ken down in individual parts and then put altogether to see how she would ing on the moves made by the user, to learn how to use it one just simply try and play that same song live, with many instruments running at the moves around and discovers more and more of the software capabilities same time. The result was amazing and showcases the importance of hav- as their abilities increase. Also to keep with the inclusivity objectives, the

ing movement experts in the process as they are capable of imagining or path of simplified gestures was developed with inspiration from the cho- Sebastian Cabo Portugal de reaching moves that we never could. I also found it super interesting the reographed version from Rachel (using the simplest steps), so people from way she managed to play the song “live” with many instruments running at any age could get started and going immediately and discover the app in a the same time. Even though this was never the aim for us, it proved that the playful manner, by creating straight away. The interface is distributed in a app could also be used to play live in a music venue. She also spotted the way that is not so easy to do when you’re limited to visual design, and this is importance of having an instrument like this for both musicians and danc- where all the new opportunities arise. ers, to cross pollinate between each art. Musicians could now make music in a more playful way and with less pressure, to find inspiration; and dancers could now create music through their moves instead of the other way round which is what they usually do.

Breaking the boundaries of what we think is VR This concept really is a new way of thinking about VR, about being in an immersive experience that surrounds you and gets affected by your actions, the difference between this concept and the VR Audiogame is that here you can see your surroundings instead of having all the visuals off, but that doesn’t affect the outcome, whether you can see or not doesn’t change the MFA Thesis Project 2020 Thesis Project MFA music the user can making, putting everyone to play on the same level (you won’t get extra info from seeing).

- 73 - NON VISUALS Walk + Write app values This idea was the least developed of the three products but the way it came Use less tech devices out was via trying to find a “office type” application which could be trans- The idea of walking continually while you write came as an exploration of formed through multimodal interactions, how would they work outside of how someone’s walk could be used as an input, for example if you stop it a screen, if it was controlled through movement in space, gestures, voice makes a full stop and if you stop for longer it goes to the next paragraph, so commands, touching objects... I first tried to imagine how would MS Excel the software uses less tech devices and more human capabilities as sourc- look if it was multimodal but I’m not an expert in that software and it pre- es of input and ways to. sented the challenge that, similar to when translating a book into a movie,

Excel is very good at its use of 2D space through rows and columns and that Designing for a richer sensory experience Sebastian Cabo Portugal de gives it a very visual approach because it was created with that intention, The zoom workshop was really essential during the ideation and develop- but then to redesign it to be multimodal would mean a reimagining of what ment process of this idea because in that workshop we tried to commu- databases mean in this new context, what is it really a series a rows and nicate between each other by just using gestures, or sounds (no words) columns or could it be a range of objects in your house? or sometimes both, and there I discovered how some things which we commonly see every day or that are very iconic are easy to communicate Strengthen other senses and most people get the gestures quite fast, whilst anything uncommon or Moving away from Excel the next one I thought of was MS Word, and I im- that doesn’t happen in our lives so commonly would require more informa- mediately started thinking of a voice controlled Word. I was fully aware that tion than just the gestures, for example objects or sounds to accompany speech to text translators already exist and in are commonly used as acces- the movements. This is why I kept the gestures to a minimum, not only to sibility features. But I wanted to not only use the voice to control something simplify it for the user but also for the machine as it’ll have less ambigu- from afar (because that would make it like every other product out there) ous things to understand from the movements, and anything that falls out but also how the use of body language and gestures could add extra infor- of the gestures it knows won’t affect the writing (which is good because mation in the text, like making something italic, bold, or asking a question. some people gesture a lot while talking and it won’t affect the writing in Adding an element of mild exercise to the work to fight against the desktop weird uncontrollable ways). Exploring form and expression of non-visuals metaphor where we have to be sitting down in order to do anything. This concept really turns the desktop metaphor that to write you need to be sitting down over its head to encourage its users to walk and move in- MFA Thesis Project 2020 Thesis Project MFA Exploring the affordances that use the natural capabilities of humans stead, using your voice instead of your fingers to type as well as the rest Then I thought how my best ideas come whilst I’m walking normally, be- of your body and movement. Real multimodal ways of interacting with the cause of a shared effort where my brain is not just concentrated but where software that makes it a richer sensory experience for all, thus enabling VI my body is also engaged, so the brain is more relaxed to come up with ideas. users to also participate in the same way as everybody else would. So I decided to try this approach out and film the concept to show others in order to get feedback from them, and to my surprise this app was the most The values of inclusivity, richer sensory experiences and reducing the popular out of the three because everyone could see themselves walking amount of devices one needs to control VR or AR technologies are all shown around the office, their house or a park whilst writing emails and how that in this concept as well as flipping the norms to make the known unknown. would make their lives much better than how its currently done, they could see the value of the effect walking has on their inspiration and the fact that you’re moving around instead of sitting down to write was an eye opener for many people.

- 74 - NON VISUALS Home rehab app values This application was the developed in collaboration with two physiothera- Exploring form and expression of non-visuals pists - one in London and one in Gran Canaria, and with whom I made con- The meeting was super useful and enabled me to change the exercises tact in order to get more information on how our body suffers from being shown in the final video for one commonly used in physiotherapy – ask- sat down all day and how immersive technologies could help people with ing the user to lift their left arm and later their right arm to the same level problems like Alzheimer’s or cerebral palsy to continue their rehabilitation and then to make them open their eyes to see if their arms are in line (more in a way they can understand and carry out at home while they wait for their often than not, they aren’t). This is why closing one’s eyes is so important, net appointment. Whilst interviewing the physiotherapists I realized that because then there’s no excuse to look and copy someone else doing the

everything they were saying which made good therapy were the same val- exercise - to trigger the mirror brain. Sebastian Cabo Portugal de ues as I was trying out in my design approach: closed eyes, voice instructions, no physical touch or handholding, but with plenty of feedback on how Designing for a richer sensory experience you’re doing whilst carrying out the exercise. The hardest part to develop was the idea that there is no one correct answer or way of doing the exercises, so the system has to be quite flexible to be Use less tech devices able to train many different users with hugely different capabilities. There This communication line with the physiotherapists and this concept came is also no shame in not being able to complete a task, this is why therapists quite late in the process and by this point I hadn’t investigated much dur- won’t spot you like people do in the gym, because the work and motivation ing my experiments on body health, but there was one experiment that did has to come from oneself in order to recuperate. This is why the sounds inspire me: the first music maker workshop – we could use some of the follow the movements more than the moves follow the sounds. Again this same ideas, like connecting sounds to movements, in order to guide the approach is much more inclusive thanks to the richness in interactions and user through the therapy, as well as voice commands; this would be a way feedback than current methods, if you thought on how this could be done of giving continuous feedback which wouldn’t be too invasive and which in the normal “visual” way, users would probably get videos showing them could be also used to relax the user through playing meditative sounds. the actions they need to complete and the users would just copy what they see in the video. That is one of the biggest problems the physiotherapist Strengthen other senses had, that they copied movements instead of trying to do it themselves from The initial idea was then filmed in a small video sketch, and shown to one of scratch, and this is solved by having your eyes closed.

MFA Thesis Project 2020 Thesis Project MFA the physiotherapists to get feedback and see if the idea would provide value or at least make sense to them. First thing was to check if the exercises and the way it was giving feedback were correct, and here already many things changed – first the idea of the music, the sounds should be the same and not a continuation of a chosen song like it was initially proposed. The voiced feedback also needed some reworking as it had to happen more often than it currently was and it had to be more informative like saying “try lift your right arm 90 degrees” – try is the keyword in this situation as well as the detailed description of the movement so the user understands it easily and can do it back by themselves. Giving motivational feedback also changed a bit like “good job!” had to feel more human and personal - maybe even using the voice and way of describing things of your own therapist to make it even more familiar, making their space to rehab at home feel even safer. - 75 - NON VISUALS Home OS values After the apps were completed I was trying to find a way to link them to- But this also creates a problem for visual users too, as we get showered with gether through a menu system, to try and figure out how a non-visual menu visuals all-day every day to try and keep our attention in one app instead of would work like, but this quickly escalated into thinking about an operating another, stressing our brains whilst our bodies deprecate from being still system. This was just a beginning, as a full operating system is a huge task all the time, and making it harder for all of us to concentrate. On the other on its own, but enough to show some of the major differences and capabil- hand, having a surrounding environment, where we can walk around, full of ities of this new approach vs what we currently have on desktops, phones sounds and 3D objects that we can touch is our natural way of interacting or VR home screens. with the environment; so immersive tech like VR and AR should focus on

more senses than visual to become more natural and intuitive for all. Sebastian Cabo Portugal de Exploring form and expression of non-visuals The main inspiration for this concept came from reading and chatting with Designing for a richer sensory experience M Eifler, an MR designer now working at Microsoft Research who has cre- The main opportunities this new approach showcases in this example is for ated an immersive podcast app with three levels of interaction – wearable, contextually located apps/spatial interactions: were the user chooses are- holdable and habitable scale. It also came from the first sensory experi- as around the house and tags them as entertainment, work, relaxation are- ment where I started thinking about remembering objects in a room just as… so apps that aren’t included in those tags won’t work in that area (there by listening to the sounds they produce, like listening to an orchestra when will be some global tags or apps like the music player which can follow you you go into a room (where chairs could be strings, a sofa could be bass and around the house). This enables the user to fully concentrate on the mat- some table violins – all connected by their shape i.e. bigger objects could ter at hand instead of losing concentration constantly through some quick have deeper sounds) instead of a spoken list of objects. The communicat- eye candy. We can imagine these as the folders in iOS. For example, maybe ing through movement workshop was also very insightful and was the main you just keep entertainment apps in your room and meditation or relaxation reason why the system is on its first level controlled by voice. Asking peo- apps for when you go to sleep, and maybe your wardrobe triggers a clothes ple to gesture an object or a situation can get very complex when you ask shopping app (so you’ll see what you have already and what you need). someone to describe a situation that is not in our day to day life or that is not very iconic.

MFA Thesis Project 2020 Thesis Project MFA Exploring the affordances that use the natural capabilities of humans Currently we’re dictated by the desktop metaphor, which means that all our apps are in one place, a desktop or screen of some type and we’re sitting or standing still controlling all its capabilities with some fingers. This creates a bad situation for VI users, that even though they have screen readers, they have to move around this 2D system, which some blind people can find inconceivable after living all their life surrounded by 3D objects and sounds.

- 76 - NON VISUALS Home OS values Breaking the boundaries of what VR/AR means Use less tech devices There are three levels of interaction in the system (inspired by M Eifler), The user chooses objects in their house to store the apps, and the apps the first level is the simplest and is controlled through voice commands are triggered by simply touching the object. This is done through object through no GUI like Alexa, the second level could have a GUI even though in recognition technology so no additional devices have to be added to the my examples I chose to show a version with no visuals and this is triggered AR glasses or wireless spatial headphones. Not every object in the future by touching an object containing an app. This enables you to do medium has to have a chip on it, I believe this task is actually impossible due to the complexity tasks if the software enables it, and it can be controlled through limitations of our resources, even if it was possible it would lead to chaos;

the object being touched (an example could be introducing building a song imagine how much we will suffer if we’re already suffering having chips on Sebastian Cabo Portugal de with LEGO like in the workshop for the MUSIC MAKER app); and the third our phones. This is why I think object recognition is the best way to move level is where the full capabilities of an app can be shown, in the case of an forward into this idea of the Internet of Things (IoT). This way no extra elec- immersive app like MUSIC MAKER you would be surrounded by and inter- tronics would need to be embedded in chairs, tables or throwaway packag- active world in the same way as you would on VR, even though you can’t see ing, and it could still be made smart (talk to you or trigger an AR/VR app) by it all around you, but as soon as you move your body or in space, you’ll inter- simply recognizing the object and letting the user link it to a specific app. act with something and realize you’re surrounded by triggers that react to how you act. I feel that by taking visuals off everything feels more playful, as in you’ll have to play around with the app to learn it; tutorials for these apps would look more like teaching yoga moves than describing everything through writing like we have today. It’s still hard to teach a VI user how to use this system based on gestures but I feel the moves could be shown if they can see a bit, or described through voice: “make a fist with your hand and place it next to your head to stop the song”. MFA Thesis Project 2020 Thesis Project MFA

- 77 - NON VISUALS Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

7.0 Discussion

- 78 - NON VISUALS Is this better than what we had before?

The question now is not "is this better than that..." this is a time for exploration and we should feel free to go wild with these technologies to unlock their full potential and see what they’re really capable of, its use and application on real life problems comes later, once we understood the playing field we are dealing with. That is how it worked during my process, for example with the home rehabilitation application - it felt like everything the physiotherapists were telling me were the ideal conditions to carry this type of rehab, were exactly the same points I was looking at by exploring the technology without visuals. A perfect example of an application connecting the design principles and the exploration that had been done with the real needs of human beings.

One of the biggest potentials of technologies like VR is that one can live in someone else’s shoes, be closer than ever to try and see from their point of Sebastian Cabo Portugal de view and with this learn to empathize better with them. This project and the games/apps that came out of it, were a way to explore the technology to get users to empathize better with visually impaired people, but not in a “their lives must be so hard because they’re blind” way, just the opposite: for people with full visual capabilities to find again the power in their other senses, to reconnect with them, so they can realise that vision is not everything in life and it should not be everything in tech either, and the understanding they get through their other senses. Whilst at the same time empower blind users by putting everyone on the same playing field and letting them soar, and to expand the capabilities of our technology and ourselves in a more humane way, designing for all our senses and movement abilities. Because VR is not only a storytelling vehicle but also an empathy vehicle.

Currently our main measure of sucess is efficiency and ease of use, and even though we can agree with Dieter Rams principles of good design, there is more to design than pure functionality. Can we leave behind the quest for ultimate efficiency? Turn ourselves back into humans instead of machines that only require efficiency and the consumption of goods to be happy. We also need downtime and some space to think and relax in order to be balanced and mentally healthy, which is why my approach seems more like it’s focused on Slow Design than Efficient Design. In Home OS you have to move from room to room or touch a different object to open a new application and they would just be shown one at a time, letting our minds concentrate and leaving behind the trap of multitasking. So in some ways this approach will result in designs that are better than their current versions, but that depends on the context we want to situate it in and what part of our body or emotion we want to concentrate on.

This project started as a question about exclusion, in this case of visually impaired users, trying to find ways to include them into the technology; later

MFA Thesis Project 2020 Thesis Project MFA to find the most important realisation - the fact that we are all excluding ourselves, all of us no matter their sensory capabilities, by turning us into the big eyed alien shown in pg.24. This is what we should be working mostly on, on turning user-centered into human-centered design by thinking of people as more than just a pair of eyes and some fingers. I feel any advancemnt that we do towards making this happen willl mean our lives will be better, fuller, and feel more natural even in the unnatural world we are creating around ourselves because of it.

I expect these technologies that feel stuck today, to move into the things I’m exploring in this thesis because the novelty of realistic graphics will wear off soon enough, and as new applications and needs are found, and more exploration gets done, they will be created through new ways of thinking and doing. This is just the tip of the iceberg, and as I have stood on the shoulders of others to produce this, others should use whatever you can learn from this to continue the journey, one step at a time - towards a different, if not better future.

- 79 - NON VISUALS What’s changed in the design process?

We don’t have to reinvent the wheel, it’s hard to imagine what the design process of the future will look like, right now most of VR and AR applications are games, thus the method used to develop these is a sprint-based game development environment with its required expertise like: animation, programming, level design, writers, etc. But once we move from this phase into more work-related applications or day-to day use applications (what I’ve been calling utilitarian applications) I feel we will need a blend between the game development environment and the current user-centered design process, although in the case of this project it could be deemed human-centered due to the focus on our senses and human capabilities.

This new human-centered design for developing immersive technologies will require as much experitise from architects and from physio- Sebastian Cabo Portugal de therapists as from programmers or UX designers, due to the new affordances with movement of body and in space that these technologies give us; we need to include health experts to see how these applications could not only help our minds but our bodies too, it will also move away from flat graphic design into 3D spatial design and animation. So these are the kinds of experts I’ve been including in my explorative process and which have given the best result in terms of insights and understanding, knowledge about ourselves to design better for ourselves as human beings, not just users.

A starting point that is not concentrated on visuals but on our other senses was a great way to innovate and become more inclusive, as well as adding experts with different sensory capabilities from early on, so as not to retrofit solutions later on. Visuals can be added at any point throughout the process, it doesn’t always have to be the center of attention. If you first start with movement instead for example, you’ll end up with a much more dynamic and physical application than if you had started with visuals, get stuck in that space for most dev time and then forget the other senses exist.

Experience Prototyping and first-hand sensory exploration are key in order to understand and ideate better for these kinds of scenarios, as it has been pointed out before in this report it’s very hard to talk about movement but very easy to just start moving and doing. When things aren’t visual, you can’t just show them through a simple sketch so it’s better to fake the experience so the user can feel close to how it would be to use it, and get better insights than if they tried and imagine what you meant from a sketch on a post it or by explaining it verbally. The best way to show sketches in a project like this is through videos; once you’ve created the physical sketch you want, with the movements and sounds needed, the way to record it and relay that information with others is through quick and rough videos so they can be understood as sketches instead of a final product ad. MFA Thesis Project 2020 Thesis Project MFA Bodystorming and improv theatre workshops should be a common method used to create lo-fi prototypes even before we type a single line of code, with them you can ideate and play out the whole scenario you want to create and even test it with users and quickly improvise problems to find solutions on the fly. Bodystorming is Brainstorming for the senses.

- 80 - NON VISUALS Conclusions There’s a triad of things happening that make this project and its contributions more important than ever to carry out further into the future, and even adopt as the new iteration of this user-centered design approach we currently have; in order to really create products that are human-centered and to take the future into our own hands, instead of blindly following behind what technology dictates:

- First, the realisation that technology can be bad for us if designed against our human values and capabilities instead of working for them. The way we interact with technology nowadays, being stuck in front of screens for most of our days affect us not only mentally, as all the

workload goes to our brains, but this extremely sedentary lifestyle has obvious health risks - from obesity to back and neck pain,etc. We had Sebastian Cabo Portugal de to adapt to the machines we created instead of the other way round. So a better balance of workload between brain and body would help us in ways we can’t yet predict, but in mental and physical health terms it would improve them both.

- Second, that we’re moving our digital reality closer and closer into our physical reality until at one point in the future they will be blended into one continous reality where we are both physically and digitally there and available all the time at the same time, and this new reality is being developed through the advancements in the current technologies that have been explored during this thesis (VR/AR/IoT). So if this will be our new "physical/digital reality", we need to develop & use the technology in the most natural way to us, as humans, to break away from the problems caused by current tech. Finding ways to create NATURAL TECHNOLOGIES.

-Third, the technologies that we are developing have exactly the affordances we need (even though we’re not making the most of it yet) to make our technologies work more for us instead of against us, to use our full human capabilities - the movement in space and time, the ability to interact and hold 3D objects and an environment that surrounds us - are all things we are used to from our physical reality. So it would be wasteful for that technology to end up being used to create even more sedentary lives: sat down, where everything is triggering our visual senses so more workload goes into our brains, decreasing our concentration abilities due to all the data of the world being in front of our eyes continuously. If it’s already hard enough to concentrate using a computer or a phone without any apps that help you block things off, imagine MFA Thesis Project 2020 Thesis Project MFA this reality to the nth degree and this time you can’t just put the screen away, it’s inescapable because it will live with us, continuously surrounding us. Yes, you could take those AR glasses off and you wouldn’t see all the mess, but we could also chose what should be visual and what could be used or triggered through other senses.

I don’t believe in a single future like many predict where "Everything is gonna be in VR! We’ll turn into the Matrix". I know we will still have screens even after AR glasses become mainstream, we’ll still have cinemas in the same ways we still have analogue cameras in a world where digital photography is always at the tip of your fingers. The future is about plurality not singularity. With this project I’d just like for us to think better about what modality and sense we want to use for each case, should this be visual or tactile? maybe it can just be audio... To not end up in a world overwhelmed by visuals and ads, increasing our stress and decreasing our cognitive abilities, and instead live in a world that is fit for us as human beings, in the same way as nature is. Nature is fit for us because we come from it, and so technology should become more human because it comes from us.

- 81 - NON VISUALS Sebastian Cabo Portugal de MFA Thesis Project 2020 Thesis Project MFA

8.0 REFLECTIONS

- 82 - NON VISUALS Postface This project was really unexpected for me, at the beginning I thought it was all about showing im- pactful solutions which would make people think differently about how we do things, but by the end of it I realized that the impact, the real change, came from changing the process itself, from questioning our own design process paradigms in order to make the next iteration required for the future technologies to come. New design materials for a new reality.

One big question I kept asking myself is: wouldn’t this be much more annoying than the way we

do things now? which is so simple and effortless in terms of physical strain - efficient, but then all Sebastian Cabo Portugal de the strain is passed solely on to the mind. So even though these applications would be physically harder, without reaching the point of turning into working like a miner, the mental load would be diminished, the body and mind would share the strain of work and be more balanced as a result. BALANCE is the key in all of this.

This move from creating products and software all based on efficiency and quick response started with the idea of liberating time for the population to work less and enjoy more of what they wanted to do with free time; but instead of that what we did was fill up that free time with more tasks which could be done faster and faster - to make us more productive, which would then affect us mentally so much that in our free time all we wanted to do was lay in the sofa and not think. Our minds numb from the stress.

Maybe this approach of physicality will break away with our current health problems which come from being sat down all day typing in front of a screen, eyes, fingers, back and many other areas of the body and mind are currently suffering the consequences of these advances in efficiency that were brought by the way our technologies enable us to work and interact with them. MFA Thesis Project 2020 Thesis Project MFA The key to my process has been on two sides, the experience prototyping side where I did many experiments which weren’t purposefully designed to develop the concept ideas I had but it was to understand our body, movements and senses better in a general scale, so they could serve as input for any of the ideas as well as inspiration for newer ones.

The other main side was the interviews and chats with people who aren’t normally included in the design process when developing VR or AR applications. In this case it was visually impaired users, physiotherapists, dancers and design historians, as well as more common people like game designers and MR/VR designers. This enables the scope to be broadened hugely as well as bringing a more philosophical side to the process, helping to question the way things are done and why they are currently done this way. To show everyone the possibilities of it all.

- 83 - NON VISUALS References [1] - Hiroshi Ishii. 2008. Tangible bits: beyond pixels. In Proceedings of the [13] - Microsoft Innovation Stories – Project Torino, Allison Linn, 2019 - 2nd international conference on Tangible and embedded interaction (TEI https://news.microsoft.com/innovation-stories/project-torino-code-jumper/ ’08). Association for Computing Machinery, New York, NY, USA, xv–xxv. [14] - M. Mott, E. Cutrell et al. Accessible by Design: An opportunity for Vir- DOI:https://doi.org/10.1145/1347390.1347392 tual Reality, Microsoft Research, 2019 - https://www.microsoft.com/en-us/re- [2] - Salih, Dilman. 2015. Natural User Interfaces. University of Birmingham search/publication/accessible-by-design-an-opportunity-for-virtual-reality/ [3] - Thieme, Anja & Bennett, Cynthia & Morrison, Cecily & Cutrell, Ed- [15] - Y. Zhao, C. Bennett et al. Enabling People with Visual Impairments

ward & Taylor, Alex. (2018). “I can do everything but see!” -- How People to Navigate Virtual Reality with a Haptic and Auditory Cane Simulation, Mi- Sebastian Cabo Portugal de with Vision Impairments Negotiate their Abilities in Social Contexts. 1-14. crosoft Research, 2018 - https://www.microsoft.com/en-us/research/publica- 10.1145/3173574.3173777. tion/enabling-people-visual-impairments-navigate-virtual-reality-haptic-au- [4] - Asifa Majid, Seán G. Roberts, et al. - Differential coding of perception in ditory-cane-simulation-2/ the world’s languages. Proceedings of the National Academy of Sciences, [16] - https://andreasrefsgaard.dk/project/sound-controlled-wolfenstein-3d/ 2018; 201720419 DOI: 10.1073/pnas.1720419115 [17] - Augmented Reality: Zach Lieberman’s audio in space. YouTube. 2018. [5] - Is there a universal hierarchy of human senses? ScienceDaily.com. 2018 - https://www.youtube.com/watch?v=RPCB4GRJ9xc - https://www.sciencedaily.com/releases/2018/11/181105160852.htm [18] - Christy Smith, Meet Christy! caniplaythat.com, 2020 - https://caniplay- [6] - Lawrence de Martin, What senses do we use most and least? Quora. that.com/2020/02/08/meet-christy/ com, 2017- https://www.quora.com/What-senses-do-we-use-the-most-and- [19] - Milgram, Paul; H. Takemura; A. Utsumi; F. Kishino (1994). “Augmented least Reality: A class of displays on the reality-virtuality continuum” (PDF). Pro- [7] - Anil Dash, the old boys club is for losers, 2007 - http://anildash. ceedings of Telemanipulator and Telepresence Technologies. pp. 2351–34. com/2007/02/23/the old boys cl/ Retrieved 2007-03-15. [8] - Pascolini D, Mariotti SP. Global estimates of visual impairment. 2010. Br. [20] - Gartner Hype Cycle. 2009. Gartner Firm - https://www.gartner.com/en/ J. Ophthalmol. Dec.2011 research/methodologies/gartner-hype-cycle [9] - Quillen DA, Common causes of vision loss in elderly patients, 1999 - [21] - Marion Buchenau and Jane Fulton Suri. 2000. Experience prototyp- MFA Thesis Project 2020 Thesis Project MFA https://www.ncbi.nlm.nih.gov/pubmed/10414631 ing. In Proceedings of the 3rd conference on Designing interactive sys- [10] - How I play Diablo 3 without seeing the game, YouTube, 2018 - https:// tems: processes, practices, methods, and techniques (DIS ’00). Association www.youtube.com/watch?v=S2afeyEBtcw for Computing Machinery, New York, NY, USA, 424–433. DOI:https://doi. [11] - Blind gamer gets round 28 solo on Black Ops 3 Zombies on Nacht org/10.1145/347642.347802 Der Untoten, video description. YouTube, 2018 - https://www.youtube.com/ [22] - You’re doing mixed reality wrong, M Eifler, Medium.com. 2017 - https:// watch?v=4msqc4iXkoQ&t=466s medium.com/@blinkpop/youre-doing-mixed-reality-wrong-d32aa54ae8af [12] - Guide on how to play World of Warcraft blind, Reddit, 2019 - https:// [23] - Rec Room Has An In-World, Spatial, Embodied, Full VR Programming www.reddit.com/r/wow/comments/9w7mr1/guide_on_how_to_play_world_ Language Called Circuits, New World Notes, 2019 - https://nwn.blogs.com/ of_warcraft_blind/ nwn/2019/09/rec-room-social-vr-evelyn-eastmond-spacialized-programming.html

- 84 - NON VISUALS References IMAGE CREDIT [24] - N. Schaffert, K. Mattes et al. A Review on the Relationship Between Introduction cover - Victorien Ameline - unsplash Sound and Movement in Sports and Rehabilitation, 2019 - https://www.fron- Background cover - Lux Interaction - unsplash tiersin.org/articles/10.3389/fpsyg.2019.00244/full Approach + methods cover - David Werbrouck - unsplash [25] - Torball General Information, International blind sports federation web- Material exploration cover - Gjon Mili - stop motion photography site - http://www.ibsasport.org/sports/torball/ Concept examples cover - Kazuend - unsplash [26] - John Roach, Using AI, people who are blind are able to find familiar Analysis cover - Data sketch using d3.js

faces in a room, Microsoft Innovation Stories, 2018 - https://news.microsoft. Discussion cover - Gabriel @natural - unsplash Sebastian Cabo Portugal de com/innovation-stories/project-tokyo/ Reflections cover - Abstract water ripples background - public domain [27] - Facing Emotions, helping the visually impaired sense emotions, Hua- wei True Stories - https://consumer.huawei.com/uk/campaign/truestories/ facingemotions/ [28] - Body.Processing by Alvin Arthur in Vimeo. 2018. - https://vimeo. com/296007218 [29] - Why do we like to dance--And move to the beat? , Scientific American, 2008 - https://www.scientificamerican.com/article/experts-dance/ [30] - T.Winkler, Making Motion Musical: Gesture Mapping Strategies for In- teractive Computer Music, Brown University, 1995 - https://www.brown.edu/ Departments/Music/sites/winkler/research/papers/making_motion_musi- cal_1995.pdf [31] - Trans-Reality Gaming, Craig A. Lindley, Institution Technology, Art and New Media, University of Gotland, Visby, Sweden (2004) [32] - Bonsignore, Elizabeth M., et al. “Mixed reality games.” Proceedings of MFA Thesis Project 2020 Thesis Project MFA the ACM 2012 conference on Computer Supported Cooperative Work Com- panion. ACM, 2012. doi:10.1145/2141512.2141517

- 85 - NON VISUALS Sebastian Cabo Portugal de

This is not a framework or a guideline, these findings aren’t a solid truth and are subject to change as the process continues to evolve. MFA Thesis Project 2020 Thesis Project MFA This is an exploration of design materials to inspire future designer to think holistically when designing for VR/AR.

- 86 -