NON VISUALS MATERIAL EXPLORATION OF NON VISUAL INTERACTION DESIGN MFA MASTER THESIS 2020 SEBASTIAN DE CABO PORTUGAL NON VISUALS Sebastian de Portugal Cabo Sebastian “To understand something is not to be able to define it or describe it. Instead taking something that we think we already know and making it unknown thrills us afresh with its reality and deepens our understanding of it” – Kenya Hara MFA Thesis Project 2020 Thesis Project MFA - 2 - NON VISUALS Forward This project really scares me, and that’s why I had chosen to do it, I didn’t want to do something that I had done before, I wanted to try new technologies and new design methods that were completely novel for me. This is why I chose to work with VR and AR technologies, something I have barely touched before. I wanted to really understand how to build things for these kinds of technologies, and more importantly find ways to make prototypes and lo-fi iterations without having to do massive amounts of code, or even no coding at all. de Portugal Cabo Sebastian This is also why I worked with audio, something I’ve done sporadically during filmmaking but has always been my weak spot due to my heavy focus on the visual side of things. Ironic I know. Working with visually impaired people is a user group I haven’t collaborated with yet and this also scared me, from an ethical and empathetic point of view and from a pragmatic one, for example how do I show people concepts or initial sketches if they can’t see a sketch. Finding non visual ways to communicate my ideas. “This is not a problem driven project or a technology driven project, but an experiment driven one” Normally the projects we’ve done in the past dealt with a specific problem or user group and in the end we could evaluate against these points of contact to see if the outcome had been successful or not. - if we had "solved" the problem at hand. But on this project I wanted to try a research through design method, a more experiential process, one where you can have many different starting points, even op- posing ones, and you develop the research and concepts through making - its use will come out after. MFA Thesis Project 2020 Thesis Project MFA Even though everything about this project scares me, and I can’t imagine what the outcome will be, I feel confident that something good will be produced, I have to learn to trust the process, and myself as a designer, and that is the main reason I wanted to put myself in this situation, to prove myself I can get out of any situation successfully even if I don’t know anything about the subject at all at the beginning of it. In the future I won’t be able to choose exactly what I’ll be working on so I like to train myself to be prepared for any situation and find ways to improvise my way out. I just couldn’t predict how the world would change in the short time this thesis was being produced, and how challenging that was going to turn out to be. This is why I chose to work on a topic that scared me most as my final project in this master’s degree. Sebastian de Cabo Portugal - 20/01/2020 - 3 - NON VISUALS Abstract Design is all about visuals, or that is what I have found out during this thesis, from the pro- cess materials to the outcome our main entry point to any problem is how will we solve it visually so it’s understandable for the general user. This aspect is problematic in itself due to the fact that we, as humans, understand the world and the things around using all our senses continuously, even though we can forget as visuals are so overpowering. There is a huge opportunity area in exploring our other senses and bringing them back to technology, and this can be seen in works in the past like Tangible Interactions [1] or de Portugal Cabo Sebastian Natural User Interfaces [2]. But in this moment in time, where all these new technologies like VR/AR and IoT are about to enter our lives and change them forever, this topic is more important than ever. We have already seen what happens when we turn humans into mere machines with some fingers as interactive inputs, and barely any senses to process all the information given to us. Now that these technologies are still young and malleable, we can direct the future to where we want it instead of being guided by the technology itself. To do this we need to reimagine the design process, not reinvent the wheel, but add ex- perts which we currently leave behind and I argue are key to unlock these technologies, experts not only of the technological side of things but on the human side too, like physi- otherapists and dancers. Add also people who we never think about when we think of VR like visually impaired users, which could make these technologies inclusive since early on, insteadof as an afterthought like we usually do. Not only people, but we also need to add new materials to understand how we use our senses and explore ways that we can under- stand and explore them differently; like bodystorming and improv theatre because when things aren’t visual, how do you sketch it? A sketch turns into a video about movement. MFA Thesis Project 2020 Thesis Project MFA The end result provides a wide breadth of examples of the types of innovations that can come out of using these new design materials, and to open new frontiers. From a VR game with no visuals whatsoever to an AR location based story game, to a home sized multi- modal operating system containing several different apps controlled through physical movement. The examples open up the space instead of closing into a single solution. This is just the tip of the iceberg, a hope that others will be inspired by it and continue with this journey that has just started, to guide the future into one that is more technological and at the same time more human than ever before. What we know is that VR does not equate Visual Reality. Keywords – immersive environments, virtual reality, mixed reality, wearable computing, virtuality continuum, internet of things - 4 - NON VISUALS Contents FOREWARD -3- 4.0 MATERIAL EXPLORATIONS -27- ABSTRACT -4- - Working in Sprints based on Design Principles -28- CONTENTS & ACKNOWLEDGMENTS -5/6- - How a Loop looks from the inside -29- EXPECTATIONS & ASSUMPTIONS -7- - My Activities -30- 1.0 INTRODUCTION -8- INTERVIEWS Sebastian de Portugal Cabo Sebastian - Uniting digital and physical worlds -9- - Interview - Luca Contato -31- - Ethical Considerations -10- - Interview - Christy Smith -32- - Philosophical Considerations -11- - Interview - Christy Smith (cont.) -33- - Interview - M Eifler -34- 2.0 BACKGROUND -12- - M Eifler Studio Metaphor -35- - The world we come from/the world to come -13/14- - M Eifler 3 levels/scales of interaction inside VR -36- - Focusing on visually impaired users -15- - Interview - Arianna Ortelli -37/38- - Audiogame or be prepared to grind hard -16- - Interview - Enrique Sanchez -39- - How are we adding senses to VR? Add more tech -17- - Interview- Pablo Gody and Nelson Sanchez -40- - Why read a book when you have audiobooks -18- - Inspiration -19- EXPERIMENTS - Inspiration from Microsoft Research -20- - Playing with existing tech - Amazon Alexa -41- - Games, an excuse to improve the whole tech -21- - Playing with existing tech - VR games -42- - Playing with existing tech - VR games -43- 3.0 APPROACH + METHOD -22- - Playing with existing tech - Torball -44- MFA Thesis Project 2020 Thesis Project MFA - Insights from background research -23- - Experiment 1 - Spatial awareness through music -45/46- - Flip the norms - "make the known unkown" -24- - Experiment 2 - Playing fussball blindfolded -47- - DESIGN PRINCIPLES/Perfect Timing -25- - Experiment 3 - Controlling Logic Pro with moves -48- - Research through design -26- WORKSHOPS - Workshop 1 - Music Maker with Toby & Julien -49- - Workshop 2 - Physical Game Jam/Bodystorming -50/51- - Suddenly the world changed -52- - Workshop 3 - Communicating through movement -53/54- - 5 - NON VISUALS Acknowledgments 5.0 CONCEPT EXAMPLES -55- This thesis wouldn’t have been possible without the help and support from: - Introduction to Provocation & Examples -56- - My parents, for always being close even when I’m so far away - Playing games without vision - VR Audiogame -57- - My supervising tutor Monica Lindh-Karlsson - Playing games without vision - AR location based -58- - My thesis supervisor Stoffel Kuenen - Using our own bodies/movements/senses/objects - The best class anyone could ask for, my IXD colleagues - The rest of UID for always being up to try one of my weird experiments as controllers/input- Home OS -59- - Amanda Stendahl and the Torball group in Umeå - Using our own bodies/movements/senses/objects - RISE Interactive studio Umeå de Portugal Cabo Sebastian as controllers/input- Physicality apps -60- - Luca Contato from Rising Pixel - Physicality apps - Music Maker through movement -61- - Christy Smith from caniplaythat.com - Physicality apps - Walk + Talk -62- - M Eifler from Microsoft research - Physicality apps - Home Physical Rehabilitation -63- - Arianna Ortelli from the Blind Console - Laura Vendrell from VRave 6.0 ANALYSIS -64- - Enrique Sanchez from Orbisnauta - Insights from explorations -65- - Pablo Godoy Physiotherapist at NHS Trust - How to develop a concept -66- - Nelson Sanchez Physiotherapist at Fisiotec - Example in use: Music Maker -67- - Julian Loretz from APD - Julien Desvaux from IDI - Design methodology insights -68- - Rachel Cheung from Goldsmiths University Design values in the Concept Examples -69- - Javier Viera aka MakerFly - VR Audiogame values -70- - Teenage engineering