MULTI-SENSORY MATERIALITY Expanding Human Experience and Material Potentials with Advanced HoloLens Technologies and Emotion Sensing Wearables MARCUS FARR1 and ANDREA MACRUZ2 1American University Sharjah / Tongji [email protected] 2Tongji University [email protected] Abstract. What is the current state of human/material perception relative to advanced architectural technology? What sensory experiences are possible, and how are they designed and deployed? What happens when advanced HoloLens technologies are used in conjunction with wearable emotion-sensing technologies to connect people with deeper sensory experiences relative to materiality and space? Does this offer a heightened pedagogical perspective when teaching architecture? This paper responds to these questions by expanding on and critiquing a small scale digitally augmented project created in an academic setting. The project focuses on relationships between technology and human sensory experience relative to specific augmented and sensorial engagements. It employs an overlap of HoloLens technology to make and enhance the design experience and wearable emotion sensors to evaluate the human experience. Keywords. Technology; Sensory; Materiality; Augmented. 1. Introduction The work explores what it means to design spaces with high degrees of sensory feedback and potential. It is made in tandem with technology using Rhino, Houdini, Fologram, and HoloLens as a primary decision-making tools relative to the creation of space. The Rhino plugin Fologram extends the experience by offering a user an additional sensory experience while wearing the HoloLens headset or by using the iPhone app. To monitor and evaluate the reaction to the projects, users also wear emotion-sensing UpMood bracelets that collect biodata from the user and result in 11 different emotional states, stress level and BPM, every 1.5 to 3 mins. This project evaluates the use of these converging technologies as a way to improve our understanding of how we design for a deeper human experience. Through this process, the paper discusses user insight into emotional patterns and management. The resultant methodology and outcomes offer a new theoretical agenda for using technology in the architecture/teaching process and creates a conversation that critiques our current design process in academia. RE: Anthropocene, Proceedings of the 25th International Conference of the Association for Computer-Aided Architectural Design Research in Asia (CAADRIA) 2020, Volume 1, 721-730. © 2020 and published by the Association for Computer-Aided Architectural Design Research in Asia (CAADRIA), Hong Kong. 722 M. FARR AND A. MACRUZ With this small augmented installation, the paper follows the introductions of overlapping augmented reality into the design space by pairing emotional sensing wearables with Microsoft HoloLens technology as an extension to the traditional architecture/build workflow. The use of these technologies offers an expanded mode of inquiry relative to a host of architectural potentials, dialogue, critique and a deeper sense of concept development. This process also lends itself to teaching as a new pedagogical model whereby students and teachers can discuss project potential in real-time and in context relative to a given sensory-material scenario. The benefits include real-time simulation of scale, form, color, light, shadow, and the experimentation of these as they relate to a host of different material situations. Working in teams to share interactive experiences is also a profound advantage for teaching in the architecture profession. Moving further, this paper documents and critiques the process and results from the projects, which took place in 2019, and explores these concepts relative to digital design and material interaction in the field of architecture and constructed environments. It begins by designing with typical parametric & computational software as an exercise, and then moves directly from the digital model to the process of construction. Instead, the project used Fologram to portray a full-scale hologram in a physical artspace as a digital projection or as an image on an iPhone. Its purpose is to engage people with multi-sensory experiences that force humans to re-evaluate and re-perceive our material world. Adding to the experience are additional sensory stimulations that can be experienced when wearing the HoloLens headset. These experiences are subsequently monitored in real-time by using emotion sensors that are designed to collect biodata from heart rate and contractions through a PPG sensor (i.e. photoplethysmogram - used to detect blood volume changes in the microvascular bed of tissue). Algorithms turn the data from human pulse waves into analytics to generate a discussion of interactive stress levels, heart rate, vitality levels, and emotions. Figure 1. Fologram iPhone integration; Microsoft HoloLens augmentation. Using this project as a vehicle, the paper goes on to theorize an agenda that is rather interested in supplementing our design process by new tools and advanced technologies that allow us to experiment in extremely informed ways relative to the perception of physical context. This heightened mode of inquiry proposes a new MULTI-SENSORY MATERIALITY 723 way of looking at the process of designing full-scale projects by interacting with materials in a unique sensorial way. Because of this, it questions the relevance of current material conversations and proposes further discussions involving student, teacher, architect and client. 2. Sensory and Perception “Materials are the flesh and bones of objects and buildings. Glass, Wood, and silicone breathe, shift, and sigh. A curtain or bench absorbs sound. Light bounces, reflects, and floods the periphery - light is not a thing, but it changes everything. Textures speak to the eye as well to the hand, incising flat surfaces with real or imaginary depth where the restless gaze can wander. Sensory design considers materiality across multiple dimensions, from the visible to beyond.” -Lupton, Ellen, Lipps, Andrea, “The Senses: Design Beyond Vision”, 2018. In this project, it was important to understand materiality by creating a more compelling relationship with the potentials of sensory perception and technology in architectural space. To do this, the installation offers tactile, olfactory, and auditory interactions that enhance the aggregation of surface material and patterning on an otherwise normative architectural surface. The surface patterning bends and flows along the course of 6.5 meters (1.8 meters height) meters offering a user the opportunity to come into contact with them along a particular path of circulation or as a random encounter. To contextualize the experience, salt crystals are used as the primary material reference. Salt and the generation of crystals by this material offers a alternate material palette that is extremely texturial and has a unique interaction with light. This is extremely meaningful in that it embodies powerful symbolic associations with collective memory, ritual, landscape and purification. Because of our context, living and practicing in the desert, salt and sand are common materials that are very much a part of daily life in our region, however these materials are seldom used or explored in an architectural way. With this experience, users can perceive the space differently, giving meaning and value to its physicality through embodied experiences and an expanded notion of spatial awareness. In this, the wall is occupying a physical and conceptual place in the space of architecture. It is given a new importance, one that is not merely structure or partition, but one that is designed to play a larger role in the experience of architecture, beyond the normative, beyond the visual. It is offering a fresh starting point for architecture by redefining the point of origin from one that is specifically for partition and structure, to one that is positioning itself within a broader set of human surface and sensory relationships. For us to revise and redefine our material perception, it is essential that our experience with material occurs deeply and intensely. This notion of materiality calls for an embodied response, and it calls for the architect to move beyond the current definitions of what it means to make a something as mundane as a wall. The glorification of architecture is often presented through the concept of material craft, but not as often does it include the potentials of the “sensorial craft”. This is largely given to theory and speculation and rather than being 724 M. FARR AND A. MACRUZ present in architectural space, and many times are presented in the form of art installations or gallery experiences. This paper argues that the need for sensorial craft is one that is intrinsic to human nature and also a part of material craft. Our relationships to touch, smell, sound and sight are, in fact, part of our search for a deeper understanding of place and meaning. The richness of this architecture is created with the encounter between the tangible aspects of architecture (brick, stone, glass, etc.) with the intangible (light, wind, sound, etc.). What sensory experiences are possible, and how are they designed and deployed? In the book, The Air From Other Planets, A Brief History of Architecture to Come, author Sean Lally positions architecture as a discipline in need of a re-centering, arguing that instead of expanding on the traditional concepts of architecture as a container, the discipline