MULTI-SENSORY MATERIALITY

Expanding Human Experience and Material Potentials with Advanced HoloLens Technologies and Emotion Sensing Wearables

MARCUS FARR1 and ANDREA MACRUZ2 1American University Sharjah / Tongji [email protected] 2Tongji University [email protected]

Abstract. What is the current state of human/material relative to advanced architectural technology? What sensory experiences are possible, and how are they designed and deployed? What happens when advanced HoloLens technologies are used in conjunction with wearable emotion-sensing technologies to connect people with deeper sensory experiences relative to materiality and ? Does this offer a heightened pedagogical perspective when teaching ? This paper responds to these questions by expanding on and critiquing a small scale digitally augmented project created in an academic setting. The project focuses on relationships between technology and human sensory experience relative to specific augmented and sensorial engagements. It employs an overlap of HoloLens technology to make and enhance the experience and wearable emotion sensors to evaluate the human experience. Keywords. Technology; Sensory; Materiality; Augmented.

1. Introduction The work explores what it means to design spaces with high degrees of sensory feedback and potential. It is made in tandem with technology using Rhino, Houdini, Fologram, and HoloLens as a primary decision-making tools relative to the creation of space. The Rhino plugin Fologram extends the experience by offering a user an additional sensory experience while wearing the HoloLens headset or by using the iPhone app. To monitor and evaluate the reaction to the projects, users also wear emotion-sensing UpMood bracelets that collect biodata from the user and result in 11 different emotional states, stress and BPM, every 1.5 to 3 mins. This project evaluates the use of these converging technologies as a way to improve our understanding of how we design for a deeper human experience. Through this process, the paper discusses user insight into emotional patterns and management. The resultant methodology and outcomes offer a new theoretical agenda for using technology in the architecture/teaching process and creates a conversation that critiques our current design process in academia.

RE: Anthropocene, Proceedings of the 25th International Conference of the Association for Computer-Aided Architectural in Asia (CAADRIA) 2020, Volume 1, 721-730. © 2020 and published by the Association for Computer-Aided Architectural Design Research in Asia (CAADRIA), Hong Kong. 722 M. FARR AND A. MACRUZ

With this small augmented installation, the paper follows the introductions of overlapping augmented reality into the design space by pairing emotional sensing wearables with Microsoft HoloLens technology as an extension to the traditional architecture/build workflow. The use of these technologies offers an expanded mode of inquiry relative to a host of architectural potentials, dialogue, critique and a deeper of concept development. This process also lends itself to teaching as a new pedagogical model whereby students and teachers can discuss project potential in real-time and in context relative to a given sensory-material scenario. The benefits include real-time simulation of scale, form, color, light, shadow, and the experimentation of these as they relate to a host of different material situations. Working in teams to share interactive experiences is also a profound advantage for teaching in the architecture profession. Moving further, this paper documents and critiques the process and results from the projects, which took place in 2019, and explores these concepts relative to digital design and material interaction in the field of architecture and constructed environments. It begins by designing with typical parametric & computational software as an exercise, and then moves directly from the digital model to the process of construction. Instead, the project used Fologram to portray a full-scale hologram in a physical artspace as a digital projection or as an image on an iPhone. Its purpose is to engage people with multi-sensory experiences that force humans to re-evaluate and re-perceive our material world. Adding to the experience are additional sensory stimulations that can be experienced when wearing the HoloLens headset. These experiences are subsequently monitored in real-time by using emotion sensors that are designed to collect biodata from heart rate and contractions through a PPG sensor (i.e. photoplethysmogram - used to detect blood volume changes in the microvascular bed of tissue). Algorithms turn the data from human pulse waves into analytics to generate a discussion of interactive stress levels, heart rate, vitality levels, and emotions.

Figure 1. Fologram iPhone integration; Microsoft HoloLens augmentation.

Using this project as a vehicle, the paper goes on to theorize an agenda that is rather interested in supplementing our design process by new tools and advanced technologies that allow us to experiment in extremely informed ways relative to the perception of physical context. This heightened mode of inquiry proposes a new MULTI-SENSORY MATERIALITY 723 way of looking at the process of designing full-scale projects by interacting with materials in a unique sensorial way. Because of this, it questions the relevance of current material conversations and proposes further discussions involving student, teacher, architect and client.

2. Sensory and Perception “Materials are the flesh and bones of objects and buildings. Glass, Wood, and silicone breathe, shift, and sigh. A curtain or bench absorbs sound. Light bounces, reflects, and floods the periphery - light is not a thing, but it changes everything. Textures speak to the eye as well to the hand, incising flat surfaces with real or imaginary depth where the restless gaze can wander. Sensory design considers materiality across multiple dimensions, from the visible to beyond.” -Lupton, Ellen, Lipps, Andrea, “The : Design Beyond Vision”, 2018. In this project, it was important to understand materiality by creating a more compelling relationship with the potentials of sensory perception and technology in architectural space. To do this, the installation offers tactile, olfactory, and auditory interactions that enhance the aggregation of surface material and patterning on an otherwise normative architectural surface. The surface patterning bends and flows along the course of 6.5 meters (1.8 meters height) meters offering a user the opportunity to come into contact with them along a particular path of circulation or as a random encounter. To contextualize the experience, salt crystals are used as the primary material reference. Salt and the generation of crystals by this material offers a alternate material palette that is extremely texturial and has a unique interaction with light. This is extremely meaningful in that it embodies powerful symbolic associations with collective memory, ritual, landscape and purification. Because of our context, living and practicing in the desert, salt and sand are common materials that are very much a part of daily life in our region, however these materials are seldom used or explored in an architectural way. With this experience, users can perceive the space differently, giving meaning and value to its physicality through embodied experiences and an expanded notion of spatial awareness. In this, the wall is occupying a physical and conceptual place in the space of architecture. It is given a new importance, one that is not merely structure or partition, but one that is designed to play a larger role in the experience of architecture, beyond the normative, beyond the visual. It is offering a fresh starting point for architecture by redefining the point of origin from one that is specifically for partition and structure, to one that is positioning itself within a broader set of human surface and sensory relationships. For us to revise and redefine our material perception, it is essential that our experience with material occurs deeply and intensely. This notion of materiality calls for an embodied response, and it calls for the architect to move beyond the current definitions of what it means to make a something as mundane as a wall. The glorification of architecture is often presented through the concept of material craft, but not as often does it include the potentials of the “sensorial craft”. This is largely given to theory and speculation and rather than being 724 M. FARR AND A. MACRUZ present in architectural space, and many times are presented in the form of art installations or gallery experiences. This paper argues that the need for sensorial craft is one that is intrinsic to human nature and also a part of material craft. Our relationships to touch, smell, sound and sight are, in fact, part of our search for a deeper understanding of place and meaning. The richness of this architecture is created with the encounter between the tangible aspects of architecture (brick, stone, glass, etc.) with the intangible (light, wind, sound, etc.). What sensory experiences are possible, and how are they designed and deployed? In the book, The Air From Other Planets, A Brief History of Architecture to Come, author Sean Lally positions architecture as a discipline in need of a re-centering, arguing that instead of expanding on the traditional concepts of architecture as a container, the discipline should move to one that is more focused on concepts of energy and amplifications of spatial energies as the primary means of making decisions about space. He writes that the potential futures of design are already all around us, the physical and sensorial, and can be found in sensorial “microclimates, shadows, lighting, vegetation”, but there is a fundamental problem in the way architects design with this information because we are addicted to lines that represent walls, columns, and floors. The answer instead is to re-imagine what and how we draw and what sensorial feedbacks are already available to us, embracing the technologies that offer a heightened sense of place.

3. Touch, Sound and Smell The project achieved sensorial experiences through expanded notions of texture. To do this, a pattern 6.5 meters in length, was laser cut into 32 sheets of flat material and mounted to a wall. The paper material was mixed with different densities of salt to create the texture. The size and scale of this was important because it offered the users an opportunity to experience different densities and orientations of pattern and texture. The material was further covered with different densities of crystallization coatings, and salts that changed the weight, texture, and density of the surface. Further, layers of different colors were combined, creating contrasting light to the textures and new color interactions throughout the existing layers. This combination offered a noticeable sense of tactility and visual intricacy, not commonly found on wall surfaces. In the book, The Eyes of the Skin, Juhani Pallasma asks the far-reaching question why, when there are five senses, has one single sense, sight, become so predominant in architectural culture and design. The book moves beyond concepts of phenomenology and discusses the need for deeper dimensions of human experience in architecture. The author goes on to write, “the flatness of today’s standard construction is strengthened by a weakened sense of materiality.” In most architecture, this flatness is pervasive, resultant of standardizations and ease of shipping and manufacturing. Many times decisions are made based on the materiality of convenience. This project, of course, critiques these decisions by asking new questions and asking for new responses. This installation presents a comparison in contrast, combining the perception MULTI-SENSORY MATERIALITY 725 of visual texture with the tactility of the physical. Because of this, as a user touches the surface, they feel the harsh tactility of the salt crystals, along with the softness of its visual texture. Moreover, as Neal Leach questions in his book Camouflage, “If we have this natural urge to assimilate to our environment, what role might the environment play in facilitating that process?” In our process of assimilation, the identity of the self through differentiation and identification with the environment is enhanced when a multi-sensory approach takes place.

Figure 2. Tactile surface with salt; Installation with multi-sensory surface.

According to the legendary Randy Thom, Director of at Skywalker Sound on why sound matters: “sound is important because it can tell us about character, place, and time. It’s important because it informs us and moves us in ways visuals can’t, and because certain combinations of sound and visuals can evoke what neither can do alone. It’s also potentially important because it can help to determine what we see.” Sounds of nature are particularly interesting when their scale is amplified, or the sounds are transformed in a way that we are not used to listening to it. Like that, we can perceive them differently, tending to fixate this experience in our head and helping us to build with a better accuracy a picture of places and events. This becomes a compelling feature for our installation in that the sound of salt crystallization was essential in forming a mental dialogue with the space. Crystallizing sound effects were edited through the online webpage www.sodaphonic.com, creating rigid, striking soundtracks. After this experience, our intent was that the visitors would be able to pay more attention to the sound properties of a determined material and that they would be able to relate more closely with sounds aligned with texture, color, and visual aspects in order to have a more holistic view of a specific material or architecture. Kant also wrote in Reflexionenzur Anthropologie, “all the senses have their own descriptive vocabularies, e.g. for sight, there is red, green, and yellow, and for there is sweet and sour, etc. But the can have no descriptive vocabulary of its own. Rather, we borrow our adjectives from the other senses, so that it smells sour, or has a smell like roses or cloves or musk. They are all, however, terms drawn from other senses.” 726 M. FARR AND A. MACRUZ

This philosophy illustrates why it was important that the installation experiment with virtual environments and real environments in a symbiotic way to help us to re-perceive materials through human sensorium. Smell, sound, and touch were used in this installation as a way to ground the visitors in the real environment, bringing the perception back to their body. At the “Scent Fountain: Fear and Volatile Marilyn” at the Senses exhibition at the Cooper Hewitt Smithsonian in NY, the visitors were instructed to touch the surface and smell the scent. A different scent accompanied each texture. One called Fear is electric, and cold while Volatile Marilyn is curvy and soft. It was a piece designed for visitors to experience the relationship between texture and smell proposed by Studio Joseph, Dmitry Rinberg, and Perfumer Christophe Laudamiel. These examples point us toward the importance of combining smells, sounds, and textures in a holistic environment. These kinds of inclusive environments offer a broader spectrum of elements to the sensorium that can expand architecture’s role in our lives. Therefore, in the installation the smell of salt was combined with sound, texture, white color, and AR resulting in a compelling connection and a very integrated sensory experience.

4. Augmentation According to Clark, as natural-born cyborgs, devices such as computers, smart phones, and other nonbiological constructs, props, and aids are becoming extensions of our minds that help us to solve problems related to survival. Many of these devices currently are portable, reliable, flexible, and personalized, increasing the extension of our minds. We therefore transform our reach, thought, and vision, upgrading the cognitive structure in our minds when we employ nonbiological devices. We create these supportive constructs, props, and aids, but they also re-create us. In the installation, new software and devices were used for design, construction and feedback which acted as an extended mind for us as to perceive differently. Also, its aim was to analyze how these devices help us to extend our perception, how they transform us, and what kind of environments are we going to create after this experience. This process was significant in that it allowed for the exploration of new ways of thinking, making, and presenting spatial potentials that can affect the senses. But, beyond designing for senses, this project looks at what it means to not only evaluate a sensory experience but to heighten it with digital technology. This paper began by asking the question, “what happens when advanced technologies, such as those offered by Microsoft, HoloLens, are used in conjunction with physical architecture to connect people with deeper sensory experiences relative to materiality and space?” This augmented potential allows us to create a more rooted exploration of space, merging the physical with the digital, the buildable with the not yet buildable, the fabricated with unfabricated. The project was interested in augmented experiences rather than ones that were purely virtual, thus distinguishing between a computer-generated simulation and one that offered an augmented overlay of experience. It was also meaningful to allow this augmented MULTI-SENSORY MATERIALITY 727 reality to be a part of how the project was measured through the UpMood wearable technology. While VR, virtual reality, offers a complete immersion, such as entry into a different environment, a historical place, or a game world, the augmented experience, AR, offers an overlay onto the current physical installation, including additions to the scale, light, and potential materialities, combining real with unreal. These digital overlays are projected or layered along with the real world via glasses or iPhone monitors using Fologram. This technology is being used in many different industries beyond design as a means for translating complex tasks, teaching, delivering information, and promoting an idea of efficiency through technological assistance. For this project, AR was used as a way to increase the perception of potential material enhancements. Using software such as Rhino and Houdini, an augmented version of the installation was created that expanded on the physical by growing, changing densities, colors, and increasing the size and scale of the patterns. As one interacts with the installation, they can also interact with the digital by using a Fologram linked cell phone device. It alters the perception of the original space and creates a stronger awareness of the potential of the original pattern sequences. Ultimately, this paper is interested in the overlaps of the architectural space, the sensory space, and the augmented space.

5. Evaluation Technology / Wearables In order to predict the potential overlap between the architectural, the sensorial, and the augmented, we worked with UpMood technologies to begin understanding how to measure and estimate how people might feel or respond in a given environment. To monitor and evaluate how people perceive certain criteria, users wore emotion-sensing bracelets when they visited the project. These collect biodata from the user and result in 11 different emotional states: calm, pleasant, unpleasant, happy, sad, excitement, anxious, confused, challenged, tense. This data was continuously fed into an App that revealed the different states back to the user. The evaluations and the use of these overlapping technologies acted as a way to gain insight into a more profound human experience. Through this process, the project addressed user insight relative to emotional patterns and management. In the paper, The Nature of Feelings: Evolutionary and Neurobiological Origins, the author Antonio Damasio writes, “Survival depends on a homeostatic range”, and “feelings and experiences facilitate the learning of the conditions for homeostatic imbalances plus the anticipation of conditions. Feelings are mental experiences that accompany a change in the body state.” He goes on to write that, “external changes - displayed in the exteroceptive maps of vision or (sensorium) are perceived, but largely not felt. It can trigger drives or emotions, causing a change in the body state, and subsequently felt.” Homeostasis, the tendency towards a relatively stable equilibrium between interdependent elements, especially as maintained by physiological processes, is a big part of human survival, and is relevant to the theories of Damasio. Human survival depends on homeostasis, or the regulation of the body’s self-repair and 728 M. FARR AND A. MACRUZ defense. The body can regulate itself without the person having a feeling, or “conscientious experience”. However, when the person does have a feeling, and therefore he/she is aware of it, it facilitates the learning of a change in body state for a better prediction of future situations and thus increases behavioral flexibility. With these concepts in mind, this installation tries to increase felt experiences, using the different stimuli to increase senses and also offer a recording of what a user felt to bring about potential self-awareness. Wearing UpMood bracelets allowed us to monitor different emotional states in real time as we interacted with the project. What we found is that the technology can be sometimes accurate and sometimes surprising indicating that our heart rates change in different ways and can be highly situational. It was an interesting part of our experiments with this project because it allowed us to interact with a user in a way not normally accessed in architectural projects. The conclusions for these findings via UpMood wearable technologies indicate that stable peaks, highs and lows, in experience are crucial to accuracy and variance from person to person can differ. There can also be discrepancies between what the users “thought” they were feeling, and what the technology actually indicated. Fluctuation in experience will also affect the results.

Figure 3. Augmented surfaces with salt crystallization.

6. Methodology There is one profession and only one, namely architecture, in which progress is not considered necessary, where laziness is enthroned, and in which the reference is always yesterday.” Le Corbusier, ”Towards a New Architecture”, 1923. As discussed in the abstract, the methodologies used in this project are interested in looking toward the future, and creating a conversation about the potential of architecture relative to technology and the sensorial experience of a given space. The project involved the design and construction of a wall surface, the discussion of its material and sensorial qualities, and its virtual augmentation as a catalyst for new experiences. It employed digital technologies to help build, perceive, and evaluate the overall experience. The methods used in the project work with the existing paradigms of architectural design, but also speculate on a new agenda for current processes due to the use of new technologies in the process. Because of this, the project suggests a “re-centering” or a new starting point for how we can incorporate technologies into our design process as well as in our experience of a design project. Moving further, rather than designing MULTI-SENSORY MATERIALITY 729 and then building and then experiencing, we believe that these activities have the potential to overlap more and inform each other in a real way throughout the course of a given design exercise in a way that moves beyond theory and offers both quantitative and qualitative results. The methodology does not aim to provide solutions from the outset, but rather looks at using the process of augmented design and technology to inform solutions along the way, and then attempts to evaluate the potential impact of a set of sensorial experiences. Its aim was to build upon its own theoretical underpinnings, but is also interested in informing best practices and how this process can contribute or be applied to future projects. As discussed in the initial abstract, the installation explored what it means to design spaces with high degrees of sensory interaction and feedback. The use of software assisted design also extended the experience and allowed us to evaluate how people were interacting with the project via technology. This offered insight into user attitudes and feedback. The wearable emotion sensors were used to collect biodata (interactive stress level, heart rate, vitality levels), and the associated algorithms turn the data from human pulse waves into analytics to generate a discussion of emotions. The work allows for the overlapping of augmented reality into the design space by pairing emotional sensing wearables with augmented technology as an extension to the traditional design/build workflow and an extension of the material and spatial experience. The use of these technologies offers an expanded mode of inquiry relative the design process and outcome. While there are many critiques and areas still yet to be developed, we discovered a rich dialogue and a deeper sense of concept development in using this process. One of the dialogue streams in this process that has yet to be expanded on is that it lends itself to teaching as a fresh pedagogical model whereby students and teachers can discuss project potential in real time and in context relative to a given sensory-material scenario. The benefits include real time facsimile of scale, color, form, light and shadow, and the experimentation of these as they relate to a host of different material situations.

7. Conclusion This paper documents and critiques the process of an installation that was designed to induce a sensory experience and ask questions about the capacity to design with technology and a wider taxonomy of choices that can impact our experience in architecture. It explores these concepts relative to digital design and augmented material interaction in the field of architecture and constructed environments. It started by employing typical parametric & computational software, thinking of the potentials between the digital and the real, and incorporated this potential by accepting the role of augmentation as a way to interact with architectural space. The purpose was to engage people with multi-sensory experiences that force humans to re-perceive our physical world. Adding to the experience are additional sensory stimulations of sound, touch, and smell that can be experienced in the augmented space. 730 M. FARR AND A. MACRUZ

The findings of this process come in the form of data collected by the wearable technology from user feedback. In the end, the data collection devices, while not as sensitive as we would have liked, offered a unique perspective to the way that a particular user felt as they interacted with the project. The outcomes imply that through research and design, stronger sensorial experiences can be used to increase awareness, perceptibility, and create new design conversations. Using this project as a vehicle, the paper goes on to theorize an agenda that is interested in supplementing our design process by new tools and advanced technologies that allow us to experiment in extremely informed ways relative to the perception of physical context. This heightened mode of inquiry proposes a new way of looking at the process of designing full-scale projects by interacting with materials in a new sensorial way. Because of this, it proposes and questions the relevance of current material conversations and proposes new conversations involving student, teacher, , and client. “There is much more to the experience of life than successfully meeting basic needs. Humans are emotional beings. We embrace concepts like beauty, awe and wonder, which contribute to the connection we have with each other and our environment. As practitioners who have the perspective, tools and intention to design a new reality for humanity, we need to pay attention to the emotional context and intention of our .” - Senova, Melis. “This Human: How To Be The Person Designing For Other People”, 2017.

References Chang, A.: 2017, The Tao of Architecture, Princeton Architectural Press. Clark, A.: 2003, Natural Born Cyborgs: Minds, Technologies, and the Future of the Human Intelligence, Oxford University Press. Kant, I.: 1882, Reflexionen Kants zur kritischen Philosophie. 1, Forgotten Books. Lally, S.: 2017, The Air from Other Planets, A Brief History of Architecture to Come, Lars Müller Publishers. Leach, N.: 2006, Camouflage, MIT Press. Lupton, E. and Lipps, A.: 2018, The Senses: Design Beyond Vision, Princeton Architectural Press, New York. Pallasma, J.: 1996, The Eyes of the Skin, Wiley. Sevona, M.: 2017, This Human. How To Be The Person Designing For Other People, Bis Publishers, Amsterdam.