Art and Technology at Play

University of Art and Industrial Design Linz, Austria Department of Media Interface Cultures

ART AND TECHNOLOGY AT PLAY

A Practice-Based Exploration of Embodied Interaction

by Eng. Lic. Tiago Miguel Santos Martins

Advisors: Prof. Dr. Christa Sommerer1, Prof. Dr. Nuno M. Correia2

1 Interface Cultures - University of Art and Industrial Design Linz

2 Faculty of Sciences and Technology - New University of Lisbon

This dissertation is submitted for the degree of Doctor of Philosophy

Approved: ...... …

Linz, January 2017

For Mom, Pedro and Inci.

Acknowledgements

The author would like to thank all those who contributed to this dissertation and the experience, practice and works that supported it.

First and foremost, a big heartfelt thanks to supervisors Prof. Dr. Christa Sommerer and Prof Dr. Nuno Correia, for their role in starting it all, their patience, continued en- couragement and support.

Thanks also to the excellent, creative and hard-working people who collaborated on the works: Christina Heidecker, Thomas Wagner, Andreas Zingerle, Ricardo Nasci- mento, Maša Jazbec, Justyna Zubrycka, Vesela Mihaylova, Claudio Pina, Tom Bieling, Andrea Clemens, Inci-Ana Zohrap, Chiara Esposito and Fabian Werfel.

Thanks to the people of the Interface Cultures department of the University of Art and Industrial design in Linz, including: Prof Dr. Laurent Mignonneau, for the guidance with a sense of humour; Michaela Ortner and Gertrude Hörlesberger for their prompt support; all the good friends I have made there (so many of you!); and overall for an amazing environment to experiment, create, discover and grow.

Thanks to the Delavski Dom Trbovlje and the people from Trbovlje who put up an awe- some show every year, offer a venue for crazy ideas and always make me feel at home when I am there. You are my Slovenian family.

Thanks to the people at the Design Research Lab of the Berlin University of the Arts, for providing an engaging and pleasant environment to work; to Tom Bieling and Ges- che Joost for the opportunity to work on Speechless; to Corinna Schmidt for her promptness and optimism; to Katharina Bredies, Hannah Perner-Wilson and Jussi Mikkonen for the expert tips; to Pauline Vierne and Fabrizio Lamoncha for making everything even more fun. Thanks also to Elizabeth Scharler and John Von Bergen for their great help.

Thanks also to the people at the Empowerment Informatics group of the University of Tsukuba, for having me there and receiving me so kindly, especially Aki Yamada, Maša Jazbec and Monica Perusquía Hernández.

Finally, thanks to my friends and family for the continued support throughout the years; and especially "Grandpa" Vítor, who is my hero.

Abstract

This thesis explores the use of Embodied Interaction and playfulness as effective creative strategies in the design of human-machine interactions, employed towards aspects of usability and expressiveness of participatory experiences and interactive artworks. It focusses on a body of original works which include multimodal interfaces, interactive narratives, videogames, art installations, and critical design concepts.

As a whole, the works constitute a critical reflection upon the normative design of information and communication technologies, as they address phenomenological limitations, highlight ideologies and provide expressive alternatives – in a playful and embodied way.

A practice-based approach to inquiry was chosen, supporting the situated exploration of opportunities and potentials under the benefit of a holistic and critical perspective. This involved stance allowed the creative development of original and informative experiences of human-computer interaction, as opposed to designs which iterate on established product genres and preconceptions.

After introducing the topics of technology, Embodied Interaction and playfulness, with emphasis on aspects that have served to contextualize and/or inform the practitioner’s approach, an account of each work’s ideation, development and outcomes is offered. Each account highlights specific ways in which embodiment and play contributed towards the work’s successful aspects, while charting the researcher’s progression through different environments of creative practice, refinement of techniques and the development of a holistic and critical approach.

vii

Table of Contents

List of Figures...... xii 1.INTRODUCTION...... 1 1.1 ·Objective...... 2 1.2 ·Methodology...... 4 1.2.1 ·Creative Practice as Research...... 5 1.2.2 ·Paradigm...... 9 1.2.3 ·Critical Design...... 11 1.2.4 ·Art For The Masses...... 13 1.3 ·Structure...... 15 2.TECHNOLOGY...... 17 2.1 ·Evolutionary Synergies...... 18 2.2 ·Culture and Mass Production...... 21 2.2.1 ·Black Boxes...... 23 2.2.2 ·Domestication and Appropriation...... 24 2.3 ·New Media...... 26 2.3.1 ·What is New...... 26 2.3.2 ·Participation and Creation...... 28 2.4 ·Augmentation...... 29 2.4.1 ·Tools as Extensions...... 30 2.4.2 ·Cyborgs...... 31 2.5 ·Magic and Enchantment...... 32 2.5.1 ·Technological Super-powers...... 32 2.5.2 ·Magic, Mysticism and Misdirection...... 34 2.5.3 ·The Technology of Enchantment...... 35 2.6 ·Overloaded...... 36 2.6.1 ·Beyond Evolution...... 36 2.6.2 ·Cognitive Load...... 39 2.6.3 ·Mobile Interaction...... 40 2.6.4 ·Reaching a Limit...... 42 3.EMBODIED INTERACTION...... 45 3.1 ·Tangible Interfaces...... 48 3.2 ·Wearable Interfaces...... 53 3.3 ·Objects as Media...... 58 3.3.1 ·Meaning...... 58 3.3.2 ·Cognition...... 59 3.3.3 ·Materiality and Agency...... 61 3.4 ·Strategic Values...... 64 4.PLAYFULNESS...... 67 4.1 ·Psychology and the Magic Circle...... 67 4.2 ·Creativity...... 69 4.2.1 ·Videogames and Hacking...... 70 4.3 ·Structured Play...... 71 4.4 ·Evolutionary Fitness...... 73

ix 4.5 ·Play and Art...... 75 4.5.1 ·Evolution and Cognitive Play...... 76 4.6 ·Interactive Art...... 79 4.6.1 ·Participation and Interactivity...... 81 4.6.2 ·Procedural Rhetoric...... 83 4.7 ·Strategic Values...... 85 5.WORKS...... 87 5.1 ·The Gauntlet...... 89 5.1.1 ·Background and Concept...... 89 5.1.2 ·Design and Development...... 92 5.1.3 ·Outcomes and Interpretation...... 99 5.2 ·Noon – A Secret Told by Objects...... 102 5.2.1 ·Background and Concept...... 102 5.2.2 ·Design and Development...... 103 5.2.3 ·Outcomes and Interpretation...... 110 5.3 ·Headbang Hero...... 112 5.3.1 ·Background and Concept...... 112 5.3.2 ·Design and Development...... 114 5.3.3 ·Outcomes and Interpretation...... 123 5.4 ·Rambler...... 127 5.4.1 ·Background and Concept...... 127 5.4.2 ·Design and Development...... 128 5.4.3 ·Outcomes and Interpretation...... 135 5.5 ·Weltschmerz...... 138 5.5.1 ·Background and Concept...... 138 5.5.2 ·Design and Development...... 140 5.5.3 ·Outcomes and Discussion...... 143 5.6 ·Wcielenie...... 146 5.6.1 ·Background and Concept...... 146 5.6.2 ·Design and Development...... 147 5.6.3 ·Outcomes and Discussion...... 152 5.7 ·Tripo...... 154 5.7.1 ·Background and Concept...... 154 5.7.2 ·Design and Development...... 157 5.7.3 ·Outcomes and Discussion...... 159 5.8 ·Lorm Hand...... 161 5.8.1 ·Background and Concept...... 161 5.8.2 ·Design and Development...... 162 5.8.3 ·Outcomes and Interpretation...... 171 6.CONCLUSION...... 175 6.1 ·Summary of Strategies...... 175 6.1.1 ·Embodied Interaction...... 177 6.1.2 ·Playfulness...... 177 6.1.3 ·Summary of Works...... 178 6.2 ·Closing Comments...... 180 REFERENCES...... 183 APPENDIX A · Exhibitions and Distinctions...... 197

x APPENDIX B · Relevant Publications...... 201 APPENDIX C · Press Clippings...... 203

xi List of Figures

Figure 1: First version of the Gauntlet...... 96 Figure 2: First (top) and second (bottom) versions of the Gauntlet...... 96 Figure 3: Design sketch for the Gauntlet...... 95 Figure 4: Component placement sketch for the Gauntlet...... 95 Figure 5: Gauntlet hardware prototype...... 95 Figure 6: Gauntlet hardware placement...... 95 Figure 7: Finished prototype for the Gauntlet...... 95 Figure 11: PDA screens for Noon...... 107 Figure 12: First version of Noon, with observers trying to view the PDA's screen...... 107 Figure 13: PDA encased in a book, used in the first version...... 107 Figure 14: Participant touches one of the objects...... 107 Figure 8: Noon installation (second version)...... 108 Figure 9: The participant reads a memory from the snowglobe...... 108 Figure 10: The participant changes the time-period of memories...... 108 Figure 15: Hardware prototypes for Headbang Hero...... 119 Figure 16: Headbang Hero starting screen...... 119 Figure 17: Headbang Hero disclaimer...... 119 Figure 18: Headbang Hero gameplay GUI...... 119 Figure 19: Headbang Hero difficulty selection...... 119 Figure 20: Headbang Hero (second version) at the Festival Aucard de Tours...... 120 Figure 21: Headbang Hero personal Damage Report...... 120 Figure 22: Player-created score board...... 120 Figure 23: Children playing Headbang Hero at the Campus Party Mexico...... 120 Figure 24: Heabang Hero wig, with on/off snap button...... 120 Figure 25: Rambler technical overview...... 133 Figure 26: Rambler hardware schematic, left and right sneakers...... 133 Figure 27: Rambler sneaker showing on/off snap button...... 133 Figure 28: Rambler app. Left: first version. Right: Second version (emulator screenshot)...... 133 Figure 29: One of the Rambler sneakers, with bird logo and text with outline...... 134 Figure 30: Rambler Twitter page (@ramblershoes). Captured 11.03.2010...... 134 Figure 34: Weltschmerz installation with projection at Speculum Artium 2011...... 141 Figure 35: Meat-like aggregation and revolver...... 141 Figure 31: Weltschmerz installation with TV screen at Ars Electronica 2011...... 142 Figure 32: Weltschmerz participant and observer...... 142 Figure 33: Participant performing Russian roulette...... 142 Figure 39: Mnemoscope hardware prototype...... 149 Figure 40: Mnemoscope hardware...... 149 Figure 41: Mnemoscope hardware placement...... 149

xii Figure 42: Wcielenie map, Kazimierz neighbourhood...... 149 Figure 43: Mnemoscope and Wcielenie map, with RFID tags and indications for placement...... 149 Figure 36: Wcielenie participant outside the bookstore...... 150 Figure 37: Wcielenie participant at the playground...... 150 Figure 38: Wcielenie narrative map (cheatsheet)...... 150 Figure 44: The Tripo device...... 155 Figure 45: Inside of the "reader" can...... 155 Figure 46: Tripo poster with instructions...... 155 Figure 47: Tripo participant using the device...... 156 Figure 48: Tripo participant exploring a secluded location...... 156 Figure 49: Tripo hotspot (sticker over RFID tag)...... 156 Figure 50: Rotating the device to "tune in" to different stories...... 156 Figure 55: The Lorm Hand, first version...... 165 Figure 56: Lorm Hand at the protest march...... 165 Figure 57: In-house testing with a deaf-blind participant...... 165 Figure 58: Lorm Hand, first version (on black base instead of pedestal)...... 165 Figure 59: The Lorm alphabet...... 165 Figure 51: Lorm Hand, second version...... 166 Figure 52: Lorm Hand, third version...... 166 Figure 53: Detail of the flexible circuit (second version)...... 166 Figure 54: Cabling study (third version)...... 166

xiii

1. INTRODUCTION

Art, technology and play have been key aspects of communal life and development since the evolutionary dawn of humankind. In academic discourse they may frequently be dealt with as separate fields, yet in practice they are mutually influential and even inseparable to a degree. Multidisciplinary efforts are increasingly applied in the research of new technologies, designs and applications; as well as investigating the sociocultural impact of technologies and their adoption within existing and emerging social settings. Research teams gather specialists from different backgrounds; artists engage with technology, new media and scientific research; and enthusiasts and tinkerers adapt or subvert consumer technologies to new purposes, forming online communities to openly share their findings.

This dissertation, consisting of a body of artworks and the thesis text, is the result of a practice-based approach to the exploration of Embodied Interaction, under a constant overtone of playfulness. Coming from a background in Computer Science and Engin- eering, the author has engaged in artistic practice to research and produce original work – spanning multimodal interfaces, interactive installations and artworks – towards the expression of new ideas and possibilities beyond a techno-centric stance.

Employing an approach based on Embodied Interaction and playfulness, the works critically reflect upon the status quo in the design of information and communication technologies, addressing phenomenological limitations, highlighting ideologies and providing expressive alternatives – in a playful and embodied way.

Embodied Interaction is a foundational approach to the design of human-machine interactions (HCI) laid by researcher and scholar Paul Dourish, focussing on “the creation, manipulation, and sharing of meaning through engaged interaction with artifacts” (Dourish 2004:126). Proposing a more human-centric approach to the design of technical artefacts and computer-supported collaborative work, embodied interaction draws from phenomenological perspectives, including the works of Heidegger and Merleau-Ponty, to reject the cartesian dualism (conceptual separation of mind and body) which permeates the way digital technologies are designed.

Building upon Dourish’s work, this dissertation additionally draws on anthropology,

INTRODUCTION | 1 cognitive science and evolutionary psychology in order to effectively exploit the ways humans understand physical spaces, artefacts and each other. The resulting works illustrate and promote the design of technical artefacts and human-machine interactions addressing our biological and cognitive being-in-the-word in an empowering and expressive way – as opposed to acquiescing to a way of being (perceiving, reasoning, acting) which is constrained by the pervasive cartesian dualism and inherent logic of computational technologies.

Together with Embodied Interaction, a measure of playfulness is employed in the author’s practice-based approach, both as a lens for observing aspects of technology in social use, and as a kaleidoscope to generate new ideas and possibilities. A majority of the works also employ playful tactics facing the audience – including game-like structures, irony and humour, fantasy and exploration – to stimulate the engagement of participants, reward participation and strengthen expressive qualities.

By employing Embodied Interaction and playfulness as strategies in creative practice and interaction design, the resulting body of work provides a contrast to consumer technologies as we know them, shedding light on the gulf between market-driven design and the limited expectations it creates; and the synergistic potential of technology, art and play harnessed towards expressive, affective and empowering interactions.

1.1 · Objective The objective of this dissertation, the practice and works which form its core focus, has been the creation of original works illustrating approaches to the design of human-computer interaction (HCI) based on Embodied Interaction and playfulness.

As a whole, the body of works is driven by the exploratory design of tangible and wearable interfaces within specific contexts, with the outlook of enabling intuitive and expressive interactions contributing towards immersive and engaging participatory experiences; and constituting dynamic forms of critique upon the normative status quo of the interaction design of commodities and services for the consumer electronics market.

New media and digital technologies have made possible new forms of information ac-

2 | ART AND TECHNOLOGY AT PLAY cess, communication and augmentation of human capabilities. As a result, socio-economic aspects of contemporary life are ever more tightly dependent on the capabilities that technological means provide. This of course affects modes of expression and overall social processes; not only in the perception of what is possible but also of what is expected and how it should be achieved. While computational systems, social networks, mobile devices and even wearable interfaces play a perceivably indispensable role in our social life, they still largely do so in their own terms – ignoring the interde- pendency of cognitive and motor skills of humans, as well as resulting predispositions and unique capabilities, refined throughout millions of years of evolution as embodied beings.

Embodied Interaction and playfulness have been applied as strategical approaches Through practice-based inquiry, developed throughout the practitioner’s continued engagement. As strategies, they were employed in the specific context of each work to to develop core functional aspects (e.g. usability and interactivity) but equally to support the work’s expressive qualities (e.g. aesthetics and rhetoric).

Presented chronologically, the works chart the author’s progression through different environments of creative practice involving participatory experiences with digital technologies. Collaboration and interdisciplinary perspectives contributed not only to the exploration and refinement of techniques and strategies, but also to the development of holistic and critical perspectives.

Besides illustrating approaches and testing strategies which are refined through practice, the works also foreground existential situations, expose cultural assumptions or highlight perspectives amidst the plurality of human experiences. Some works are intended as critical design pieces and may thus be perceived as speculative product concepts, welcoming occasional ambiguity. Others were developed under more design-oriented contexts, while remaining creative and thought-provoking; and may serve to illustrate the application in “real-world” (non-speculative) scenarios of the strategies explored in creative practice.

The resulting collection is heterogeneous to some degree. Ultimately, each work has its own contextual background, themes, rhetoric and finality. As a collection, however, they share strategic approaches and a motivation towards providing critical (and potentially disruptive) alternatives to the disembodied, cartesian-dualist mindset pervad-

INTRODUCTION | 3 ing consumer electronics in the digital age.

1.2 · Methodology In order to gain distance from designs which iterate on established product genres and instead follow unusual directions, the methodological choices support exploratory work through participation in and dialogue with the context and process of creation, developing a holistic and critical perspective. The bulk of the inquiry has been conducted through artistic practice, supported by a multidisciplinary theoretical base and having a body of works as foremost result. As such, the present dissertation is methodologically framed as practice-based research.

Practice-based research is perhaps more common in art and design contexts. While in design the usual focus is on the nature of practice and how to improve it, in the visual arts a bigger emphasis is placed on the creative process and the works themselves, which play a vital part in the generation of new knowledge (Candy 2006).

Given a long tradition of scientific research under positivist paradigms, a thesis or dissertation is traditionally conceptualized as a written text describing in detail a hypothesis, experiments and quantitative results thereof, which prove and validate (or occasionally deny) said hypothesis. Research into humanities and social sciences has greatly contributed to the eventual acceptance of qualitative methods and results, as the multi-dimensionality of lived human experience can seldom be quantified, and is better analysed and interpreted with regards to process and context. Even in such cases, however, the thesis itself is commonly formalized as a bulk of text (or collection of related texts) and reference material.

Where practice in the domain of visual arts is taken as research method, however, the resulting artworks can hardly be ignored. They constitute a part of the thesis, even if in varying degrees. Depending on the nature of the inquiry, a series of works may be accepted as exclusively constituting the thesis, however accompanied by adequate documentation and a summary, for future reference (Jones 2002).

It may be contentious that an artwork can, in itself, constitute “new knowledge” in the sense required as a valid result of research. But whereas an artwork “by itself” can refer to (and is eventually experienced in) a context, it can also be contextualized as

4 | ART AND TECHNOLOGY AT PLAY the embodiment of new knowledge; appreciated not alone but within a body of works forming a more complex and yet thematically cohesive discourse; and can certainly be a highly effective medium for the transmission of ideas, perspectives and intellectual or phenomenological experiences which would otherwise be cumbersome (or altogether impossible) to convey.

Therefore the present text should not exclusively be considered the thesis, but rather a component of it. The core of the thesis is the collection of works produced. While the present text may not have the same emphasis or objective detail as other recent practice-based dissertations in interactive art (Stenslie 2010, Seevinck 2011, Mendes 2012), it is also not a mere documentation and summary. It serves to contextualize and critically address the elements of practice involved in each work and its outcomes: the motivations, theories and works which inform and support it; aesthetic, rhetoric and technical strategies and how these reinforce each other; the results and observations at each stage, and how they inform further developments (Hannula 2009).

1.2.1 · Creative Practice as Research As the research goal implies steering away from the status quo, the choice falls quite naturally on creative practice as a means of inquiry; which supports exploratory approaches and enables critical perspectives to emerge, through contextualized, tacit knowledge.

In art and it’s traditions, several critical thinkers argue, one may find “humankind’s most advanced and enlightening forms of thought” (Bardzell & Bardzell 2013:3302), not least in:

“[...] the aptness of its allusions; by the way it works in grasping, exploring, and informing the world; by how it analyzes, sorts, orders, and organizes; by how it participates in the making, manipulation, retention, and transformation of knowledge” (Goodman 1987).

Artistic practice may be used to enquire into “a creative opportunity to be explored or exploited; or an issue to be examined” (Jones 2002:3) and as such it is deemed ideal for the present purpose.

As inquiry, it places emphasis on the creative and critical process of practice – the practitioner’s dialogue with a situation and the “mental buzz” which supports creative

INTRODUCTION | 5 activity – and the works that are generated (Schön 1983, Sullivan 2006, Candy 2009). Instead of the more traditional concept of original investigation, it can perhaps be better addressed as original creation (Scrivener 2002).

When engaged in creative practice, the practitioner continuously develops a dialogue with the situation (Schön 1983) and from this process “the programmatic generation of new knowledge in a defined field” is achieved (Jones 2002:2).

Creative practice is highly empirical, concerned with providing ways of understanding (Scrivener 2002); and embracing the continuous evolution of knowledge and understanding itself as “an adaptive process of human thinking and acting that is informed by our experiences and encounters” (Sullivan 2006:96). For this, it seeks out ambiguities and marginalities, embracing plurality in approaches where openness, complexity and uncertainty are not risks but rather necessities, when dealing with the realm of human experience (Hannula 2009). Levi-Strauss’s analogy of the artist as bricoleur (Strauss 1966) is especially fitting when characterizing a researcher involved in creative practice.

Reflective Practice Donald Schön offers an informative and in-depth view of the importance of tacit skills continuously developed and refined during practice, where practitioners engage in contextualized problem-solving by resorting to unformalized capabilities and strategies found lacking in the predominating technical-rational model of knowledge (Schön 1983). He dubs this approach reflection-in-action. According to him, the model of technical rationality places emphasis on problem-solving while ignoring emerging aspects of the context or setting which may be “puzzling, troubling, and uncertain” (Schön 1983:40). In real-world practice however, the practitioner must also be able to frame (and at each step re-frame) the problem: “define the decision to be made, the ends to be achieved, the means which may be chosen” (ibid.).

Acknowledging that “practical activity is itself intrinsically intelligent”, Schön’s view un- burdens itself from traditional dichotomies which separate theory and practice, and seem to “imply that you switch your brain off in order to make art or design [...] and then switch it on again in order to reflect on what you have made” (Jones 2002:1). As we will see, theoretical dualisms between the experience of mind and the experience

6 | ART AND TECHNOLOGY AT PLAY of body have greatly contributed to limiting the way in which our digital devices are designed (Dourish 2004).

Reflective practice accounts for knowledge surfaced during hands-on experience, as the practitioner makes new sense of situations of uncertainty, puzzlement or uniqueness. A tacit repertoire of previous experiences is drawn upon providing references, precedents or metaphors, allowing the practitioner to construct a grounded theory for the unique case he/she is facing, dynamically building new understandings which will in turn inform subsequent approaches (Schön 1983). The practitioner “reflects in and on his practice”, on aspects which are “as varied as the kinds of phenomena before him and the systems of knowing-in-practice which he brings to them”; and which may include the tacit knowledge and assumptions underlying a judgement or the framing of a problem, implicit strategies and theories which support practice, or even the practitioner’s role within larger context (62). The practitioner’s tacit understanding is surfaced critically, articulated and tested, and may form the basis for a new theory of the phenomenon which was the initial cause of surprise or puzzlement (63).

As a method of research, reflective practice facilitates situated (and embodied) inquiry as it “involves looking to our experiences, connecting with our feelings, and attending to our theories in use” (Smith 2001); and a dialogical process where “one questions one’s practices in the light of critical theorizing” and in turn “interrogate one’s theoretical studies in light of practical experiences” (Stenslie 2010).

Artworks and New Apprehensions It can be contentious whether artworks may, of themselves, represent new knowledge. Stephen Scrivener argues that we “should not attempt to justify the art object as a form of knowledge and should instead focus on defining the goals and norms of the activity that we choose to call arts research” (Scrivener 2002).

Within the more traditional definition of research – i.e., “an original investigation undertaken in order to gain knowledge and understanding” – it can be hard to validate an artwork as constituting new knowledge, yet not if we consider inquiry which seeks to “generate [culturally] novel apprehensions [...] by undertaking original creation” (ibid).

The art-making process “brings things into existence” as an individual draws on “nat-

INTRODUCTION | 7 ural and artificial worlds and imagination” to generate “objects that must be grasped by the senses and the intellect” and provide “ways of understanding what is” (ibid).

Art can be more than an instrument used to illustrate scientific research or impart new knowledge in a traditional positivist sense. It can also be a medium of “emotional experimentation” providing insight into lived human experience (Sommerer 2015). Art- works and aesthetic experiences can raise “awareness about societal topics”, appeal to our emotional investment and lead people to “intuitively understand the messages” conveyed through artistic expression (ibid).

In the context of new media and interactive art, one may find the production of interactive artworks (and analysis of process and/or outcome) as part of the methodology for doctoral research (Stenslie 2010, Seevinck 2011, Mendes 2012). In such a context at least, artworks can serve different purposes for creation and dissemination of new knowledge: they may illustrate perspectives, topics and findings related to the research; support new understandings about the experience of participants; and surface new knowledge through reflective practice (Seevinck 2011:66). Seevinck additionally points out that artworks “have their own authority as aesthetic objects and can exist independently of the thesis” and are consequently able to “increase the applicabil- ity and significance of the thesis to the creative arts and public domain” (57).

Anthony Dunne has shown how a practitioner may incur in interpretation “of design processes and their relationship to culture and society”, in his particular case by developing speculative alternatives using “estrangement, subversion, or humour [...] with the goal of allowing refusal, curiosity and innovation” (Seago & Dunne 1999:15). The same applies to several of the works which constitute this dissertation, regardless of whether they are more oriented towards art or design agendas. Technology – and our mediated experiences – can be approached from an artistic standpoint to explore experiential, emotional and aesthetic values, “turning attention away from ‘optimum performance’ and towards more fundamental philosophical issues” (14).

Exploratory Work The context of the creative arts also effectively lends itself to exploratory work. Explor- ation is immanent within the reflective practitioner’s activities, as he is faced with unexpected situations which he must analyse and solve with resort to his repertoire of

8 | ART AND TECHNOLOGY AT PLAY tacit skills (Schön 1983). To be fair, our daily behaviour is to a great degree exploratory and improvisational, rather than meticulously planned – something which often es- capes the formulation of technical systems and human-machine interactions (Such- man 2007).

Exploratory methods are especially adequate in uncovering opportunities, blind spots and limitations which can only be found off the beaten path (Valentine 2013:10). As mentioned before, a research question may address a creative opportunity to be explored, whether it be “technical, procedural, philosophical, theoretical, or historical” (Jones 2002:3). Limiting the formulation of research questions to a style matching that of the humanities and social sciences tends to exclude open-ended curiosity (2).

The original research topic – multimodal interfaces for ubiquitous games – was somewhat more directed and pragmatic. It did, however, share similar concerns and out- looks: an improvement in the design of playful interactions in the age of ubiquitous computation driven by a disillusionment with screen-based mobile media (see 5.1.1). Furthermore, it set a starting point for what would become a more diverse, critical and even emotional body of works.

While overall, the body of works and originating practice do chart a progression which can be seen as a developing narrative, the research goal itself has certainly mutated naturally through – paraphrasing Stenslie – the intertwined development of evolving interests and tacit, unspoken knowledge, in academic, professional and even personal contexts (Stenslie 2010:46).

1.2.2 · Paradigm Research in the arts is typically flexible in concept and method. Artists “flock to the ambiguities and marginalities that cause others to flee” (Becker 2002:5) and “reconcile roles that many have learned to view as discreet and distantly related” (Thompson 2006:2).

Relying on practice-based exploration of alternatives with a critical component, the present research “focuses on real, located practice, and it is based on an interactive research process” uncovering “cause-effect processes in a local, contextualized” way (Gi- aldino 2009). The main interest lies in how the world is experienced through everyday interactions with and through digital technologies; and how mass-produced digital

INTRODUCTION | 9 goods assist, disrupt or originate social processes (ibid.).

Qualitative in nature, it acknowledges the necessity for plurality, openness and uncertainty (Gialdino 2009, Hannula 2009) methodologically eschewing the “technical rationality” model (Schön 1983). As it deals with art, speculation and fiction and, it wishes to provide “imaginative insight” – which remains somewhat elusive to traditional validation as “new knowledge and understanding that is individually and culturally transformative” since it is occasionally less concerned with plausibility and prob- ability, and more with possibility (Sullivan 2006:96).

Claude Lévi-Strauss (1966) compared the artist to a bricoleur, an especially apt analogy in the present context. As research in the field of the arts constitutes “interpretive practice revolving around interactive situations coloured by culture and zeitgeist” (Stenslie 2010), a degree of “eclecticism” is beneficial in grasping and making sense of otherwise theoretically disparate phenomena (Sullivan 2006, Olsen 2010). The artist as researcher may attempt to “dislodge discipline boundaries, media conventions, and political interests” not necessarily for disruption and agitation, but perhaps more importantly to look beyond boundaries and barriers towards “potential pathways that can link ideas and actions in new braided ways [...] within a realm of aesthetic experience, cultural commentary, and educational relevance” (Sullivan 2006:150-151).

Therefore, a mix of paradigmatic approaches have been used, concurrently or in turn, throughout practice: inductive, constructivist/interpretive and critical. These were not necessarily set out from the start, but rather emerged as a result of the overall pragmatic and dynamic nature of reflective practice (Gialdino 2009).

Inductive processes inevitably permeate all exploratory work, distilling knowledge or emerging theory from observations made without a strict theoretical point of depar- ture (Mackenzie & Knipe 2006). In the present case, the practitioner’s strategic choices, as well as the significance and outcomes thereof are contextualized in practice by resorting to a repertoire and existing theory.

Interpretivist and constructivist approaches are necessary to make sense of the practitioner’s own tacit, unspoken knowledge, and how said knowledge is continuously built upon. Interpretation also plays a role in making sense of the experience of participants and users, obtained through observation and informal interviews. At the very

10 | ART AND TECHNOLOGY AT PLAY least, it allows contrasting the participant’s perceptions with the author intentions and expectations. It may occasionally be necessary to construct a “theory of the unique case” (Schön 1983:68) either during actual practice, or to address – and possibly take advantage of – observed phenomena in the reception of any of the works.

Several of the works are also conceived and/or framed within a critical design perspective: they facilitate a dialectical contrast between present reality and speculative scenarios as a means of exposing the dominant hegemony of the consumer electronics industry (Seago & Dunne 1999). By eliciting “sensitive and insightful rather than crude aesthetic reactions” (Bardzell & Bardzell 2013:3302), they contribute towards cultivating – in a wider audience – an aesthetic sensibility of the everyday social usage of technology (ibid.).

1.2.3 · Critical Design The term “critical design” was coined by Anthony Dunne in his 1999 thesis Hertzian Tales, and further developed in joint practice with Fiona Raby (Dunne & Raby 2005, 2014).

The designer duo does not consider critical design as a newly found methodology, but rather as a term useful in grasping the attitude behind both their own work and that of other practitioners (Dunne & Raby 2005). The term characterizes design practice making use of “speculative design proposals to challenge narrow assumptions, preconceptions and givens about the role products play in everyday life”, often as the means “to question the limited range of emotional and psychological experiences offered through designed products” (ibid.).

With roots in critical traditions, it is “suspicious of the potential for hidden ideologies that can harm the public”, intentionally using speculative design to provoke and “cul- tivate critical awareness in designers and consumers” (Bardzell & Bardzell 2013:3300).

Besides not characterizing it as an actual methodology, the designers also imply a pur- poseful separation of critical design from art contexts which, according to the duo, limits the audiences addressed by the works and the way these are received and subsequently interpreted (Dunne & Raby 2005).

Jeffrey and Shaowen Bardzell have attempted to address these apparent limitations, while also bringing some clarity to Dunne and Raby’s original definitions. Their inter-

INTRODUCTION | 11 pret the concept as follows:

“Critical design is a form of research aimed at leveraging designs to make consumers more critical about their everyday lives, and in particular how their lives are mediated by assumptions, values, ideologies, and behavioral norms inscribed in designs.” (Bardzell & Bardzell 2013:3297, original emphasis)

As a form of research then, critical design is seen as potentially valuable in the field of human-computer interaction (HCI) research, drawing focus away from typical concerns about usability and professional tools, and towards aspects of user experience related to social justice, activism and values-oriented design (ibid.).

This leads the authors to expand on the formulation, encouraging others to participate "not by decoding whatever Dunne and Raby might have meant, but by actively and creatively developing critical design in ways that we as a community want to see it used" (3297). It is certainly intended that the present work may contribute towards this end.

Critical design is a means of breaking out of the limited experience currently prolifer- ated in product genre through underlying values of immediate pleasure, conformism, passivity and the illusion of choice (Dunne & Raby 2001). This is achieved by applying critical sensibilities in a unique aesthetic medium that engages the users imagination – the speculative designed object (ibid.). Designed objects inevitably engage our cognitive faculties in a way that leads us to mentally consider scenarios and experiences beyond the object itself (Olsen 2010:328, Pinker 1999:328); and so speculative objects have the power to engage the audience in "a sort of speculative material culture, fictional archeology, or imaginary anthropology" (Dunne & Raby 2013:140).

The reasoning behind the separation from art contexts – that art is expected “to be shocking and extreme” where critical design “needs to be closer to the everyday” – while understandable, seems to be based on a rather narrow vision of what art is and the range of experiences it can offer (Bardzell & Bardzell 2013, Dunne & Raby 2014). On the other hand, Dunne & Raby also claim that if an object or experience “is regarded as art it is easier to deal with, but if it remains as design it is more disturbing, it suggests that the everyday as we know it could be different” (ibid.). Although an

12 | ART AND TECHNOLOGY AT PLAY equally understandable reasoning, it can also be seen to contradict the previous one.

Critical design does admittedly “borrow heavily from art in terms of methods and approaches” (ibid.) and the authors frequently employ “a conceptual vocabulary strongly associated with art” (Bardzell & Bardzell 2013:3299). The resulting artefacts, as well as the fictions they suggest, promote criticality and operate outside of global capitalism, foregrounding “provocation and transgression, existential situations [and] exposition of cultural assumptions" (ibid.). Critical design is itself based on traditions which see "the arts as most advanced and enlightening forms of thought” (3302).

While it may be retained as tactical orientation that critical design should not let itself “be absorbed into the social practices of the artworld” and “institutional structures of exhibitions, museums, and funding”, it is also reasonable (and less short-sighted) to consider that “critical design and art may or may not overlap” (3304). It is the author’s hope that the present body of works may, among other things, serve to support this position.

1.2.4 · Art For The Masses The avoidance of the traditional gallery setting expressed by Dunne and Raby is nevertheless understandable. Austerity, passivity and a respectful distancing can constitute limited or vicarious displays of reverence inherited from the cloistered tradition of western art display and appreciation (Kaprow 2003:56). Such an attitude is especially inadequate in the reception of participatory artworks, as it constrains (or outright pre- vents) meaningful dialogical experiences.

The author believes that the aesthetic experiences and insights that art can offer must extend as directly as possible to a wide audience. While tools, forms and processes abound, so much of contemporary art is still developed in a way that excludes the “un- educated” masses while it caters to an “educated” minority – artists, curators, critics and scholars fluent in the subject matter (Kaprow 2003). Art becomes idealized and restricted to “certain classes of culturally sanctioned objects and events”; and “the capacity to have an esthetic experience [is itself] estheticized, becoming the purview of ex- perts” (Kelley in Kaprow 2003:xii).

Western culture seems to split spiritual and practical matters (ibid.) and persist on notions of “fine art” or “high art”, whereas in eastern cultures a myriad of artefacts, prac-

INTRODUCTION | 13 tices and experiences are appreciated as art forms (Kusahara 2006:3). For instance, famous Japanese artists produced woodblock prints to be sold to the public (ibid.). Rather than being confined to the austerity of the museum, their works became “a part of people's lives” (5). The western view of what constitutes an artwork seems to imply that multiplication and distribution are ultimately destructive of its symbolical function, as if the expressive capabilities of an artwork are inseparable from exclusiv- ity and existential uniqueness:

“what happens when works are multiplied as serial objects is that they indeed become commensurate with ‘a pair of stockings or garden chair’, and assume meaning in relation to them. They no longer exist as works of art, as materials with meaning, and as open significations in opposition to all other finished objects, but have become finished objects themselves” (Baudrillard 1999:71)

Such limiting notions have been critically tested and disproved by modern and contemporary artists, including Marcel Duchamp, László Moholy-Nagy and Sol Lewitt, among others (Frieling 2008).

As part of its mission, the present body of works attempts to reach broader audiences by employing a language that is familiar to most. In their playful qualities and (perhaps more importantly) in the way they examine, appropriate and highlight pop-culture, social media and digital games, most works in the collection employ formats or rhetorics which facilitate reception by the “general public”.

The same approach (whether intentional or not) is well illustrated by Japanese device art, which “integrates art and technology as well as design, entertainment, and popular culture” and makes frequent use of “interaction, a positive attitude towards technology, and playfulness” as creative strategies (Kusahara 2006:1). According to researcher and scholar Machiko Kusahara:

“Japanese artists often choose to make their works entertaining rather than being negative toward technology, even if their works display criticism against media society or try to reveal certain natures of media.” (Kusahara 2010:4)

Several of these artists conduct their practice while involved “in fields such as design,

14 | ART AND TECHNOLOGY AT PLAY entertainment, and commercial production” (Kusahara 2006:1) and this often results in works which have perceived artistic value, in spite – or perhaps as a result – of being approachable and even entertaining.

As similar case can be made for videogames. Although some authors still resist equating videogames with art (Jones 2012) – perhaps because they don’t really understand one or the other – the author of the present text does believe in their expressive power as art forms. That is not to say that all videogames (or even a majority) are a noteworthy works of art. Nevertheless they appear to threaten conventional distinctions between high art and popular culture (Salen & Zimmerman 2004:518). Video- games borrow much of their aesthetic qualities from film, animation and television; and like these, are shaped as products catering to pop-culture sensibilities (Lister et al. 2008:298). But more than that, they are also a combination of artforms which may excel as meaningful dialogical experiences by incorporating concepts from different media (Croal 2016); and employing interactivity as a form of rhetoric, that is, a means of effective expression (Bogost 2010).

1.3 · Structure This dissertation is structured in five chapters. Chapter 1 begins by providing an introduction to the motivating context and objective of inquiry, followed by a characteriza- tion of the chosen methodological approaches. Prompted by the limiting aspects of interface design in consumer technologies, the researcher has engaged in creative practice to explore scenarios outside the established status quo, relying on Embodied In- teraction and playfulness as strategies.

Chapter 2 addresses technology and its inescapable presence in human life. It starts by highlighting its evolutionary origins and its ancient ties to the development of culture. Some of the possibilities and challenges associated with modern digital technologies are highlighted next, which are especially pertinent for the thesis. Possibilities include powerful forms of communication and expression, and the augmentation of human capabilities in perception and agency. Challenges include the opaqueness of digital commodities, and the disregard for inherent dispositions of human cognition in favour of established computational designs.

Chapter 3 introduces Embodied Interaction as a foundational approach to human-

INTRODUCTION | 15 computer interaction design which favours human cognitive predispositions of evolutionary origin. It addresses the inseparability of action and understanding as a virtue of our physical existence, and how these are inevitably tied to evolutionary and social processes. Tangible and wearable interfaces are then presented as concrete approaches to Embodied Interaction, in connection with themes discussed in Chapter 1. Finally, the use of Embodied Interaction as a strategy is introduced, with reference to the upcoming works.

Chapter 4 explores the concept of play, focussing on its benefits as form of evolutionary fitness, driving experimentation and discovery. A connection between play and art is then introduced and further highlighted in the context of participatory and interactive art, which is of particular importance to the present thesis. Similarly to the previous chapter, it ends with the introduction of playfulness as a strategy, with reference to the works described in the chapter that follows.

Chapter 5 presents and discusses the eight works which constitute the core of the dissertation. Each work is approached in turn, starting with its context and motivations. This includes research and art projects, previous works by the author and pop-cultural phenomena that set the stage for, inspired or informed the creative act. Following is a detailed narrative description of the process of ideation and conception of the work, including aims and expectations, creative decisions, tools and techniques, prototypes and different versions. Outcomes are then presented and interpreted in regards to the work’s intended effect, observed and manifest experience of participants, but also emergent or ancillary occurrences; punctuated by salient outcomes of the use of Em- bodied Interaction and playfulness.

Chapter 6 offers a summary of the strategic value of Embodied Interaction and playfulness supported by successful aspects present throughout the body of works. Em- bodied Interaction is seen as providing usability and reinforcing expression. Playful- ness was employed as a mindset towards innovation, but also as a means of engaging participants and an effective mode of discourse. The chapter finalizes with the author’s closing comments.

16 | ART AND TECHNOLOGY AT PLAY 2. TECHNOLOGY

While some primates, other mammals and even birds are able to employ sticks and stones as primitive tools (Pinker 1999:194), the ideation, fabrication and use of more complex artefacts is an exclusively human achievement (F. R. Wilson 1999:173).

When we think of technology, we may certainly picture computers, tablets and smartphones, power plants, vehicles, the atomic bomb or the Mars Rover, among a myriad of other artefacts and systems. While these are indeed materializations or applications of technology, they are quite heterogeneous and do not necessarily account for the epistemological aspects at their origin.

The etymology of the term technology in its Greek ancestry indicates the meaning “knowledge about skilful practices” (Lister et al. 2008:87). Technology extends beyond technical artefacts, to encompass technical/scientific knowledge and the application thereof. We can define the term ‘technology’ as denoting “artefacts and processes of economic or social usefulness”, i.e., the cumulative combination of knowledge, processes and tools which allows us to design, produce and improve complex systems not readily found in nature (yet exploiting properties of the natural world), as the means to an end regarding our subsistence and improvements to our quality of living (429).

Throughout this text, the term ‘technology’ will be used interchangeably to refer to:

· the general concept of techniques, tools, and their application as means to an end, as in “life as we know it would be impossible without technology”;

· specific instances or applications of technical means, as in “Bluetooth and Wi-Fi are both based on the technology of radio transmission.”

It is hoped that the context of usage is clear enough in regards to which meaning should be interpreted. Admittedly, it can be difficult to bound any one given technology, as techniques and process are often cumulative and/or interdependent. Yet this will seldom if at all be of impact in the present text.

Technology is a heterogeneous aggregation, “a rearrangement of existing parts, some from nature, some from culture, some from existing technologies” (Lister et al.

TECHNOLOGY | 17 2008:405). Indeed, technology and culture are tightly interconnected and mutually-influential in many ways, as to be inseparable (Williams & Edge 1996, R. Williams 1990). Taking the microscope – a technical artefact – as an example:

“The microscope exploits the properties of glass, brass, nerves, retinas and brains, the sciences of chemistry and biology, and the practises of lens- grinding and industrial manufacture. It is inseparable from the development of science, and from the institutions and practices it unites. The simple re- combination of all these elements is precisely what a technology effects.” (Lister et al. 2008:406-406)

This interconnectedness also shows that the development and use of technology in daily life has been with us since the dawn of humankind. We shall go into more detail in the following sections. But it should be readily understandable the impact that technology has on how we live: the way we gather and structure information about our environment; the way we harness and utilize its resources; and the way we communicate and transfer all kinds of information, including technical knowledge itself (Lister et al. 2008:405-411, Thackara 2005:30).

2.1 · Evolutionary Synergies Philosopher and cultural theorist Jean Baudrillard has observed that “humans and their techniques, needs and objects are structurally interlocked come what may” (Baudrillard 2005:134). This truth runs deep into our evolutionary history. Technical knowledge and the production of artefacts has accelerated since the more primitive Homo sapiens evolved into the anatomically modern human, Homo sapiens sapiens, between 200.000 and 100.000 years ago (Thackara 2005:30). “Artifacts” writes cognitive scientist Steven Pinker “come with being human. We make tools, and as we evolved our tools made us” (Pinker1999:327).

Tool-making and tool-use represent evolutionary fitness, increasing chances of survival and reproduction (Pinker1999:327, Pinker2003:236, Boyd 2010:381). Other species have been observed to employ basic tools, such as chimpanzees using sticks to “fish” for termites or otters using stones to break clam shells. Humans, however, are the only ones who devise solutions using the refinement and combination of different materials and shapes (F. R. Wilson 1999:173), as this requires “levels of mimetic and

18 | ART AND TECHNOLOGY AT PLAY conceptual ability that were (and are) unique to hominids and Homo sapiens” (McCon- achie 2011:38).

Production and use of complex tools sprang forth in parallel with the evolution of the brain and hand (F. R. Wilson 1999:79, 130, 134). Compared with that of other primates, the human hand is “an organ with a vastly increased manipulative range, with the capability to grasp and control objects of various sizes and shapes, powerfully or delic- ately – and constructively” (168).

The hand is directed by the brain (Pinker1999:12) but in turn it also provides the brain with information, especially that which can only be obtain by acting upon an object (F. R. Wilson 1999:276). As hominids evolved, hand and brain together “began to redefine the demands and possibilities of a life in which forelimbs had been freed of the obliga- tion to support body weight” (79). This happened in a synergistic way since “brain and hand vitalize one another, and the capacity to learn grows continuously as we fashion our own personal laboratory for making things” (295).

Besides the hand's amazing dexterity, the brain's ability to think and hypothesize in abstract terms is of equal importance. Our “ability to conceive an unlimited number of new combinations of ideas” constitutes “the powerhouse of human intelligence and a key to our success as a species” (Pinker 2003:236). Together with abstract thought, the ability to develop new ideas may owe equally as much to our propensity for playful and artistic behaviours, examined further below (see 4.4).

Driven by ideation, experimentation and improvement, technological developmentcan thus be seen as a Darwinian process, akin to genetic evolution but acting on a smaller time-scale, potentially that of a single generation (Boyd 2010:381). Thanks to their craftiness, humans have the advantage of “attacking in this lifetime organisms that can beef up their defenses only in subsequent ones”, a sad proof of which lies in the extinction of several species due to human depredation (Pinker 1999:190).

Humans and their hominid ancestors have been able to conceive dynamic “scenarios in their mind's eye and figure out new ways of exploiting the rocks, plants, and animals around them”, a trait which, as with the hand, “may have co-evolved with language [...] and with social cognition [...] yielding a species that literally lives by the power of ideas” (Pinker 2003:238). As such, the synergy between hand and brain also contrib-

TECHNOLOGY | 19 uted in the development of advanced forms of communication and cooperation (Wilson 1999, Tomasello 2009).

Transmission of knowledge can happen at a primitive level through imitation, a behaviour found in species pre-dating ours (Boyd 2010:104). More than any other species, Humans are prone to imitate from an early age (ibid.). Starting with our ancestors as far as 2 million years ago, imitation became a mainspring for collaborative tool fabrication and use, associated with early forms of art “involving mime, dance, music, and spectacle” (McConachie 2011:42). Mimetic behaviour, essentially a natural learning method, is deeply related to art and play (Caillois 2001, Dutton 2010).

Beyond mimicry, language and cooperation developed together with hand and brain, allowing for the transmission of more complex types of knowledge. More specifically, language may have evolved through the need of communication and coordination in cooperative tasks (Boyd 2010:53, F. R. Wilson 1999:170-171). The evolutionary fitness resulting from their interaction can be illustrated as follows:

“In a species equipped with language, an intuitive psychology, and a willing- ness to cooperate, a group can pool the hard-won discoveries of members present and past [...] Hunter-gatherers accumulate the know-how to make tools, control fire, outsmart prey, and detoxify plants, and can live by this collective ingenuity even if no member could re-create it all from scratch.” (Pinker 2003:63)

The transmission and application of technical knowledge relies on our unique ability to cooperate, requiring a unique kind of empathy and abstraction. This includes hypo- thesizing another individual’s intentions and probable course of action (Boyd 2010:46- 53) leading to the capability for “shared focus of attention at a higher level, differentiated into perspectives at a lower level” (Tomasello 2009:170). The human brain is biologically equipped with a “module” for “intuitive psychology, which we use to understand other people” (Pinker2003:220), facilitating collaboration in tool production but also enabling “subjunctive play, the ability to invent and perform games, drama” among other forms of art and fiction (McConachie 2011:37).

This “ability to create with others joint intentions and joint commitments in cooperative endeavors [...] structured by processes of joint attention and mutual knowledge” is

20 | ART AND TECHNOLOGY AT PLAY referred to as shared intentionality, a trait which seems to be unique to our species and close ancestors (Tomasello 2009:xiii). And thus are humans able to assume heterogeneous roles in pursuit of common goals, including specialized tasks required in hunting and tool manufacture (F. R. Wilson 1999:170-171) – a heterogeneity of roles and specialization so ubiquitous in contemporary life as to be taken for granted, and referred to as “profession”, among other names.

As we have seen, the first steps in technological development are interwoven within the concerted, synergistic development of the human hand, brain, cooperation through heterogeneous roles, advanced social empathy, and distinctly human language. Better communication and tool-making are inevitably and proactively linked to the development of art forms (Boyd2010:9, McConachie 2011:37).

The development of cooperation, language and technology allow for techniques of representation, storage and transmission of knowledge beyond basic imitation (Bell & Gemmel 2009:15). As knowledge can be taught and inscribed, it may in principle – un- like food, water, tools or other material possessions – be “duplicated at negligible cost” and “given away without anything being subtracted from the giver” (Pinker 2003:238). So technological advancement accumulates, allowing humankind to proverbially stand on the shoulders of giants and quickly develop new solutions for emerging problems (Boyd 2010:120). Entangled in human culture, technology finds itself subject to socioeconomic and political forces (Williams & Edge 1996).

2.2 · Culture and Mass Production As we have seen, technology hasn't developed in a void. Its inception, together with the rise of the Homo sapiens, is deeply tied to evolution and social processes.

Increasingly throughout human history, technology has helped to tackle, and eventually automate and abstract many different tasks, fulfilling some needs, creating new ones and in turn (Williams & Edge 1996:876, R. Williams 1990:49); and modifying our natural as well as our social environment (Williams & Edge 1996:890, Thackara 2005:33).

Contemporary reality is quite different from that of our ancestors, even those of the same species, living up to tens of thousands of years ago. In a vast majority of cases,

TECHNOLOGY | 21 we do not manufacture our own tools, nor do we hunt or grow and harvest our own food and resources. We instead buy them in markets and stores, which have in turn obtained them from suppliers and factories, where they are mass-produced using machinery and techniques largely unknown to us as individuals within a system.

The tools and artefacts we use on a daily basis are the cumulative result of thousands of years of technological advancement. Cutting-edge commercial applications of technology are available to us as sanitized, opaque 'black-boxes' (Williams & Edge 1996:877), concealing almost completely their origins and inner workings. We live in a consumer society with highly specialized roles and depend on a plethora of interconnected technological frameworks (Thackara 2005:6). Consider what kinds of knowledge, resource-gathering, refinement, design and production methods are required to produce a smartphone (and afterwards, advertisement and distribution). This includes the microprocessor and memory, screen, battery, casing, software as well as the supporting communications infrastructure.

In 2009, designer Thomas Thwaites attempted to build a simple toaster – something which he found commercially available for GBP 3.99 – from scratch. Beginning with mining raw materials, the whole process has taken a lot more time, effort and money than an everyday purchase, and yielded a barely functional and terribly misshapen artifact: a veritable museum piece of speculative design. Thwaites observes:

“The contrast in scale between between consumer products we use in the home and the industry that produces them is I think absurd – massive industrial activity devoted to making objects which enable us, the consumer, to toast bread more efficiently [...] the scale of industry involved in making a toaster (etc.) is ridiculous but at the same time the chain of discoveries and small technological developments that occurred along the way make it entirely reasonable.” (Thwaites 2012)

Research upon new technologies and applications, as well as the distribution of technical means and products, has equally come to rely on highly complex networks con- stituted by heterogeneous actors, making direct ties to actual human needs hard to follow (Williams & Edge 1996). While there are lobbies exerting influence towards fulfilling capitalist agendas, even these do not exist in a void and they are at some point dependent on politics and legislation, as well as social norms and practices (R. Willi-

22 | ART AND TECHNOLOGY AT PLAY ams 1990, Williams & Edge 1996). As a specific technology passes through different stages of development or maturity, and even after it has become widely available, it will be subject to different kinds of social forces, including forms of domestication and cultural appropriation, as we will see.

2.2.1 · Black Boxes The ‘black-box’ approach in the design (and distribution) of consumer technologies is still the most common way to abstract the underlying technical complexity in the attempt to simplify and streamline the user experience (Suchman2007:188, Williams & Edge 1996:876). Observation and statistical methods are applied by producers in order to improve their products (875), to make them more appealing (in every possible way) to different markets and groups (Suchman 2007:278).

Nevertheless, ‘black-boxing’ introduces a design challenge in the intelligibility of the device, since it will obscure or altogether eliminate affordances which would otherwise provide a mental model of the device's workings and therefore guide the user's actions (Hjorth 2011:14, Moggridge 2007:541, Norman 2002:9), trading actual usability for shallow visual aesthetic.

When the outward appearance of a device is almost entirely unconstrained by its physical content – as it is with most digital devices – then it is certainly easier to play with shape, colour and overall aesthetic. This facilitates differentiations detached from actual performance of the device qua tool, effectively serving as social codes (Barnard 2002:39, Baudrillard 2005:160). A consumer electronics product is introduced and marketed with “an implicit fantasy scenario of its domestic consumption [...] and an ideological rationale for its social function” (William Boddy in Lister et al. 2008:254).

According to Baudrillard “the truth of the object in consumer society” is that it becomes a gadget, “defined by the relative disappearance of its objective function (as a tool) to the benefit of its function as a sign” (Baudrillard 2005:77) within an overall scheme of of differences and meanings consumed through advertising (67, 197).

One can argue that Baudrillard's view seems to reduce objects to mere motivated signs and cast aside any possibility of actual material or functional/technical properties influencing the perception of the objects in use – when such perception comes not of a detached cartesian mind, but rather of a mind within a body with both a his-

TECHNOLOGY | 23 torical and evolutionary past (Boyd 2010, Pinker 1999, 2003, Tomasello 2009).

Baudrillad's vehemence is perhaps rhetorical in nature. As we will see:

“while the fantasies spun around the launch of any media device shape and the symbolic status it may accrue in its everyday usage, shape its meanings and uses, they by no means wholly determine these uses and meanings” (Lister et al. 2008:256)

Objects and artefacts are more than “motivated signs” whose “meaning or social significance [...] primarily derives from outside” (Olsen 2010:145). Rather, they “possess their own nonverbal qualities and are involved in their own material and historical processes”; their intrinsic material qualities can greatly influence their potential usage and how they may directly affect reality (146, 172). This is not to deny the existence or importance of symbolic value, but rather to state that materiality and material properties can certainly influence its symbolic or connotative value in social (and economic) contexts (Barnard 2002:49, Barthes 2000:117, Olsen 2010:146).

Symbolic value is not by any means strictly an affair of modern or postmodern consumer society and advertisement. It is observable as much in “civilized” cultures as in more primitive ones. It is celebrated and amplified through decoration, ceremony, gift- giving, exhibition; and it may equally involve religious artefacts, art pieces or fully- functional tools (Fowler2004, Gell 1998, Miller 2008).

The case is perhaps that in our own “consumer society”, given the relatively new effects of standardization, globalization, black-boxing and advertising, this symbolic role carries an increased complexity and scope; or a different “rationale” which needs to be understood (Barnard 2002:161-166, Hjorth 2011:14, Olsen 2010:32-33). Critics like Baudrillard, although in apparently rhetoric denial of inexorable aspects of materiality, have certainly contributed towards that end.

2.2.2 · Domestication and Appropriation When new technologies are introduced to the market they are commonly quite expensive and specialized. They are initially marketed to companies, who can potentially afford the product, rather than to individuals. Such was the case with computers and mobile phones. As new production techniques are developed, the same device or technology can eventually become accessible to consumers (Fujimoto 2006:80, Lister

24 | ART AND TECHNOLOGY AT PLAY et al. 2008:72, R. Williams 1990:49), who may develop scenarios of usage which complement or even subvert marketing expectations.

As computers were introduced to the domestic market, they were bundled with productivity programs. However, computers were quickly and overwhelmingly appropriated by young males to play videogames (Williams & Edge 1996). This appropriation in turn “shaped the evolution of home computers, leading to the creation of a special- ised market” surrounding videogames (889) which includes not only software but also hardware (game controllers and 3D graphics accelerators), periodicals and more recently, e-sports.

As an extremely versatile tool, computers can be seen as affording appropriation for different uses. But even simpler devices can be appropriated and personalized. In Ja- pan, pagers originally aimed at businessmen were eventually adopted by schoolgirls (a demographic opposite), who developed their own numeric codes (Kohiyama 2006, Okada 2006); and later, as alphanumeric displays became prevalent, expressive emoticons composed from western and Japanese characters, eventually dubbed kao- moji (ibid.). This was a move to “domesticate, humanize and customize” technology, “creating emotion and affect as a type of warmness in the coolness of technological spaces” (Hjorth 2011:79). A similar process of domestication was later exercised upon mobile phones, or keitai (occasionally spelled ketai). In contrast with the bare black “salaryman” mobile phones, japanese teenage girls' keitai are decorated with “adhes- ive seals and jewels” and one or several “straps”, which can be “bigger than the ketai itself, with a plush toy for example” (Kusahara 2010:5). For them the keitai became “more than just a tool”. It is an expression of their group identity and tastes, “something they are highly motivated to animate and to customize as a dreamcatcher, a good luck charm, an alter ego, or a pet” (Fujimoto 2006:80); “a ketai is part of their identity” used in “representing the user's profile toward the rest of the world” (Kusa- hara 2010:4-5).

Curiously enough, the introduction of the telephone in rural North America seems to have undergone a narrative which is quite similar:

“Perhaps the most striking example is the telephone, which was originally conceived and promoted as a business communication tool for conveying price information to farmers, but which was re-invented by people in rural

TECHNOLOGY | 25 areas, particularly women, as a medium for social communication” (Williams & Edge 1996:889)

Specific technologies can be seen to undergo a three-stage paradigm of adoption and social availability, beginning in research or military applications, maturing in business and corporate applications, to finally become available as consumer technologies (Fujimoto 2006:80) – and in some cases even as entertainment and toys. Such was the case of the kite, the radio, the phone and the computer (Fujimoto 2006:80, Lister et al. 2008:76, Williams & Edge 1996:889).

Technology and human culture cannot be dissociated (Lister et al. 2008:404). Specific cultures are characterized by their values, practices and art as much as by the technologies they employ and how they employ them (Pinker 2003:65, Thackara 2005:30, Barnard 2002:37). Once again, this is not a matter exclusive to “primitive cultures” either. The japanese girls' adoption of keitai as symbol of both their lifestyle and indi- viduality is one contemporary example among many.

2.3 · New Media As illustrated by the above examples, a given technology or finished product can be put to use with different purposes, even those which were not anticipated. When used for “communication, expression, representation or imaginative projection” technologies are said to constitute media (Lister et al. 2008:88). In the last decades a great deal of enthusiasm has been shown by consumers and scholars alike, regarding the possibilities afforded through the widespread adoption of digital information and communication technologies, known as new media (253).

New media have come to embody technology in sociocultural use; a use which may not been entirely idealized or anticipated but rather constructed over time “through many complex social transformations and transitions” (56). As we have seen, technology and culture are intimately connected and indissociable.

2.3.1 · What is New Inasmuch as it involves communication and expression between people, usage of technology as media can be seen as re-staging pre-existing cultural conventions (Hjorth 2011:32, Lister et al. 2008:60); thus scholars have often wondered what exactly

26 | ART AND TECHNOLOGY AT PLAY is new about “new media” (Lister et al. 2008:242). While it cannot be dissociated from pre-existing customs and codes, new media make new forms of expression possible, and also provoke questions and ways to conceptualize media and social exchange in general (Hjorth 2011:14, Lister et al. 2008:56). New media can be contrasted with “traditional” media such as newspapers and television both in the fact that they are based on digital technology as well as by the enhanced possibilities for interaction and participation that they offer (Hjorth 2011:32)

Due in great part to their own abstract inner logic, digital computational technologies are quite versatile. The same processor can run different programs. Different uses imply less a replacement of mechanical parts than a replacement or extension of software – information which is virtually weightless and easy to copy, modify and transmit (Dourish 2004:140-141).

While “traditional” (previously existing) media “existed in discrete analogue forms (e.g. the newspaper, the film, the radio transmission)”, new media “converge into the unify- ing form of digital data” and can therefore “be either converted to or generated as a series of numbers which are handled by computers in a binary system” (Lister et al. 2008:422). Heterogeneous types of information are abstracted into binary data and thus can be stored, processed and transmitted using the same methods: source code, programs, 3D models, text, images, audio, video. Potentially anything can be “digit- ized”, given the right methods of extraction or data-mining and enough storage space (Bell & Gemmel 2009:128-133). When compared to physical objects, data is relatively immaterial, making it easy to copy, share and modify – provided a complex infrastructure is available, which is not always the case.

The formal characteristics of digital technology facilitate certain types of usage. This is not to be taken as either technological determinism or an essentialist view, but rather as an acknowledgement of the interplay between technological and social factors, instead of considering these in isolation or as strictly competing forces (56). The versatile and near-immaterial nature of digital data allied with the domestication of computational devices and the increasing ubiquity of digital communication networks served as the technological foundation for new media. Working together these factors enable what is perhaps the defining characteristic of new media: interactivity and participation at a global scale (Manovich 2008:67).

TECHNOLOGY | 27 2.3.2 · Participation and Creation Participation enabled by new media ranges from relatively simple acts such tagging and as sharing content (O’Reilly & Battelle 2009, Vinh 2011), to full-fledged content creation, starting with blogs, moving on to podcasts and video-logs and eventually ar- riving at YouTube channels and self-made internet celebrities (Manovich 2008:67)

In the age of Web2.0 users are frequently framed “as co-developers, in a reflection of open source development practices” as the web is designed “for ‘hackability’ and re- mixability” and “the most successful web services are those that have been easiest to take in new directions unimagined by their creators” (O’Reilly 2005:4). Ideally, new media products and services are designed to be customized by users. This includes those based on the web as well as any device or system which can run networked applications. Developers and designers are faced with “an exercise in negating the designer's authorial privilege” focussing instead on “creating the conditions under which [...] rewarding conversations can happen” (Vinh 2011:131).

Before the Web 2.0 set in, tools for the development of user-created (or user-generated) content became prevalent in digital games (Herz 2001:170-171, Hjorth 2011:54, Manovich 2008:72). The producer-consumer dichotomy becomes blurred as players are allowed to create and share their own content, customizing their experience according to their preferences, creativity and exercising different skills (Morgana 2010).

As consumers creatively appropriate a product, so are the industries driven to exploit user-generated content in their own benefit (Hjorth 2011, Manovich 2008), at points effectively “hijacking [the users’] creative labour to reap financial remuneration” (Hjorth 2011:50).

Web 2.0 further helped communities of produsers to flourish and connect, collaborat- ing in the creation of new content as well as in extensive documentation of creative practices, game guides and fictional lore (Hjorth 2011, Herz 2001, McGonigal 2011).

Participation in the webwork of new media of can also be involuntary, inferred through passive acts, as even basic interactions with digital data (viewing content on a browser, following a link) can be logged and processed by the data's host or third- parties (O’Reilly & Battelle 2009, Vinh 2011). It is stated that the user will then benefit from overall intelligence gathered from data mined in the cloud (O’Reilly 2005, O’Reilly

28 | ART AND TECHNOLOGY AT PLAY & Battelle 2009), and such is indeed possible – for instance, to “track the spread of the flu by noting when people enter words like flu symptoms, aches, sore throat, cough, and fever into the Google search engine” (Bell & Gemmel 2009:112). On the other hand, different types of inference can be made through combined data; and these can potentially be more revealing than mere shopping habits for targeted advertising, giving rise to issues of privacy (Lister et al. 2008:216).

In all its ramifications, new media have expedited the continuous evolution of culture and afforded new avenues for creating and sharing knowledge. As we have seen in contrast with traditional mass media, much of the effect is achieved in enabling interactive modes of participation, blurring at points the distinction between producer and consumer (Hjorth 2011).

In the interplay of availability and the characteristics that it exhibits, new media and its constitutive technologies have become both tools for and a focus of artistic experiences in postmodern practice (Daniels 2008, Kwastek 2008, Manovich 2008); and post- modernism is in many ways associated with new media and its characteristics. Used as tools for artistic expression, new media afford their own peculiar plasticity regarding not only audiovisual forms per se but also aesthetic explorations of time-based behaviour and interactivity (Daniels 2008, Kwastek 2008, Morgana 2010). We shall return to this topic in due time (see 4.6).

New media have indeed something new to offer, and constitute technology in social use, having been “incorporated into the fabric of everyday experience” relatively fast (Lister et al. 2008:242). It must be stressed that new media and the processes they enable exist in the here and now in a material world, not in "a self-enclosed cyberian apartness" (237). Therefore “any progressive understanding of the potentialities of new media in everyday life is only possible [by] recognising the materiality of these technologies and their place in everyday lived experience” (242).

2.4 · Augmentation Our everyday life is filled with technically complex items and wondrous possibilities, developed and passed down the generations, which effectively extend our innate or a priori biological capabilities. Such motor, sensory or infrastructural enhancements allow us to reach and thrive in places otherwise inhospitable; or study atoms and galax-

TECHNOLOGY | 29 ies alike.

2.4.1 · Tools as Extensions Philosopher Henry Bergson postulated early enough that:

“If our organs are natural instruments, our instruments must then be artificial organs. The workman's tool is the continuation of his arm, the tool- equipment of humanity is therefore a continuation of its body. Nature, in endowing each of us with an essentially tool-making intelligence, prepared for us in this way a certain expansion. But machines which run on oil or coal [...] have actually imparted to our organism an extension so vast, have endowed it with a power so mighty, so out of proportion with the size and strength of that organism, that surely none of all this was foreseen in the structural plan of our species.” (Bergson, quoted in Lister et al. 2008:92)

Evolution is not a matter of planning and a species cannot reasonably be said to have a “structural plan”. Yet evolution has itself “evolved in numerous ways” (Boyd 2010:120), yielding second-order mechanisms of variability which act in the time- frame of less than a generation, such as the immune system or even Human intelligence and culture (Plotkin 1994). As mentioned before, our inventiveness and sharing of knowledge gives us an incredible advantage over other species (Pinker 1999:190).

Merleau-Ponty’s example of the blind man’s stick illustrates how a tool in use “becomes an extension of the body”, “an extrasensory organ that extends the radius of the blind man's touch” (Olsen 2010:130). Heidegger’s concept of equipment (Zeug) and how our attention focuses to the accomplishment of a task effectively incorporating the tool into our proprioception (zuhanden) is also closely related (Dourish 2004:139, Olsen2010:69-75).

A more technically complex example can be had in telesurgery. Surgical procedures require knowledge, concentration, expert tools and a steady hand. Applied technology allows for certain surgical procedures to be accomplished over a distance, mediated by networks, screens and robotic arms, effectively augmenting the surgeon’s agency. While this mediation could initially be thought of as somehow disruptive, it has been found that operating surgeons may “experience themselves as proprioceptively shifted more directly and proximally into the operative site, with the manipulative instru-

30 | ART AND TECHNOLOGY AT PLAY ments serving as fully incorporated extensions of their own acting body” (Suchman 2007:266).

Human-machine assemblies are not exclusively a thing of post-industrial life. As discussed before, technology and culture are interwoven. Human society can be seen an assembly of heterogeneous actors “in imbroglios and mixtures, the seamless and rhizomelike fabrics of culture and nature that link humans and non-humans in intimate relationships” (Olsen 2010:138; see also Law 1992).

Yet now more than ever, technology facilitates much of our interaction with the world, not constrained to the immediate surrounding context, allowing us to perceive and act remotely in many ways, in heterogeneous modes, scales and rhythms. Writing about objects and consumer “gadgets” in 1968, Baudrillard could have been prophesying the smartphone when describing how “a few simple gestures evoke its power without making it manifest [...] cloaking and eliminating an energy that has been made into an abstraction” (Baudrillard 2005:56).

Long before we had tele-surgery and smartphones, we had magnifying glasses, tele- scopes, test tubes, chemical reagents and other scientific paraphernalia. Perceiving phenomena which would be otherwise (biologically) out of our reach allows us in turn to better understand our environment and its properties. This opens up further possibilities to unravel natural phenomena in the name of scientific knowledge, but also to philosophically contemplate the vastness of space or the semblance of deliberate design in evolution.

2.4.2 · Cyborgs Inseparable from technology, a human may be conceptualized as a cyborg, “a hybrid of machine and organism, a creature of social reality as well as a creature of fiction” (Haraway 1987:149) constituting “a kind of disassembled and reassembled, postmodern collective and personal self” (163).

Digital technologies were once seen as effectively constructing a “cyberian apartness” from the pre-existing natural, physical world (Lister et al. 2008:237), giving way to pos- tulations (and dystopian fictions) of futures where disembodied minds wander virtual worlds (Viseu 2005:5). Instead, the cyborg (or augmented human) is instead conceived in a paradigm where the human body constitutes “an integral component of the global

TECHNOLOGY | 31 sociotechnical information systems” (6). This “digital embodiment” of “folding corpor- eality and code across its many differentiated instantiations” is seen “as a process of living in information culture” (Munster 2006:184).

While this may all sound overly positive, augmentation is not without risks. In McLuhan’s work we find the concept of technological extension associated with the counter-effect of amputation, anaesthetization or atrophy of other capabilities (McLuhan 2001). McLuhan’s points seem to harken back to naive Lamarckian evolution theory, where organs atrophy over the timespan of a generation and the changes are then passed to descendants. Yet this dual effect of extension/amputation manifests beyond direct sensory perception patterns, onto the human psyche and social processes. Culture and customs, conceptualized in transmissible units as Dawkins’ memes, do “mutate” and “replicate” much faster than genes (Pinker 2003:65-66); and overall McLuhan’s views, although sanctimonious, do hold a measure of truth and clever observation. They are symptomatic of a clash between the slow evolution of biological traits and the overwhelming (in speed and complexity) potential of technological augmentation, to which we will soon return (see 2.6).

2.5 · Magic and Enchantment In social use, technology serves more than a strictly functional role. A view of technology as exclusively functional is short-sighted, especially when all around us technical means are used in mediating communication, creating and reshaping possibilities, facilitating the exploration of ideas and questioning the boundaries of the body and personhood. While audiences immerse themselves in the beautiful 3D and realistic sur- round sound of the latest Star Wars movie, fashion designers embed electronics into their creations as amplifiers of fantasy (Seymour 2008:12), a researcher in Osaka creates a cutting-edge android doppelganger (Nishio et al. 2007) and japanese girls are inseparable from their mobile phones which they treat as magic mirrors and lucky charms (Fujimoto 2006:87).

2.5.1 · Technological Super-powers Considering the possibilities for the design of ubiquitous computing technology, Adam Greenfield points out that:

32 | ART AND TECHNOLOGY AT PLAY “[...] folklore is replete with caves that open at a spoken command, swords that can be claimed only by a single individual, mirrors that answer with killing honesty when asked to name the fairest maiden in the land, and so on. Why, then, should anyone be surprised when we try to restage these tales, this time with our technology in the central role?” (Greenfield 2006:119)

Indeed one can often find technology in use as “diegetic artifact”, characterized as magic within the fiction of immersive role-playing activities (Montola et al. 2009:168). Par- ticipants can even be provided with “body extensions made possible by wearable technology” in order “to create the illusion that the player has superhuman abilities” (ibid.).

Portrayed throughout modern fiction we find characters (of all shades throughout) whose extraordinary abilities are made possible through special technology e.g. super serums, utility belts, powered suits of armour, body implants, talking self-driving cars, submarines, spaceships and time-machines. Thanks to frameworks of interconnected wireless devices, sensors, databases and algorithms we can easily picture certain applications of technology as granting us “super-powers” (Bell & Gemmel 2009:17, O’Re- illy & Battelle 2009:6).

If this sounds too optimistic, rest assured it is not the case. The “powers” granted through technology can be used in many ways. Fiction has seen portrayals of enthusiasm towards technological utopias, but has also conveyed cautionary tales or the outright depiction of nefarious possibilities – the myths of Icarus and Golem, Mary Shel- ley’s Frankenstein, Aldous Huxley’s Brave New World and most of the villains in the James Bond series serving as fine examples.

In our everyday life, we frequently resort to such “super-powers” as the ability to converse at a distance, forecast the weather or summon up lively representations of places we have never actually been to and events we have never witnessed first-hand. Many of these capabilities rely not on a single device, but on networks of people and machines. As “communication and computing technologies amplify human talents for cooperation” (H. Rhinegold quoted in Thackara 2005:129), digital networks and services can enable people to connect, collaborate and together achieve results which would be otherwise out of reach or too daunting for a single individual or small group (Mcgonigal 2011:268, Montola et al. 2009:128, Thackara 2005:222).

TECHNOLOGY | 33 2.5.2 · Magic, Mysticism and Misdirection All these extraordinary effects are taken for granted, while otherwise they would be as wondrous and awe-inspiring as the best magic tricks. However, thinking about technology as magic in anyway is commonly discouraged (Baudrillard 2005:77, Pinker 2003:236). Technology (viz. applied scientific knowledge) is rational and methodical, whereas magic is equated with superstition and ignorance:

“In fact, what is often called bizarre, nonrational or incomprehensible in traditional religions becomes rational and comprehensible when viewed in terms of the propositions of theoretical thinking” (Penner 1989:18).

That which is perceived as “magic” to a spectator is a process which the individual lacks immediate knowledge to explain; or explains in religious, supernatural, or otherwise non-scientific manner (Lamont & Wiseman 1999:83). The perception of phenomena as magic comes as “a by-product of uncertainty” not in the world itself but in “the knowledge we have about it” (Gell 1992:57). As such, the mix of wonder and mystery which construe such an experience can motivate scientific discovery as “the rational pursuit of technical objectives using technical means” (ibid.).

In popular fiction “the inventor is usually portrayed as the semblance of a delirious maniac”; indeed “some of the most infamous names in the history of technological invention derive their inspiration from deeply irrational, mythological, and even outright mystical sources” (Kluitenberg 2006:160). Nikola Tesla, Thomas Edison, and Alexander Graham Bell’s assistant Thomas Watson are all examples of famous technologists who not only had an interest in but were rather motivated by spiritual or paranormal phenomena (Kluitenberg2006:165-170, Midal 2011:95). They have carried themselves onto a “prominent place in the history of technological invention [...] exactly because of their mystical inclinations” (Kluitenberg 2006:161).

Despite their origin being logically explained or known, certain things can still possess an enchanted quality: they are peculiar in the way they constitute a source of aesthetic pleasure or in some way excite our cognitive faculties. Even after scientifically explained, an aurora borealis (colloquially known as “northern lights”) will still seem magical; and there is always something uncanny about looking at an X-Ray image of your own body. Magic can even be intentionally used as a strategy or aesthetic in the

34 | ART AND TECHNOLOGY AT PLAY design of interactions which “tend towards revealing effects while hiding the manipu- lations that led to them” (Reeves et al. 2005:745).

2.5.3 · The Technology of Enchantment Technical means can be employed to instil a sense of awe and wonder to be perceived as a quality of enchantment. As opposed to natural phenomena like the aurora borealis or an eclipse of the sun, the “enchantment” results from the work of a skilled craftsman or artist. As in a performance from a magician, the wonder and awe caused by the “enchantment of technology” arises from a difficulty in “mentally encompassing their coming-into-being as objects in the world”, of re-tracing “a technical process” at their origin, translating into resistance “to being possessed in an intellectual rather than a material sense” (Gell 1992:44-49, cf. Gell 1998:81-86).

The process of crafting artefacts which are “enchanted” to be psychologically effective in turn constitutes a type of knowledge and skilful practice, a “technology of enchantment” as “a vast and often unrecognized technical system, essential to the reproduction of human societies” (Gell 1992:43). Enchantment confers a powerful kind of social agency to objects (Gell 1998:68-72). It is a psychological “component of a social technology” which “encourages and sustains the motivations necessitated by social life” and “often essential to the psychological functionality of artefacts, which cannot be dissociated from the other types of functionality they possess.” (74)

The vast majority of (if not all) art forms are indeed brought about through technical means, through “artefacts and processes of economic or social usefulness” (Lister et al. 2008:429), not least of which is the knowledge and skilled work of the artist, artisan or craftsperson (Dutton 2013:273). While it would be reductionist to explain art solely as a technical system, it is beneficial (especially within the context of the present text) to acknowledge the “technology of enchantment” as a way of understanding the sym- biosis of art and technology.

Furthermore, understanding art as cognitive stimulation helps us to outline an evolutionary explanation for affinity to it, as well as its evolutionary and social role (Boyd 2010), which we later will look upon (see 4.5.1). Suffice to say for now that the quality of “enchantment” resulting from cognitive pleasure is yet another possibility afforded through technical means, another role which technology can serve: to confer a quality

TECHNOLOGY | 35 to the object itself which not only complements strictly functional use but can even render an object more effective in its purpose (Barnard2002:49-70, Gell1992:44-46, Olsen 2010:11).

2.6 · Overloaded Human techniques and related processes are termed artificial as they do not otherwise occur naturally. Yet as we have seen, they are themselves the result of evolution – a second-order mechanism of variability (Plotkin 1994, Boyd 2010). Technology extends our a priori biological capabilities by taking advantage of natural laws.

The fact that technology can be subversive and disruptive was highlighted among others by media theorist Marshall McLuhan (McLuhan 2001), according to whom media can alter “sense ratios and patterns of perception steadily and without any resistance” (19). As every “extension” is accompanied by an “amputation”, we find the idea that technology can have (or eventually comes to have) unintended or unforeseen negative consequences.

As expressed in Bergson’s words, technology has endowed humans with “a power so mighty, so out of proportion with the size and strength of that organism, that surely none of all this was foreseen in the structural plan of our species” (Henri Bergson quoted in Lister et al. 2008:92). Once again, while evolution hardly follows a “plan”, the idea is indicative of our struggle to fathom the seemingly exponential development of technology and the possibilities it affords.

2.6.1 · Beyond Evolution Evolution leading towards the modern Homo sapiens did not happen in technology- saturated environments. The tools used 100.000 years ago, although always interwoven in social processes, were relatively simple and certainly did not entail such large-scale modifications of the environment (and impacts on lifestyle) as modern transportation systems and energy sources do. Even as human biology remained relatively stable, “material culture [...] expanded, and then exploded”, passing “from a handful of stone and bone tools at the beginning [...] to agricultural fields and villages at the 90 percent mark, and then – in a virtual eye blink – to prodigiously elaborate technologies” (Thackara 2005:30).

36 | ART AND TECHNOLOGY AT PLAY The evolution of technology is a cumulative process happening at a much faster rate than genetic evolution. In the same way that other species “cannot evolve defenses rapidly enough” against human technologies (Pinker 1999:190), humans may not biologically adapt fast enough to meet the demands and expectations of the technological environment they themselves created (Pinker 2003:219-221).

The clock is a technical device now truly ubiquitous and taken as granted. Originally based on mechanical technology, it imparts a precise rhythm to our daily life and influences our perception of time. In Europe the clock was introduced in the Late Middle Ages, invented by monks “in order to structure prayer times”, later adopted by trades- man and mechanics “not only to mark the passage of time, but also to dictate the scheduling of activities” and as result to “regulate the speed of action and therefore the pace of society” (Thackara 2005:33). Before the invention of the clock, we were regulated by natural events (day-night cycle, seasons) and “work until a task was finished, or until the sun went down”; the start and end of human events did not refer to a scientific, mechanically-kept measure of time but rather to other events of a more heterogeneous nature (ibid.). And so the perception of “lived time” prevailed, as in Henry Bergson’s concept of durée: time perceived as a sequence of discrete experiential events instead of by reference to a mechanical index of discrete scientific measures (Thackara 2005:33-38).

Once introduced, “the clock would become the ultimate connection machine, organizing and binding the lives of millions into an integrated social, economic, and religious system” (Kluitenberg 2006:59). The clock has perhaps played as important a role as the steam engine during the industrial age (Thackara 2005:33).

Thanks to clocks, schedules can be optimized to maximum efficacy (at least in theory). The efficiency of the clock makes possible the speeding-up of processes, in turn setting high standards of speed on “the way we communicate, eat, travel around, and innovate products” (Thackara 2005:29). It achieves the effect of “concretising and divid- ing up time”, taming it into a “domesticated quantity” - and thus into a commodity or “object of consumption” (Baudrillard 2005:50).

Yet the chronological time of machines contrasts with the qualitative experiential time of humans, especially in what regards social aspects of life, for which we may “pay a social and personal price” (Thackara 2005:33). While pre-historic humans “survived on

TECHNOLOGY | 37 three or four hours of work a day” and used most of their time for “socializing, ritual, artwork, or just relaxing”, we now manage (and perceive) time in a way “dictated by the logic of systems beyond our control” (29, 34). The widespread prevalence of the clock as regulator of activity has created certain expectations in human society and behaviour – punctuality, regular rhythm of work and standardized vacation periods, for instance – and clashes between personal experiences of time and regulated “public time” (33).

As technology is developed to satisfy human needs and augment performance, it also opens up possibilities which may be beyond human capabilities. As an example, we can certainly develop manned aircraft that fly and maneuver above supersonic speeds; the limitation is how much G-force the actual pilot can handle.

Granted, even basic biological traits leave plenty of room for flexibility and adaptation within a single generation. The human brain is an amazingly versatile organ, able to learn and access a vast repertoire of strategies and behaviours and thus allowing for a faster natural mechanism of variability and adaptation (Plotkin 1994).

Still, the power of the brain is far from limitless: as any biologically evolved feature it is extremely cost-effective. Overall, the mind exists to “anticipate the immediate future and guide action” (Boyd 2010:134). For this, it relies on information “costly to obtain and analyze” and so minds and senses are evolved (naturally selected) to “gather and process information appropriate to particular modes of life” (14) and “register the regularities pertinent to particular species and to infer according to rough-and-ready heuristics” (134).

Our minds keep us “in touch with aspects of reality [...] that our ancestors dealt with for millions of years” (Pinker 2003:219), evolved for “the lifestyle of small groups of illit- erate, stateless people who live off the land, survive by their wits, and depend on what they can carry” (221). Yet conditions started changing quite rapidly:

“Our ancestors left this lifestyle for a settled existence only a few millennia ago, too recently for evolution to have done much, if anything, to our brains. Conspicuous by their absence are faculties suited to the stunning new understanding of the world wrought by science and technology. For many do- mains of knowledge, the mind could not have evolved dedicated machinery,

38 | ART AND TECHNOLOGY AT PLAY the brain and genome show no hints of specialization, and people show no spontaneous intuitive understanding either in the crib or afterward.” (ibid.)

Given the extraordinary possibilities afforded by technology – some, like the clock’s discrete time, so prevalent that they are almost enforced – there is often “a mismatch between the purposes for which our cognitive faculties evolved and the purposes to which we put them today” (219).

The human brain is not a digital microprocessor or general-purpose binary calculator, although in some fields of study it is usefully modelled as a computer (Pinker 1999:68- 69). As such, people won’t often try to “multiply six-digit numbers in their heads or re- member the phone number of everyone they meet” (Pinker 2009:219, cf. Boyd 2009:39). The way the mind works “makes the most sense when it is explained in terms of beliefs and desires, not in terms of volts and grams” (Pinker 1999:314).

2.6.2 · Cognitive Load Digital technology creates expectations of speed and efficiency in what regards data processing and communication (Thackara 2005:29-38). Binary data can easily be converted and processed. Large amounts of potentially ever-shifting information may be available to be used at any given time. Nevertheless “while our information technology may be digital in nature, the human beings interacting with it will always be infuri- atingly and delightfully analog” (Greenfield 2006:133). Eventually, a user may have to make sense of the data according to human capabilities (for instance, a graph display or similar abstraction is still needed if a human is to interpret the data in real-time) and need time to ponder and choose between courses of action.

Our brain processes an incredible amount of information automatically (or subcon- sciously), including piecing together and correlating signals from our senses, detecting patterns on different types of stimuli or activating organs according to motor memory (Pinker 1997:8-12). But we are also able to make logic inferences, calculations, spin hy- potheses or simply pay attention to phenomena happening in the world – in some cases due to a “breakdown” or disruptive change of conditions (Olsen 2010:73, Dour- ish 2004:139. In such cases, the issue is “bumped up into consciousness” to be addressed explicitly and consciously in the “severely limited space of working memory” where the mind “consciously attends to and manipulates information” using actual

TECHNOLOGY | 39 mental representations (Boyd 2010:47).

We now live surrounded not only by a “pre-human” natural world but also by artificial arrangements and mediated information, to which our minds will pay meaningful attention (consciously or otherwise) to keep up-to-date and up-to-shape (Boyd 2010:94). In a world where technology demands so much of our attention, we “pay a price [...] in personal brainpower” or cognitive load (Sterling 2005:21) as well as in the quality of our social relations (Thackara 2005:33).

The human mind has an appetite for patterned information, which “falls into meaningful arrays from which we can make rich inferences” (Boyd 2009:14); and for this reason our attention is captured and held by music, stories, games and art in general (85-91).

As a result, we often feel “flooded” as information comes to us “unfiltered, unsorted, and unframed” making it difficult to “select what's important” in specific situations (Thackara 2005:162); and we are led to shift our focus away from other tasks at hand (Sterling 2005:21) and change our habits and way of thinking:

“These emerging technologies are taxing our sensorial and perceptual systems in novel ways [...] New technologies repeatedly make fresh demands, asking us to learn new gestures, vocalizations, and eye patterns. The enduring ones eventually configure our bodily techniques, transforming how we navigate physical space, focusing what is left of our powers of concentration, even organizing our patterns of thought.” (Hunt 2011:51)

Even though new media does not entirely constitute a “self-enclosed cyberian apartness” (Lister et al. 2008:237), an experience “set off from everyday institutional identities and offline settings and practices” (Kato 2006:106), we are still led to frequently and abruptly shift our between our immediate surroundings and the screen of a mobile communications device in order to manage our “simultaneous presence in multiple social situations” (Ito 2006:13, cf. Hjorth 2011:130). Additionally, excess information can create a feeling of phenomenological distance, even when it is meant to augment the user’s surrounding context (Slavin 2011).

2.6.3 · Mobile Interaction Computers were originally used in office settings, where our attention could be almost

40 | ART AND TECHNOLOGY AT PLAY entirely devoted to them. As such, they established assumptions which are still prevalent in the interface design of nowadays smartphones which, given their small size and specific context and patterns of usage, are quite impractical to use as one would a desktop computer (Greenfield 2006:40, 86, Montola et al. 2009:180, Zwick 2006:100).

Mobile devices still present a challenging arena for interaction design; being relatively recent and quickly changing in form factor and capabilities, they have “fewer standardized forms of input” matching the variability in context of usage as well as complying with ergonomic factors such as the “the size and the motor functions of the human hand” (Zwick 2006:50). As the characteristics of digital information and supporting hardware allow a variety of potentially heterogeneous uses, it becomes increasingly hard “to create an unambiguous link between the physical form, the interaction concept and the software application” and so “interaction concepts must be made more general” (76) or even “borrow control elements from the old medium” (100), sometimes blindly.

On the other hand, this prompted the research and design of different forms of interaction with mobile devices, from physical components like thumb-sticks and jog-dials to speech- and gesture-recognition (Greenfield 2006:40, Montola et al. 2009:180, Ant- onelli 2011:11); as well as ways to actively operate a device without looking at a screen, based on an increasing array of sensors and actuators (Montola et al. 2009:181, O’Reilly & Battelle 2009:4, Zwick 2006:63).

A pioneering approach coming from the field of the arts can be seen in the work Mo- bile Feelings by duo Christa Sommerer and Laurent Mignonneau. The artists developed as series of specially designed mobile devices allowing users to “communicate with strangers through virtual touch and body sensations including smell and sweat” (Sommerer & Mignonneau 2003). Stimuli include “a tickle, a vibration, a small wind or humidity, a pulse, a push or a slight stroke, creating a strange and perhaps erotic ambiguity” (ibid.).

Within design research, a noteworthy example is IDEO’s Social Mobiles project, which explores the use of mobile communication devices through unconventional shapes and modalities (“Social Mobiles” 2002). The proof of concepts included an electric shock mobile, a speaking mobile, a musical mobile, a knocking mobile and a catapult mobile (ibid.). A more recent example is the excellent work of Fabian Hemmert (et al.)

TECHNOLOGY | 41 in exploring embodied and affective forms of interacting with mobile devices, including weight-shifting, shape-changing and reactive posturing, and even wetness and airflow (Hemmert et al. 2013, 2010a, 2010b).

In commercial applications, however, the screen remains the primary interface (Zwick 2006:30). Other senses are left underutilized (Moggridge 2007:515) “in favour of the once much-privileged visual” (Hjorth 2011:135) which is addressed as “the most powerful of the human senses” (Zwick 2006:30) commands a lot of our conscious attention.

When technologies are initially used in professional applications, specialized operators can be trained in order to deal with the characteristics, limitations and expectations imposed by the underlying system. A consumer-turned-user, however, may effectively be at a loss to handle all the information and possibilities a technology can potentially offer (Suchman 2007:188, 262). Consider the following example:

“There have been many attempts to transfer the pilot's augmented reality to the driver of a car, such as the head-up display installed in the Peugeot 508 [...]. For the most part these experiments have underperformed or failed. Drivers are different from pilots; their sense of the world is informed not just by what they are focused on but by what they are not focused on, what lies in their periphery, what they hear, what catches their eye.” (Slavin 2011:172)

At an early stage, when the introduction of new technologies to the consumer market as new products becomes feasible, there is a “paucity of links between designers and potential users” (Williams & Edge 1996:888) as the latter may not constitute a demographic a priori but is rather elaborated as products mature and their usage becomes more commonplace (Suchman 2007:189). As such, designers rely on market surveys and “their understanding of technological opportunities” to conceptualize “how these might be taken up” by consumers; as such, devices and services end up being designed to be “pushed” to potential users instead of fulfilling specific needs (Williams & Edge 1996:888).

2.6.4 · Reaching a Limit In 1965 Intel co-founder Gordon Moore predicted that the number of transistors on a

42 | ART AND TECHNOLOGY AT PLAY microprocessor chip would double roughly every two years; this statement became colloquially known in IT-culture as Moore’s Law, and it has proved sufficiently accurate for half a century (Waldrop 2016). Computational devices have undergone an incredible increase in speed and memory, paired with a reduction in size and energy consumption, and so developers consistently “came up with applications that strained the capabilities of existing chips” (ibid.). While the first personal computers “focused on routine and simplified information processing activities, such as payroll and account- keeping” (Williams & Edge 1996:882), smartphones, tablets and other digital devices justify their purchase through complex applications and come bloated with unnecessary extra features (Thackara 2005:186).

The fact that new media technologies are turned into products designed for users often unknown in advance may still serve as an excuse for repeating heterogeneous black-boxed designs (Suchman 2007:188, Moggridge 2007:541). In the flexibility afforded by the combination of microchips, software and network infrastructures, they are developed to be further configured or personalized as “a medium [...] elaborated in use” (Suchman 2007:278, cf. Williams & Edge 1996:874-875).

Computer-based consumer technologies come “festooned with baroque amounts of functionality” (Sterling 2005:18) and can be complex enough that they in turn imply “configuring the user” to their internal logic (Suchman 2007:189). Furthermore, they increasingly make use of the ability to “actively nag people” for “extensive, sustained interaction: upgrades, grooming, plug-ins, plug-outs, unsought messages, security threats, and so forth” (Sterling 2005:11).

Yet the laws of physics are apparently not without limits: Moore’s Law is becoming un- feasible after all and the industry is forced to turn its focus onto specific needs and modes of usage, instead of “blindly” churning out faster and smaller chips (Waldrop 2016).

At this point it becomes even more important to realize that the success of specific technologies as consumer products “does not simply reflect their functionality and price, but also the extent to which they are compatible with the skills, understandings and habitual practices of potential users” (Williams & Edge 1996:878). Consumer technologies need to be designed taking into account the fact that humans employ “a wide range of experience and (often tacit) knowledge” in “dealing with poorly defined prob-

TECHNOLOGY | 43 lems (of problem recognition, of decision-making in contexts of uncertainty, of dealing with ambiguity)” which are admittedly “difficult, if not impossible, to formalise and appropriate within software systems” (Williams & Edge 1996:882). Humans are “designed for a much richer existence than processing a dribble of data from a computer screen” (Thackara 2005:62).

Innovation in consumer technology should embrace the peculiar traits of our inherent sociability and communicative capabilities in order to augment and complement them, instead of filling up the world with gadgets and replacing social processes with human-machine interfaces based on naive conceptions and designs from last century’s computers (Moggridge 2007:xiv, Thackara 2005:99).

It is necessary to stop reiterating the “narrow definition of what it means to be human” prevalent in technical systems (Dunne & Raby 2013:34) and resist the compulsion “to reduce all human knowledge and experience to symbolic form”; instead we need to value “the knowledge and experience that we have by virtue of having bodies” (Thack- ara 2005:62).

44 | ART AND TECHNOLOGY AT PLAY 3. EMBODIED INTERACTION

The biggest motivator behind the present inquiry was the limitation imposed by screen-based media used in mobile and location-based contexts. In experiences where participants are led on the traversal and (re)discovery of physical spaces, with their focus on the layout and surrounding beauty, it seems inadequate (if not unreas- onable) to often require their attention to be focussed on a palm-sized touchscreen. This is an aspect of a jarring discontinuity – accepted as a necessary evil on a daily basis – between dealing with a lived material world to which our faculties are evolutionarily fine-tuned and acquiescing to the abstract logic of digital information systems.

Coming from a background in computer science and research in computer-supported cooperative work (CSCW), Paul Dourish laid a foundational approach to the design of human-machine interactions, based on our phenomenological existence in a world where the physical and social intertwine. Building on existing work and borrowing heavily from sociological and philosophical sources (more prominently the phenomenology of Heidegger and Mearleau-Ponty), he dubbed this approach embodied interaction (Dourish 2004).

Criticizing the predominant cartesian-dualist mentality in the conception and design of computer systems and applications, Dourish attempts to capitalize on the fact that human minds are not completely abstract or independent of a bodily existence; but rather that cognitive and motor capabilities are interrelated and fine-tuned for our material and social life-world (ibid.).

As any other organ in our body, the brain is to a great degree shaped by an evolutionary past. This is not a philosophical stance but rather a scientific fact (Pinker 2003). While highly plastic and incredibly industrious, the brain exhibits inherent structures shaped by basic survival needs and selective pressure (ibid.). In the case of humans and their close ancestors, the brain developed in a synergistic way with material and social aspects of life, including the refinement of the hand, tool usage, language and storytelling (Boyd 2010, F. R. Wilson 1999, M. Wilson 2002).

Wherever humans are concerned interaction is always “embodied”. It does not make

EMBODIED INTERACTION | 45 sense to think in terms of disembodied minds in the situated context of interaction, especially in a world where ubiquitous computation has become an everyday reality. Yet “embodied interaction” is not a redundant term or a statement of fact. Rather, it is the use of this knowledge in the field of HCI, taking advantage of the ways in which action, perception and sense-making are intimately and evolutionarily connected (Dour- ish 2004, M. Wilson 2002).

While the term “embodiment” can have several (equally valid and often interrelated) interpreted meanings, Dourish clarifies that his use of the term does not solely highlight the physical reality of our existence. While he borrows from Merleau-Ponty, whose concept of “embodiment” can be seen to “draw particular attention to the role of the body” (114-115), Dourish seems to emphasize embodiment as “participative status”:

“So, just as this perspective argues that we act in the world by exploring its physical affordances, it also argues that our social actions are ones that we jointly construct as we go along.” (18)

Dourish’s succinct definition of “embodiment” (in the context of his work) is thus a take on the connection between action, meaning and collaboration:

“Embodiment is the property of our engagement with the world that allows us to make it meaningful.” (126)

As such, besides the more physical aspects of or phenomenological relationship with the world, it also frames interactions with (and through) technical artefacts as being “part of a richer fabric of relationships between people, institutions, and practices” (56). Compounding on Dourish’s CSCW research background, embodied interaction “gives pride of place to interaction” as “the means by which work is accomplished, dynamically and in context” (4).

This position is greatly influenced by Lucy Suchman’s research into the use of technical artefacts in collaborative contexts (Suchman 2007). Suchman regards interaction as “the ongoing, contingent coproduction of a shared sociomaterial world” instead of reducing it to an “exchange of messages” (23, original emphasis). More importantly, her research shows that humans do not act according to highly-detailed abstract formal plans. Rather, plans for action are loosely-formulated, continuously reviewed and

46 | ART AND TECHNOLOGY AT PLAY open to improvisation (Dourish 2004:72). Plans are abstractly formalized only down to “the level at which it makes sense to forego abstract representation and rely on the availability of a particular, embodied response” (Suchman 2007:185). Nevertheless, for a long time “the planning model that so dominated cognitive science was also the basis of the design of interactive devices” (Dourish 2004:72). The planning model is of course connected to abstract theories of mind under a long-lived tradition of cartesian dualism (18) and fails to accommodate for a sociomaterial world of actual working practice, which “is always dynamic, arising as a way to mediate between processes and the circumstances in which they are enacted” (63).

Computers and digital media may be used to represent the world in an abstract way, and their advantage is precisely the versatility allowed by the increasing level of abstraction (81-82); but in the end they still refer to the world. Given that “we act through [computation] to achieve effects in the world” our interactions with and through technology carry intention (137).

In human-computer interaction, as in other contexts, our behaviour “is reflexively constitutive of the world's significance, which in turn gives behaviour its sense” (Suchman 2007:15). As such, an embodied approach to interaction must promote accountability, considering “the reciprocality of action and understanding” also in social and collaborative settings, where “being a competent member [...] is being able to engage in action in ways that are recognizable to other members” (79).

Humans have developed the ability to cooperate in unique ways, through psychological mechanisms constituting what Michael Tomasello terms shared intentionality:

“Shared intentionality involves, most basically, the ability to create with others joint intentions and joint commitments in cooperative endeavors. These joint intentions and commitments are structured by processes of joint attention and mutual knowledge, all underlain by the cooperative motives to help and to share with others.” (Tomasello 2009:xiii)

Humans are able to collaborate towards a shared goal assuming individual roles. This implies the ability to consider one’s role among other individuals and as such, the ability to simultaneously understand and compare different perspectives (Tomasello 2009:70). Based on the tacit, evolutionarily tried-and-true assumption “that we share a

EMBODIED INTERACTION | 47 common reality, that we act rationally within that reality” intersubjectivity becomes possible as “a practical achievement of social actors, a response to the practical problems of engaging with each other in concerted social action” (Dourish 2004:112). In- deed we are all practical sociologists to a degree (113). Our brains have, among other innate capabilities, a specialized cognitive faculty for psychology and empathy, to hypothesize and assess other’s mental states and intentions (Pinker 2003:220). In the absence of a telepathic hive-mind, this is achieved through conscious and subconscious interpretation of the actions of others under the assumption of rationality (McCon- achie 2011, Boyd 2010, Pinker 2003, Suchman 2007), “within a pattern of goals, causes, requirements, and motivations” (Dourish 2004:112). It is reasonable to hypothesize that faculties assisting empathy and cooperation have synergistically co-developed over our evolutionary history, granting advantages over other species or groups (Tomasello 2009, Pinker 2003).

After introducing the supporting concepts, Dourish arrives at a succinct definition:

“Embodied Interaction is the creation, manipulation, and sharing of meaning through engaged interaction with artifacts” (Dourish 2004:126).

In order to illustrate aspects of his foundational approach, Dourish resorts to existing examples in the fields of tangible and social computing (Dourish 2004:116). The latter is greatly based on the concepts of accountability and intersubjectivity outlined above. The present work, however, can be said to have a stronger focus on the former, prompting a closer look.

3.1 · Tangible Interfaces A greater part of our everyday interactions with computers and digital media still relies on windows, icons, menus and pointer (WIMP) or derivatives thereof. The now-ubiquitous touchscreen dispenses with an icon for the location of the mouse, as the user’s fingers are now directly touching the display, but the overall metaphor still stands.

WIMP interfaces (and derivatives) are originally based on interaction scenarios with desktop computers, where users are assumed to be within a stationary working environment, their undivided attention on the machine’s graphical user interface, or GUI

48 | ART AND TECHNOLOGY AT PLAY (Ishii & Ullmer 1997, Dourish 2004:27). Nevertheless, they make use of metaphors based on our understanding of physical space and objects within (Dourish 2004:88, Moggridge 2007:462) allowing for “direct manipulation” of “the abstract objects that make up the system's conceptual model” (such as files, folders, servers or media) constituting a virtual “inhabited world in which users act” (Dourish 2004:13).

In the late 1990’s research in HCI began to (re)address the possibility of extending digital interfaces to everyday objects and spaces, led principally by Mark Weiser’s vision of ubiquitous computation (Weiser 1991). Considering material and social aspects of computer usage, Weiser envisioned a near-future of devices “that fit the human environment, instead of forcing humans to enter theirs” (104) with concerns also on how information is accessed, presented and manipulated. One of his remarks is particularly poignant:

“There is more information available at our fingertips during a walk in the woods than in any computer system, yet people find a walk among trees relaxing and computers frustrating.” (Weiser 1991:104)

Bill Moggridge compares the boardgame Monopoly with a video cassette recorder (VCR): both exhibit an equitable level of complexity, yet as “visual description of a system” Monopoly does a much better job than the VCR which (during its heyday) was “the most frequently mentioned scarecrow of interaction design difficulty” (Moggridge 2007:542). Indeed, boardgames can inform interaction design, as a game’s formal rules “cannot work independently from objects, ideas, texts, sounds and images” in constructing the game’s rhetoric and experience (Frasca 2007:87). Among other things boardgames (or games using physical tokens) are appealing to a broad audience because their “forms of representation are often related to the materiality of the game itself” (Salen & Zimmerman 2004:64). The Nintendo Wii game console has been an excellent example of this (see 5.3.1).

Following the vision set out by Weiser, researchers at the MIT’s Tangible Media Group were among the first to explore the coupling of physical forms and computation within a research context, as the means to “rejoin the richness of the physical world in HCI” (Ishii & Ullmer 1997:1). The group is led by Hiroshi Ishii, whose greatest inspiration was “the aesthetics and rich affordances of [...] historical scientific instruments” developed as “specialized physical artifacts to measure the passage of time, to predict

EMBODIED INTERACTION | 49 the movement of planets, to draw geometric shapes, and to compute” (ibid.). One particular recurring example in Ishii’s rhetoric are childhood memories associated with the abacus, effectively “the simplest form of digital computation device”, free from the “big divide between the pixel representation and the controllers” introduced by the GUI (Ishii in Moggridge 2007:529). Through the playfulness of a child the object’s affordances (a term which we will get to soon) were explored, and the abacus naturally reinterpreted as “a musical instrument, an imaginary toy train, or a backscratcher” (ibid.). Ishii also frames the abacus as ambient media. “When my mother was busy doing the accounting in our small apartment in Tokyo,” he recounts, “I could hear the music the abacus made, which told me that I couldn't interrupt her to ask her to play with me” (ibid.).

The disconnection between the way we interact with electronic artefacts and the way we interact with mostly everything else around us is a concern shared among others by Weiser, Ishii & Ullmer, and Dourish (1991, 1997, 2004). French philosopher and cultural theorist Jean Baudrillard also criticizes “how far the mediation of gestures between man and things has been stretched” as “prehension of objects involving the whole body has given way to simple contact [...] and simple surveillance” (Baudrillard 2005:50-51). While technology and media are not utterly immaterial or virtual, their materiality manifesting in everyday experiences seems to be, at best, tolerated (Lister et al. 2008:242). Under a “rhetoric of de-materialization” (Munster 2006:17) foreseeing “the triumph of the virtual over the physical” (Dourish 2004:44), the preference has long fallen on virtual, immaterial aspects of digital media, “marching relentlessly toward a condition where everything that can be digital will be digital (Moggridge 2007:661)”. Physical objects become mere “clutter” (Bell & Gemmel 2009:28) whose es- sence can be distilled and decanted into a virtual world (Dourish 2004:44).

In their design, tangible interfaces “rely on the natural structure of the everyday world and our casual familiarity with it” by embracing qualities related to materiality (Dour- ish 2004:41). Going beyond the traditional metaphorical approaches used in the now- ubiquitous GUI, they draw more heavily on the realization that “we experience the world [...] through directly interacting with it, and [...] we act in the world by exploring the opportunities for action that it provides to us” both “through its physical configuration” and “through socially constructed meanings” (ibid.).

50 | ART AND TECHNOLOGY AT PLAY These “opportunities for action” are approached through J. J. Gibson’s notion of affordance (Dourish 2004:117). Gibson’s discourse is situated in a broader topic of ecological psychology (a branch to which he himself greatly contributed), which attempts to understand the workings of the mind within a natural and evolutionary context “involving the organism, action and the environment” (ibid.). The concept of affordance was later applied within the context of HCI by Donald Norman, as insight into improving the usability of user interfaces:

“[...] the term affordance refers to the perceived and actual properties of the thing, primarily those fundamental properties that determine just how the thing could possibly be used [...] Plates are for pushing. Knobs are for turning. Slots are for inserting things into. Balls are for throwing or bouncing. When affordances are taken advantage of, the user knows what to do just by looking; no picture, label, or instruction is required.” (Norman 2002:9, original emphasis)

Such “opportunities for action” are offered to “appropriately equipped organisms” (Dourish 2004:117) that is, an organism that is both able to understand and activate them. A human and a chimpanzee may perceive a thin stick as affording being held and inserted into an aperture – effectively as a tool for termite-fishing – whereas a cat may not. This is, of course, very much in line with embodied interaction’s focus on the connection between action and understanding; as well as ecological, environmental and evolutionary approaches to psychology where the mind is studied in regards to its natural context, in the way it focuses on information “specifically relevant to human problems” (Boyd 2010:39).

As highly adaptable tool users and makers, humans “excel at a deeply attentive mental and physical engagement with artifacts” (Sterling 2005:56) having developed specialized mental faculties for dealing with objects (Pinker 1999:191). The mind treats inan- imate objects within a category of their own (Pinker 1999:191, 2003:219-220), evidence of which can be found in certain dysfunctions where patients cannot name natural objects but they can name artefacts, or vice-versa (2003:327). Tool use and production are found evolutionarily connected to the development of the brain and hand (and possibly language), as the hand is “used to obtain information that could be obtained only by acting upon the object being held” (F. R. Wilson 1999:276).

EMBODIED INTERACTION | 51 More than properties of a single object, affordances should also be considered as combinatorial opportunities offered within an environment. As such, affordances aren’t strictly mapped one-to-one to physical characteristics of an object, but can be combinations of physical and social aspects of different actors and objects, who “join forces in offering common properties that act together” within a given context (Olsen 2010:146).

This perception of artefacts as avenues for action is also resonant with Heidegger’s concept of “Zeug” (equipment) and the chain of “in-order-to-relations” within which it effectively exists (Olsen 2010:69). Dourish also makes use of Heidegger’s concepts of “zuhanden” and “vorhanden” as modes of engagement which “are critical to effective use of technologies”, something which holds equally “for abstract representations as for physical objects” (Dourish 2004:139). Developing and using a computational tool “involves a continual process of separation and reengagement with a world of entities and artifacts, physical and virtual, each of which carries different meanings and plays different roles in the multiple, overlapping contexts in which it appears” (144). Users should be able to switch between ready-to-hand and present-at-hand modes, i.e., they should be able to use the tool intuitively with an ulterior motive but also to occasionally address the tool directly and reorient its use (139).

Digital information is typically encoded in binary form, as an abstraction of the real- world counterpart which it represents. Data physically exists as electrical charge, but it is used in representing something other than itself (Dourish 2004:81-82). For most purposes, users will not be interacting with binary data (or electromagnetic charges) but with “a set of abstract computational representations [assembled] into a tool” which is then used “to achieve some end result” (144). Instead of propagating the inner logic of computation onto the interface, abstraction also allows us to design artefacts assembled into systems with less prescriptive use, where users have a greater freedom to “manage coupling” and “create and communicate meaning” which should not be the responsibility of the designer alone (166, 170).

As tangible interfaces use the world as a medium instead of as a metaphor (101) they may appear in a great variety of forms and configurations. While it does not merely denote materiality (such as in a computer mouse or keyboard), the term tangible interface is – in the context of this thesis, at least – quite inclusive. This can mean exist-

52 | ART AND TECHNOLOGY AT PLAY ing objects being augmented with digital capabilities, able to sense some of their context, such as location and proximity to other devices (Dourish 2004, Greenfield 2006). It can also mean the design of entirely new artefacts, whose use resembles that of more traditional objects (occasionally with seemingly magical or mythical aspects) facilitating our perception of, and control over, aspects of the otherwise hidden digital world (Dourish 2004, Moggridge 2007, Montola et al. 2009). Among the myriad forms that tangible interfaces can take, are those of clothing and apparel, including footwear, bracelets, necklaces and even wigs.

3.2 · Wearable Interfaces In a world where ubiquitous computing is not only a topic of research but increasingly an everyday reality, interaction with digital information is no longer confined to a stationary desktop setting, prompting us to reconsider the myriad ways in which computers can pervade our lives (Dourish 2004, Greenfield 2006, Sterling 2005). One such way is by being embedded into our clothing or apparel, becoming wearable; and as such somewhat more personal and intimate (Gemperle et al. 1998, Seymour 2008, 2010).

As a field which emerged in the 1980s, different definitions of “wearable computer” may focus on different aspects of what a computer is, or what being worn implies (Viseu 20-22); and even consider the technological amplification of aspects of fashion and communication (Seymour 2008:11). A succinct definition is provided by Barfield and Caudell, characterizing wearable computer as a “fully functional, self-powered, self-contained computer that is worn on the body [providing] access to information, and interaction with information, anywhere and at anytime” (Barfield & Caudell quoted in Viseu 2005:20). This definition highlights the role of wearable computers as interfaces, an aspect of particular concern to the present text. Being worn opens up possibilities and challenges on how a computer or system may function as the means to an end, especially on how it interacts with the user/wearer; and how it may empower and extend an individual’s capabilities (ibid.). The terms “wearable computer”, “wearable interface” and simply “wearable” will be used interchangeably within this text.

Early wearable computers were certainly clunky and impractical. The best know ex-

EMBODIED INTERACTION | 53 amples thereof may be Steve Mann’s constructs in the late 1980s and early 1990s (Mann 1994). At the time, research in wearable computers was less focused in mimicking actual clothing and apparel and more focused in supporting mobile augmented reality, or “wearable-tetherless computer-mediated reality” in Mann’s terms (ibid.). This entailed wearing heavy equipment (a 90s computer, head-mounted display, sensors and batteries) whereas nowadays a better experience can be provided with much smaller commercially available smartphones, VR goggles or smart-glasses. Re- search into wearable computers has since then focussed on four main areas: health monitoring, augmentation of physical abilities, improving efficiency and productivity, and lifestyle applications (Viseu 2005:13). With the increased involvement of artistic and design approaches on the topic, improvement of technical means and eventually the availability in the consumer market of wearable gadgets, wearables are nowadays overall more diverse, comfortable and stylish (Seymour 2008:13).

Given their proximity to the wearer’s body, the overall rhetoric behind wearables promotes the augmentation of human sensory and cognitive abilities (Viseu 2005:11), including the ability to gather and analyse one’s own data to uncover otherwise invisible behaviours and causalities (Bell & Gemmel 209, Thackara 2005:202, Viseu 2005:16). Other common scenarios include “sport, work wear, healthcare and rehabilitation, res- cue services, elderly care, and security” (Seymour 2010:16).

The design of wearable devices and interactive garments must take into body ergo- nomics, perception, functionality, technology, materials, energy and environmental impact into account (Seymour 2008:27). One of the most obvious issues is the physical adequacy of the device, as it is worn on and supported by a human body (Gemperle et al. 1998). This implies care in the choice of forms, placement, weight, size (and size variations); as well as how these respect and respond to the dynamics of a body in motion (ibid.).

Whereas wearable interfaces were initially “considered tools designed to give wearers instantaneous and constant access to information” there has been a considerable shift in their conceptualization towards making them more proactive, responsive, networked and context-aware (Viseu 2005:22). Similarly to what is achieved through the personalization of other computational devices (Fujimoto 2006:87), wearables are then perceived not as mere tools but rather as extensions of the self, technological

54 | ART AND TECHNOLOGY AT PLAY companions, or even as a second skin (Viseu 2005:22).

A wearable interface may acquire the same connotation of intimacy as any other garment (Barnard 2002:117). Moreover, wearables located within a perceptual “aura around the body” (with distances up to 13 centimetres away from the body, depending on the exact placement) are effectively “within the wearer’s intimate space, so that perceptually they become a part of the body” (Gemperle et al. 1998:118). The process of incorporating a tool in usage is, as we have seen, of importance in the design of embodied interactions and tangible interfaces (Dourish 2004:139). Objects can be perceived as elaborations or extensions of the body, as in Merleau-Ponty’s analogy of the blind man’s stick (Olsen 2010:130); and their presence and availability characterized as a kind of closeness which, in Heidegger’s discourse, is not only a physical distance but also a sense of intimacy, “a mode of becoming familiar with something” (75). Objects in general may be employed as “territory machines” - incorporated into what we naturally perceive as our private space in order to maintain or extend it (Fujimoto 2006).

In the effort of being accepted as mainstream items, what were once bizarre and aesthetically uninformed research prototypes have come to more closely resemble clothing and other existing worn items (Viseu 2005:28-29). Compounding on the proprio- ceptive incorporation of clothing and accessories as part of one’s body, the design of a wearable interface must take into account the many personal, communicative and expressive aspects of clothing and garment (Barnard 2002), especially if expected to be worn side-by-side with existing apparel in everyday situations.

Technology can be used to improve the aesthetic and expressive qualities of an object, compounding on its efficacy as a tool (Gell 1992). Tools and highly technical objects can also be personalized and adopted into existing practices and/or as cultural symbols (Gell 1992, Miller 2009). One relatively recent example is the personalization of pagers and mobile phones by Japanese teenagers, who reinforced existing social practices and developed both private and public ways of expression around mass-produced communication tools originally targeted at businessmen (Kusahara 2010:5, Okada 2006:41-60, see 2.2.2).

Designer and researcher Sabine Seymour identifies the intersection of design, fashion, science, and technology as fertile ground for technically enhanced forms of clothing combining “aesthetics and style with functional technology”, which she refers to as

EMBODIED INTERACTION | 55 “fashionable technology” (Seymour 2008:11). Beyond the beautification of wearable computers, this entails a ground-up holistic approach to “merging a fashionable technology object deemed aesthetically pleasant with technically enhanced functionalities” where the garment’s inherent expressive role is technically empowered, that is, taken advantage of rather than merely acknowledged (Seymour 2010:10).

A majority of commercially-available wearables integrate digital technology into a wearable form as an interface oriented towards the wearer (e.g., to track the wearer’s activity or notify him/her about events). Yet technology can also be used with the main purpose of amplifying the expressive value of a garment, expanding on “traditional fashion elements such as color, texture, and cut to include movement, touch, light, sound, and interactivity” (54); creating new “functional aesthetics” and modes of interaction (10).

The design of wearables which are meant to coexist with more common apparel must also take into account (or even reinforce) other functions of clothing besides communication and expression. Malcolm Barnard identifies material and cultural functions of clothing (Barnard 2002): the former include protection, concealment and attraction; the latter, communication, individualistic expression, socioeconomic status, social role, political or religious orientation, as well as ritual and recreation (49-70). He ad- verts, however, that the separation between “material” and “cultural” is made “for the sake of argument” alone, since the material functions of clothing also “have a cultural function and serve to construct and communicate cultural identity” (49). Anthropolo- gist Daniel Miller contrasts the material characteristics of clothing and jewellery, and how these impact the object’s role in mediating a process of mourning:

“The most suitable and successful instrument of bereavement has been clothing, the most difficult has been jewellery [...] Jewellery does not seem to be amenable to the same sense of gradual incorporation as clothing. Partly this may be on account of its very materiality. Clothing is not forever, it changes and fades and can take on something of the corporality of its wearer. It has features which are gentle and encouraging of this gradual as- similation of one person into another. By contrast, jewellery stands immut- able, in a more abstract relation to the person. The point of gold and dia- monds is that they don't change or fade, they resist humanity. This is rein-

56 | ART AND TECHNOLOGY AT PLAY forced by their monetary value which, again, pitches them back into a more abstract form of value, so that they tend to be a colder, sharper instrument of transmission than clothing.” (Miller 2009:42-43)

It is also important to highlight that the protective role of clothing isn’t limited to strictly physical phenomena but can also address psychological concerns such as the “unfriendliness of the world” as well as “magical and spiritual agencies” (Barnard 2002:51). Personalization “is an essential factor in making fashion items that appeal to the public” (Seymour 2008:13) and can amplify the role of psychological protection, as in the case of late 90s Japanese teenagers for whom the mobile phone was

“more than just a tool – it is something they are highly motivated to animate and to customize as a dreamcatcher, a good luck charm, an alter ego, or a pet. I might even call it an idol or a fetish and regard it as an animistic handy object [...] stretching a spiritual barricade [...] around the body” (Fujimoto 2006:87).

Similarly to what has happened with mobile phones in Japan, wearable interfaces have great potential to excel as personal charms. If the rhetoric behind wearable computers is one of empowerment and augmentation (Viseu 2005:11) then wearable interfaces can effectively resemble magical artefacts which populate our folklore, granting the wearer special abilities (see 2.5.1).

Folklore and science-fiction are a source of inspiration in the age of ubiquitous computation (Dunne & Raby 2013:3, Greenfield 2006:119), and the use of wearable interfaces in a recreational context helps to “create the illusion that the player has superhuman abilities” (Montola et al. 168). In addressing ritualistic and recreational roles of clothing, the possibilities afforded by digital technologies allow us to design wearables as “amplifiers of fantasy” (Seymour 2008:12). Embracing spirituality and fantasy is also a way of facilitating the adoption of new technologies within traditional and even practices, including religious rituals (Greenfield 2006:22, Hlubinka et al. 2002).

As the interplay between function, material and expressive qualities of clothing are taken into account, wearable interfaces become “fashionable technology” where technical enhancement reinforces the existing roles or possibilitates new ones (Seymour 2008:13). In the way their design integrates “issues of human form and human-com-

EMBODIED INTERACTION | 57 puter interaction with the constraints of technology and the context-of-use” (Gem- perle et al. 1998:1), irrevocably enmeshing existing artefacts, material and cultural functions, meaning, intimacy and the human body, wearable interfaces can support exceptionally expressive experiences of embodied interaction.

3.3 · Objects as Media Embodied interaction, as we have seen, concerns itself with the design of technology for a physical and social world which “is made meaningful through the possibility for actions it affords” (Dourish 2004:116). More than mere materiality, Dourish’s usage of the term embodiment also denotes “a form of participative status” (Dourish 2004:18) effectively supported by the intentionality and accountability of actors and actions.

Technology and technical systems take part in social and cultural processes and are also, in turn, shaped by them (Dourish 2004, R. Williams 1990, Williams & Edge 1996). Technology can be seen to consist of "a rearrangement of existing parts, some from nature, some from culture, some from existing technologies" united by forces "divert- ing the course of causes and effects into new arrangements" (Lister et al. 2008:405).

As result, an “artifact is not simply a tool for a job," able solely to "afford certain sorts of actions [...] it also reflects particular sets of assumptions, conventions, and practices within a community” (Dourish 2004:186) and can even be a mediator of social agency (Gell 1998). As we have seen, tangible and especially wearable interfaces may thrive on being designed as tools for expression, as technically-augmented, intimate and personalized media.

3.3.1 · Meaning Within a post-industrialist consumer society we find ourselves surrounded by fabric- ated things, bearing layers of meaning – and many, traces of ideology (Baudrillard 2005). Objects populate our homes and our lives, and communicate with us “openly and actively or in subtle, subliminal ways” (Antonelli 2011:6); some “in text, diagrams, and other graphic interfaces [...] others empathetically and almost telepathically, just keeping us company and storing our memories” (ibid.).

Although tangible interfaces, “smart devices” or other new applications of digital technologies are ever more designed to do so (Greenfield 2006, Moggridge 2007, Sterling

58 | ART AND TECHNOLOGY AT PLAY 2005), the ability to “talk to us” is of course not exclusive to these. As we will see, tools, artefacts, decoration, identity, communication, expression and social practices have been connected, as indissociable components of our life since the dawn of humankind.

While it may be a fact that “humans and their techniques, needs and objects are structurally interlocked come what may” (Baudrillard 2005:134), technology and culture are usually portrayed as somehow opposite to nature – an opposition which is continually maintained by “modernity and science” (Olsen 2010:102). It has however been argued that “culture is not a domain separable from nature” and that “technology can no more be separated from nature [...] than from culture” (Lister et al. 2008:404).

3.3.2 · Cognition Evolution has itself evolved in many ways, yielding sub-processes of variation and selective retention operating in smaller time-scales than that of genetic mutation and sexual reproduction (Boyd 2010:120). Due to their shorter turnaround of results, these Darwinian processes of variation allow for higher chances of survival in highly unpredictable environments (ibid.). Among examples of these we may find art, culture, invention and technical development, the ways in which “individuals or groups construct new artifacts or ideas and, from the most promising platforms, build still higher” (381). This very ability has played an important part in our success as a species (Pinker 2003:236).

In order to manufacture even a primitive axe our ancestors needed to possess amazing inventiveness and dexterity, but also skills of communication and cooperation (Boyd 2010, Tomasello 2009, F. R. Wilson 1999), a “mimetic and conceptual ability [...] unique to hominids and Homo sapiens” (McConachie 2011:38).

In competition with the environment but also against other equally skilled groups, the evolutionary pressure put upon our ancestors led them “to expand their cognitive, affective, and social capacities to enable them to cooperate” (42) throughout an era of tool-making, fire-using, language, storytelling and primitive forms of performance (Boyd 2010, McConachie 2011). The emergence of human intelligence was fostered through interactions taking place in groups, especially the collaborative production of tools, with participants assuming heterogeneous roles to produce and assemble dif-

EMBODIED INTERACTION | 59 ferent parts of an artefact (F. R. Wilson 1999:170-171). It is therefore hardly a coincid- ence “that the human mind grasps the world – even the most abstract, ethereal concepts – as a space filled with movable things and stuff” (Pinker 1999:191).

Primates’ brains split visual information into two streams: one for the shapes and compositions of objects and one for their locations and motions (Pinker 1999:191). At around three to four months of life, human infants are able to perceive and remem- ber objects; and “expect them to obey the laws of continuity, cohesion, and contact as they move” (319). This forms the basis for a “mental module” of intuitive physics around perceived core properties of an object – that it “occupies one place, exists for a continuous span of time, and follows laws of motion and force” - which also aids in spatial perception and navigation (220). Studying patients with neurological deficits has also indicated that our brain is internally structured to make ontological distinctions between humans, animals, plants and artefacts (Boyd 2010:136, Pinker 1999:327).

The human brain is especially prepared to recognize and assess objects, assuming a “design stance” to apprehend an artefact’s basic physical properties and hypothesize its purpose (Pinker 1999:327-328). When confronted with an object, we are led to play out its origin-story, wondering about the identity and intention “of the agent who made or originated it” (Gell 1998:23) and “reconstructing a sequence of actions” that led to the object’s current state and configuration (4). Dunne and Raby take advantage of this instinct, as their critical design work consists largely of speculative objects intended to induce a whole fictional scenario in the audience’s mind, “a form of window shopping” upon an object in which the public is led to “wonder what kind of society must have produced it, how it was structured, what values, beliefs, and dreams motivated it” (Dunne & Raby 2013:140).

This sort of hypothetical thinking involves actions and possibilities thereof, as areas of the brain related to motor functions are especially active when we think about artefacts (Boyd 2010:156). For our minds, at least, artefacts are a “mixture of mechanics and psychology” i.e., they are not only “defined by their shape or their constitution” but more importantly by “what they can do and by what someone, somewhere, wants them to do” (Pinker 1999:327-328). This clearly resonates with Heidegger’s understanding of our relationship with “equipment” in terms of “proximity” or availability

60 | ART AND TECHNOLOGY AT PLAY (Olsen 2010:75); and with Gibson’s notion of affordance (Dourish 2004:117), later applied to HCI by Donald Norman and introduced above (Norman 2002:9).

As with other objects and tools, a computational artefact is better understood the more we are “able to reconstruct the designer's intentions regarding its use” (Such- man 2007:43). This is the very reason why “it is hard to see the difference between a radio and calculator by their design” as consumer electronics “are almost universally black boxes of a standard size,” a design which “does little to describe what they are actually doing” (Moggridge 2007:541).

3.3.3 · Materiality and Agency The modern (and post-modern) object has been characterized as part of an “illusory and deceptive world” (Olsen 2010:11), a semiotic system of myth wherein the object’s original function is downplayed in favour of it becoming an almost arbitrary sign (Barthes 2000, Baudrillard 2005) with “minimal function and maximal meaning” (Baudrillard 2005:85). Objects in material culture have extensively been “approached as carrying a final signified” (Olsen 2010:47) under an “assumption that the meaning or social significance of things primarily derives from outside” (145) and consequently with “disregard for the intrinsic material significance of things” (3). Yet objects serve as more than mere motivated signs or empty signifiers. Their material existence, constitution and qualities possibilitate social entrepreneurship, and even their usage as indicator of social status would not be possible in the absence of intrinsic differences:

“A house, a mountain, a bridge, or an oil reserve all have intrinsic qualities that seriously restrict their exchangeability. A kayak or an ax does have a competence that cannot be replaced by just any other signifier [...] Moreover, contrary to the linguistic sign, the reality of these entities is experienced directly, through themselves; they come to us – also – in an un- mediated way [...] What we are dealing with are not ‘empty signifiers’ but real entities possessing their own unique qualities and competences” (156).

The very durability of things can be seen as their primary "existential affordance" – a trusted “in-place-ness” which provides security and predictability (160). This characteristic, commonly taken for granted, also allows for “the gathering or sedimentation of the past” (ibid.) with the object constituting a materialized “palimpsest” of all past

EMBODIED INTERACTION | 61 events (108). As such, objects lend themselves to the role of mnemonic media, allowing “memory and meaning to be recorded and codified for later recollection” (109) for instance through their use in performances and rituals (124).

If we define media as technologies which make possible certain forms of expression (Lister et al. 2008:404), “as means of communication, expression, representation or imaginative projection” (88) then objects can certainly be regarded as media in different circumstances – for example, within the context of a traditional tea ceremony and alongside other media, objects and social practices (Fujimoto 2006:90, Kusahara 2006:5).

A resistance endures against ascribing any form of agency to entities other than the human subject – something which is also a legacy of a long-lived cartesian-dualist mentality (Olsen 2010:11, 98). Ascribing anything beyond usefulness and beauty to an artefact is seen as primitive and fetishistic; and emotional investment in an object as a replacement of “real” social relationships (Baudrillard 2005:87, Olsen 2010:94). This has led to the modern “intellectual dismissal and silencing” of a material world of artefacts (Olsen 2010:98) which form part of our evolutionary and cultural history; and which we are cognitively predisposed to engage with.

Unrestricted from defining social agency in terms of biological attributes, anthropologist Alfred Gell explores the role of artefacts as social agents in his last (posthumously published) work (Gell 1998). The terminology of “artwork” or “art object” is used interchangeably with “artefact” to refer to this category of objects which admittedly includes (among others) idols, tattoos, decorated weapons, girls dolls and artworks like Michelangelo’s David (18). This is not meant as a devaluing of a fine artwork, but rather a revaluing of a little girls' doll as social agent, placing an emphasis on the mediatory role:

“In place of symbolic communication, I place all the emphasis on agency, intention, causation, result and transformation. I view art as a system of action, intended to change the world rather than encode symbolic propositions about it. The 'action'-centred approach to art is inherently more anthropological than the alternative semiotic approach because it is preoccu- pied with the practical mediatory role of art objects in the social process, rather than with the interpretation of objects 'as if' they were texts.” (6)

62 | ART AND TECHNOLOGY AT PLAY Gell’s isn’t the only perspective which treats human and non-human equally as agents (to a degree, at least). Together with other scholars, sociologists John Law and Bruno Latour developed actor-network-theory (ANT) as a material-semiotic method, effacing "the analytical divisions between agency and structure" and treating different "materials" (people, machines, even ideas) "as interactional effects rather than primitive causes" (Law 1992:389):

“analytically, what counts as a person is an effect generated by a network of heterogeneous, interacting, materials. [...] If you took away my computer, my colleagues, my office, my books, my desk, my telephone I wouldn't be a sociologist writing papers, delivering lectures, and producing ‘knowledge’. [...] Is an agent an agent primarily because he or she inhabits a body that carries knowledges, skills, values, and all the rest? Or is an agent an agent because he or she inhabits a set of elements (including, of course, a body) that stretches out into the network of materials, somatic and otherwise, that surrounds each body?” (384)

While ANT constitutes a broader analytical framework, Gell focuses on social agency, more specifically the ability of an artefact to act as an index (in Peircean semiotic terms) of another being’s agency – whether a single human, a collective or a divine entity (Gell 1992:13). He highlights how objects may serve as a social media between beings, providing a channel for the expansion of social relations and influences between people (52). The multiplicity of relations it sustains is intertwined with the object’s story and materiality; the “thing in its concrete, factual, presence” is perceived as the “knot which ties together an invisible skein of relations, farming out into social space and social time” (62). Objects such as weapons, canoes and furnishings may represent and extend an individual or collective, expanding their social influence (168). Their basic function as tools is often tied to their role as social agents – for instance the Kula canoes, in their function as a means of transportation and trade, excel as far-reaching fields of influence (229). Such objects constitute an “extension of personhood beyond the confines of biological life via indexes distributed in the milieu”, interacting through cultural institutions and practices (104, 223).

Anthropology has provided plenty of evidence of societies where the concept of mind and soul is not exclusive to the human body (Fowler 2004:17), and as such do not

EMBODIED INTERACTION | 63 readily separate individuals and things, spiritually and socially (Olsen 2010:99). Even “though they do not necessarily become a part of the body” objects may be for all purposes perceived and treated as “a part of that person [...] distributed throughout the social world” via gift-giving and exchange (Fowler 2004:17). Items gifted or exchanged through rituals foster continued relationships, in being “inalienable from people, and from the relations between them” as opposed to most commodities which are paid for, in which the buyer “has effectively bought out of any personal relationship with the vendor” (34).

The western formulation of mind (or soul) as separate from, yet exclusive to the human body has obscured the fact that the notion of distributed personhood is not constrained to the context of “primitive” societies (17). It is found very alive in our daily affairs with people and things, in “the artisanal form of love, care and devotion, performed with such subtle grace, creativity and imagination that the ways persons become objects of care and objects become subjects of relationships blend impercept- ibly with each other” (Miller 2009:31).

3.4 · Strategic Values Embodied interaction is used throughout as a strategy for usability. All of the works consist of, or are enabled through interactions between participants and computational systems. Providing a good user experience is not only important in the design of interfaces (such as the Gauntlet) but even more so in the effectiveness of complete interactive works (unless the intention is to induce puzzlement or frustration). In all of the works, usability owes much to the design of tangible interfaces and – in some of the most successful cases – wearables. They provide interactions that are natural (Gauntlet, Lorm Hand), intuitive and spontaneous (Headbang Hero, Rambler) or at least familiar and relatable (Noon, Weltschmerz, Wcielenie, Tripo). One aspect that is missing is the possibility of controlling coupling during use, in Dourish’s terms (2004:47). One arguable exception might be Noon, where the clock affects all the other objects.

The expressive qualities of works are also bolstered through embodied interaction. Fa- miliar objects and places are approached as interactive media, with the ability to convey stories (Noon, Wcielenie, Tripo), enable alternate forms of communication and ex-

64 | ART AND TECHNOLOGY AT PLAY pression (Rambler, Lorm Hand) or question ideologies behind existing media (Ram- bler, Lorm Hand Weltschmerz). A work’s message and rhetoric can also benefit from appropriation of the meanings and uses of an object (Weltschmerz).

The participatory nature of the works results in a limited experience to observers. The choice of familiar objects and gestures, however, does not only benefit the usability of the interface, but also improves the accountability of a participant involved in a computational process. As such, some of the work’s expression is extended to observers through the participant’s performance, and may even elicit empathy (Headbang Hero, Weltschmerz, Lorm Hand).

EMBODIED INTERACTION | 65

4. PLAYFULNESS

Many decades after Johan Huzinga's seminal attempt at a study of play in culture (Huizinga 1971), a universally-accepted formal definition of play remains elusive (Salen & Zimmerman 2004:302). As a concept which is covered by a single word in several languages (English, German and French included) it encompasses structurally and phenomenologically diversified activities such as playing with toys, games and instruments, practising sports like tennis or football – and even “modding” a videogame or dressing up and acting like one’s favourite fictional character (Hjorth 2011:54).

In his work Huizinga highlighted aspects of play ingrained in cultural-constructive practices such as religion, art, law, and even philosophical rhetoric (Huizinga 1971). Play is among the “original wellsprings of culture” and as such the latter is inherently playful: the “ludic element is pervasive and fundamental” (Rodriguez 2006, para. 26).

4.1 · Psychology and the Magic Circle While not everyone may agree on exactly what play is, it can be said that “everyone knows what play is not” (Gilmore 1971:311). Other animals are able to distinguish actual fighting from play-fighting – most of the time, at least (Salen & Zimmerman 2004:450). As such, it may be tempting to define play by contrasting it with “seriousness” or “work”.

Serious activities can be found to contain ludic features (Huizinga 1971, Rodriguez 2006). As players become engrossed in the activity, the contrast between play and seriousness can be quite fluid (Huizinga 1971:5-8), even if some cases are portrayed as deviations from a “true spirit” of play (Caillois 2001:46, Huizinga 1971:197). Therefore, it is not the case that play and seriousness are always mutually exclusive.

Huizinga also postulates that although play is a necessity, it serves no immediate material interests and yields no material profit; it is a superfluous and free activity carried out at “leisure time” (8-13). Writing shortly after Huizinga, Roger Caillois agrees with the point that play is “unproductive” in the material sense, that it creates “neither goods, nor wealth, nor new elements of any kind [...] ending in a situation identical to that prevailing at the beginning of the game” (Caillois 2001:10). However games (and

PLAYFULNESS | 67 other structured play activities) commonly have some observable practical outcome, including prizes and social prestige (Frasca 2007:70). Furthermore, as we will see, playful behaviour leads to learning and discovery, yielding actual benefits beyond the immediate pleasure of the activity itself.

Improving upon the attempted distinction from other activities in everyday life, Huizinga defined play more concretely as a voluntary “stepping out” of ordinary life, taking place within a contractual, conceptual framework which limits play activities in time, space and imbues them with special meanings (Huizinga 1971:28). While giving examples of play-grounds – spaces where play occurs – he writes:

“The arena, the card-table, the magic circle, the temple, the stage, the screen, the tennis court, the court of justice, etc, are all in form and function play- grounds [...] within which special rules obtain. All are temporary worlds within the ordinary world, dedicated to the performance of an act apart” (10, added emphasis).

We may find that also Roger Caillois, writing after Huizinga, supported this same idea: that play is “circumscribed within limits of space and time, defined and fixed in advance” (Caillois 2001:9).

The term magic circle was later appropriated by play theorists and ludologists to refer to the conceptual framework which separates play from what Huizinga terms “ordinary life” (Salen & Zimmerman 2004:94-98). The original premise has been re-evaluated and expanded, leading to a formulation of the magic circle as being permeable and multidimensional: its boundaries are not merely fixed spatio-temporal ones, but also social and semiological (Montola et al. 2009, Rodriguez 2006, Salen & Zimmerman 2004, Stenros 2012). The magic circle can thus be considered “a psychological and philosophical construct that delimits the peculiar space of play”, a conceptual “bound- ary that makes the paradoxical meanings of play possible” (Salen & Zimmerman 2004:450) wherein players’ actions and events are interpreted in relation to the specific semiotic domain of the activity (Montola et al. 2009:16).

According to psychologist Michael J. Apter, it is impossible to correctly define play by relating solely to specific behaviours, observed externally (Apter1991: 13-14). Building upon Apter’s observations, game designer and ludologist Gonzalo Frasca concludes

68 | ART AND TECHNOLOGY AT PLAY that “the exact same activity can be framed as work or play, depending on the observer or the agent performing it” (Frasca 2007:51). While much of the academic focus has been on analysing play (and especially games) as rules, goals and structures, play does not occur in the absence of a player – it implies intention actualized as performance (Ermi & Mäyrä 2005, Frasca 2007).

A psychological approach leads to characterizing play as “an attitude or frame that can be adopted toward anything” (Sutton-Smith 2001:32), which helps to understand the diversity of play forms and the difficulty in achieving an accurate yet universal definition. Indeed, play is observable in many forms in other species, “widespread across animal classes and perhaps universal in mammals” (Boyd 2010:14), with closely related behaviours observed in birds, reptiles and octopi (91). While a formal definition remains a challenge, the notion of play is so ingrained in our ancestry and behaviour that we seem to have little trouble recognizing it (179). As we will eventually discuss, play serves an evolutionary role.

4.2 · Creativity Play is frequently seen as associated with learning, creativity and discovery. Scholar and educator Neville V. Scarfe saw it as a natural form of research, “the most complete educational process of the mind” (Scarfe 1962:117). Manifest as “a spontaneous, creative, desired research activity carried out for its own sake” (ibid.) play is “often independent of external needs” and leads, among other things, to “perfection of form” and artistic expression (119). Much as Huizinga perceived an aspect of freedom in the act of playing (Huizinga 1971:8) so do others frame it as an activity where alternate rules and possibilities are safely explored (Schrage 2013:25), “a safe way to raise unexpected and/or unanticipated risks” (26). In the emergent, unpredictable way it proceeds, play can “overflow and overwhelm the more rigid structure in which it is taking place” (Salen & Zimmerman 2004:305) and give rise to new ideas and realities (Schrage 2013:25).

The impulse to play is crucial in the development of human motor, cognitive and social skills, not only during childhood but also throughout adult life (Boyd 2010, Scarfe 1962). Thanks in part to a lengthier dependency upon progenitors, humans imitate and practice more than any other species (Boyd 2010:104). A broadening and length-

PLAYFULNESS | 69 ening of social play leads to acquisition and improvement of complex social skills as well as the development of creative and aesthetic sensibilities (ibid.). The more spontaneous forms of play channel creativity, exuberance and engagement through exploration (Schrage 2013:25). As such, play can inform design methodologies in order “to uncover blind spots and expose weaknesses and limitations in an individual’s position- ing when conceiving and creating ideas” (Valentine 2013:10).

4.2.1 · Videogames and Hacking In the ways that they borrow from and in turn offer to pop culture and new media, videogames in particular can be said to constitute “a significant new medium” (Lister et al. 2008:298) ripe for artistic experimentation and appropriation (Morgana 2010). In observing the communities and practices that form around them, valuable lessons may be learned in designing creative, participatory and collaborative experiences in both digital and “traditional” media settings (Herz 2001, Hjorth 2011, McGonigal 2011).

Interestingly enough, the first videogame running on a programmable computer was the result of creative experimentation. In 1961 the MIT Lincoln Laboratory was granted the engineering prototype of the PDP-1 computer. The following year, seemingly without any practical scientific purpose, a group of graduate students created a program in which the user controlled a vectorial representation of a spaceship to shoot at representations of asteroids (Herz 2001, S. Williams 2002). Spacewar!, the very first programmable computer game, had been born as a hack: a playful experiment using computational technology, its source-code freely distributed (S. Williams 2002). Be- sides kickstarting digital games as interactive media, Spacewar! also positively influenced perspectives on “traditional” play and games (Fluegelman 1976:138).

The term “hacker” is commonly used in news media with a negative connotation, referring to an individual with malicious intent, e.g. someone who tampers with computer systems in order to violate another’s privacy, gather passwords or disrupt the functioning of a system or service (S. Williams 2002). However, the terms “hack” and “hacker” are part of a long tradition of programming jargon – a compilation of which has been kept since the 60s, starting precisely at the MIT (where Spacewar! was created), in what is called the Jargon File. The original file, which later served as the source for The Hacker’s Dictionary, defines “hacker” primarily as a “person who enjoys

70 | ART AND TECHNOLOGY AT PLAY learning the details of programming systems and how to stretch their capabilities, as opposed to most users who prefer to learn only the minimum necessary;” and also one “who programs enthusiastically, or who enjoys programming rather than just theorizing about programming” (“Original Hacker's” 1988).

A later version, which gave origin to The New Hacker’s Dictionary, defines “hack” in several ways, including as verb meaning to “interact with a computer in a playful and exploratory rather than goal-directed way” (“New Hacker’s” 2000). Another playful aspect of hacker culture is found in “grammatical creativity,” that is, the invention of hacker-specific jargon and “grammatical quirks” (ibid.). Like “play” and “art”, the term “hack” may be hard to precisely define – perhaps because the three manifest the spirit of experimental freedom. Qualifying as a hack, the game Spacewar! is also a technical and creative achievement in interactive media illustrating that:

“Playfulness and the playful use of technology suggest a positive interest in acts of continuous discovery. Playfulness can become in the near future a social and psychological benefit.” (Kaprow 2003:106).

What Kaprow seemed to miss, however, is that playfulness has driven innovation and social cohesion since before the dawn of humankind.

4.3 · Structured Play Given the variety and relative heterogeneity of play forms, a common and straightforward way to relate and distinguish them is according to the amount or rigidity of structure, such as that of formalized rules and clear objectives. Caillois proposes a continuum between the extremes of paidia, “frolicsome and impulsive exuberance”, at times “anarchic and capricious” in nature, akin to what we commonly visualize as child’s play (Caillois 2001:13); and ludus, where rules and form contribute to a “gratuit- ous difficulty” and, as result of their complex formulation, “reflect the moral and intellectual values of a culture, as well as contribute to their refinement and development” (27). This allows us to position free-form play activities (like making sand castles or climbing trees) relatively to those with more complex rules and goals, such as games. Consequentially, we can approach games as highly formalized and structured play- activities; and thus a subset of play forms (Salen & Zimmerman 2004:72). Building on this notion, game designers Katie Salen and Eric Zimmerman introduce conceptual

PLAYFULNESS | 71 boundaries to facilitate discussion, suggesting “successively more open and inclusive” categories which they term game play, ludic activities and being playful:

“As a category, ludic activities includes game play, and the category being playful includes both of the previous two. Game play is really just a special kind of formalized ludic activity. Similarly, ludic activities are formalized, lit- eral ways of being playful.” (303-304)

While this view of games as subset of play is theoretically more common when dis- coursing on play in general, the same authors also deem it beneficial to adopt a stance where play is viewed as one of the principal aspects of what makes a good game, besides others such as rules and culture (73). This is of course putting focus in the study and design of games as wholesome systems and activities in themselves. It can equally be helpful to view play as an “ingredient” in the conception of other contexts where play and playfulness are prevalent or desired, including participatory or interactive art (Kusahara 2010, Rodriguez 2006) and systems for collaborative learning and creation (Herz 2001, Hjorth 2011).

As they rely more heavily on rules, objectives and other rigid structures, games lend themselves better to formal study when compared to more paideic activities (in Cail- lois’ terms). Much can be revealed and understood by framing games as cybernetic, economic or information systems, for instance (Salen & Zimmerman 2004).

Part of their complexity and appeal lies on the fact that a game – in being both limited and defined through rules – constitutes an interactive space of possibility (66). Rules limit a participant’s choices and introduce artificial obstacles, inefficiencies or limitations as part of the challenge (77), effectively contributing towards delimiting the magic circle as spatio-temporal, social and semiotic separation from “normal life” (Stenros 2012).

While the rules of a game are binding, they allow enough variation so each play- through is essentially different and may potentially lead to situations which were not predicted by the designers or scripted in advance (Salen & Zimmerman 2004:159). Designing a game entails designing and refining relatively simple sets of rules and representations which can give rise to meaningful complexity – a potentially infinite space of possibility to be explored by players (168). This openness, allowing for the explorat-

72 | ART AND TECHNOLOGY AT PLAY ory and even exuberant qualities of play, can be contrasted with the workflow of other goal-directed interactive systems such as productivity software, with well-defined sets of choices and/or enforced pathways through sequences thereof (Daniels 2008:32).

Games constitute a cultural form (Lister et al. 2008:298), inevitably carrying within their rules and representations a culturally-charged rhetoric; as well as the potential to in turn affect aspects of culture and social life (Salen & Zimmerman 2004:512-517). This realization supports the increasing acknowledgement of games as a medium for artistic expression (Morgana 2010), as well as a potential art form in itself, combining aspects of other art forms in various ways (Humble 2006).

While some mourn the loss of freedom imposed by tighter structures and a focus on winning (Kaprow 2003:122) others view games “not so much a way to compare our abilities as a way to celebrate them” (Fluegelman 1976:10). Game designer and Jane McGonigal, in particular, has highlighted ways in which games can potentiate our ability to collaborate, both in gameplay proper as in ancillary activities such as creating and maintaining wikis and other player-created content (McGonigal 2011). Played together, games are essentially collaborative activities: even players who are competing against each other must at least cooperate in maintaining the game alive by playing according to the rules – what Salen and Zimmerman term “systemic cooperation” (Salen & Zimmerman 2004:256). As previously pointed out, the psychological skill-set required for this level of cooperation seems to be exclusive to humans (Tomasello 2009:70). According to McGonigal, however, the benefits of a good game can extend beyond improved collaboration – they can be highly motivating, providing “attractive alternatives to the boredom, anxiety, alienation, and meaninglessness we run up against so often in everyday life” (McGonigal 2011:114). In understanding “the real and completely renewable rewards we get from games” their power may be harnessed to provide a sustainable source of happiness (350).

4.4 · Evolutionary Fitness In the conclusion of his book on the different rhetorics associated with play, Sutton- Smith identifies "a facsimilization of the struggle for survival" (Sutton-Smith 2001:231), establishing a parallel between Stephen Jay Gould’s scientific discussion of processes in evolutionary adaptation and contemporary metaphors for play as a model of vari-

PLAYFULNESS | 73 ability (221). Observing that “there is an occasional transfer of play skills to everyday skills” (230) he proposes, as conclusion, that the biological function of play is to reinforce "the organism's variability in the face of rigidifications of successful adaptation," covering "the full range of behavior from the actual to the possible" (231).

As playfulness and associated behaviours can be observed not only in humans but also in other species, being especially prevalent in mammals (Boyd 2010:14), it is largely agreed upon that play must serve an adaptive role (179). Like other evolutionarily selected behaviours such as eating and mating, play is intrinsically pleasurable and compulsive (Boyd 2010:92). It can provide an incredibly resilient kind of happiness (McGonigal 2011:45), one which external rewards may even subvert rather than support (Tomasello 2009:9).

As mentioned before, playfulness provides an impulse for learning, especially during infancy but also throughout adult life (Scarfe 1962). In humans as in other animals, play serves as a mechanism of practice and refinement of both motor and cognitive skills:

“Through the compulsiveness of play, animals incrementally alter muscle tone and neural wiring, strengthen and increase the processing speed of synaptic pathways, and improve their capacity and potential for performance in later, less forgiving circumstances [...] All animals [...] so far studied grow more neural tissue in enriched than in impoverished environments, with more friends, toys, ladders, and wheels to play with.” (Boyd 2010:92)

To play repeatedly and exuberantly is to “refine skills, extend repertoires, and sharpen sensitivities” (Boyd 2010:15). Research findings strongly suggest that play deprivation during childhood can be at the root of social ineptitude and even sociopathic behaviour (179). Besides the development of innate motor and social skills, play also allows for the discovery of possibilities outside of one’s limited point of view, enabling their subsequent exploration (Valentine 2013:10).

Together with the joint evolution of the brain, hand usage, social concerns and improved cooperation, humans have also developed successively more intricate and complex forms of play and performance (Boyd 2010, McConachie 2011, F. R. Wilson 1999).

74 | ART AND TECHNOLOGY AT PLAY The capability to collaboratively establish, participate in and maintain a semiotic domain with loose ties to the natural or phenomenological world is, to the best of our knowledge, exclusive to humans (McConachie 2011:41, Tomasello 2009:70). The same abilities which allow us to devise, enjoy and participate in games, fiction and performance are also at the core of social organization and coordination: they enable social institutions and status functions, such as the “collective agreement that this piece of paper is money, or that that person is president, with all of the rights and obligations those agreements entail” (96).

So while “general play came first in evolution, the cognitive demands of playing games mean that this specific form of play has probably been around for only the last 50,000 years” (McConachie 2011:41)

Games and other complex, rule-bound forms of play are part of our evolutionary heritage, pervasive in the development of human society; to the extent that “simple conventional norms of how a game is played” are perceived even by small children “as supra individual identities that carry social force” beyond their instrumental role within the activity (Tomasello 2009:37).

The refinement of skills and discovery of new knowledge incurred by a playing individual are, of course, not genetically transmissible. However, play itself contributes to the transmission of knowledge through the propensity to perform and imitate (Boyd 2010:104); as well as through the development of sensibility and expression (Scarfe 1962:119) which in turn improves the breadth and depth of the knowledge that can be transmitted. In a species which inhabits the cognitive niche (Boyd 2010:14) thriving in the power of ideas (Pinker 2003:63), play is the evolutionarily developed drive for discovery, understanding and improvement.

4.5 · Play and Art As a natural drive for refining skills, improving expression and social cohesion, play constitutes a foundation for culture in its many aspects, including art (Huizinga 1971). In its more paideic forms at least, play encourages exuberance and leads “to increasing perfection of form, to more complete expressiveness” (Scarfe 1962:119).

It is hard to deny some form of connection between play and art. Schiller’s philosoph-

PLAYFULNESS | 75 ical investigation proposes that a “play drive” (Spieltrieb), acting as mediator of a dialectic between human sensuous and formal drives (Sinnestrieb and Formtrieb), effectively constitutes an aesthetic impulse (Schiller 1967). Play and artistic creation alike seem to thrive to some degree on “the joyous exercise of spontaneity” (ibid.). And much like play, art is universal to culture. The arts “travel across cultural boundaries as well as they do because they are rooted in our common humanity” (Dutton 2013:275).

Huizinga identified aspects of play in different art forms, and equated art contexts with the “magic circle” (Huizinga 1971:10). All too similarly, Dutton identifies a “special focus” as aesthetic universal:

“Works of art and artistic performances are frequently bracketed off from ordinary life, made a special and dramatic focus of experience. [...] These objects or performance occasions are often imbued with intense emotion and sense of community.” (Dutton 2013:274)

Furthermore, the same author also places “a universal human desire to avoid repetition and boredom” – something which is sated through play – as the “mainspring of artistic change” (270).

Huizinga’s writings have influenced important developments in contemporary experimental art (Rodriguez 2006), including Guy Debord’s Situationist International movement (Montola et al. 2009:262, Hjorth 2011:87) and Kaprow’s Happenings (Kaprow 2003:121). Beyond the playfully, highly improvisational nature of his work, Kaprow ex- tols the value of non-competitive play and views the potential of artists as ludic edu- cators (122). Aspects of play can be found as core strategies also in Surrealist, Dadaist and Fluxus works which appropriate existing games, game-like objects, aesthetics or processes (Hjorth 2011:90, Kaprow 2003:173, Thackara 2005:181); and are also found prominently in Interactive Art, especially in Japanese Device Art (Kusahara 2010).

4.5.1 · Evolution and Cognitive Play While art is commonly seen as exclusive to humans, other species have been observed to engage in behaviours combining aspects of play and aesthetics (Boyd 2010:4). Indeed “chanting, drumming, dance, and song [...] have analogues” among our close ancestors (114), but there are also examples among cetaceans and birds. These include the way dolphins create and play with bubble rings (4); as well as the

76 | ART AND TECHNOLOGY AT PLAY entrancing performances of birds of paradise and the exquisite arrangement of found items by bower birds, carefully contrived to impress potential mates.

The Chauvet cave drawings, dating back to 30.000 years BC, constitute the earliest ar- chaeological finding of an artwork (8). Given the apparent level of artistic expertise, however, we may “presume a long prior process of practice and experiment on surfaces like bark or skins that have not survived” (ibid.). It has also been proposed that the numerous findings of Acheulean hand axes displaying “no sign of use” yet a “high and perhaps excessive degree of symmetry and finish,” the “occasional selection and highlighting of incorporated fossils, and the existence of forms too large or small for apparent use” constitute an early precursor to the visual arts, dated to 1.4 million years ago (78, see also Dutton 2010).

Through “surviving cave art and prehistoric artifacts” we may also glean that “rhythmic movement (dancing), beating of bone-to-bone drums and flute sounds (music), wearing masks and/or costumes while impersonating other humans, animals, or supernat- urals (theatre)” were prevalent at the time of the Chauvet paintings (Schechner 2002:221); with further findings indicating the usage of ochre in body painting as early as 120.000 BC (Boyd 2010:114). This points towards “extensive traditions of art, as something humanly shared, incorporating sophisticated representation, involving expert practitioners” (9).

Author Brian Boyd sees human art forms and artistic behaviour as “a stimulus and training for a flexible mind” and suggests that these have evolved from the pre-existing propensity to play (85). Boyd argues that while “no art can be explained without culture, [...] a purely cultural approach does not suffice” (71). According to him:

“Culture cannot explain why some species have culture, why some have it more than others, why it takes partly similar, partly different shapes across a given species, and, in the human case, why art forms such a central part of all cultures. If art were entirely cultural, and culture not shaped in part by nature, art would occur in some groups and not in others [...]” (ibid.)

This leads towards the possibility of universal aesthetic values, but does not deny that there also exist culturally-specific ones, in the breadth and depth of artistic forms and expressions (Dutton 2013:275). Even if we discarded the possibility of universal aes-

PLAYFULNESS | 77 thetic values there should, at least, be no doubt at this point that artistic expression and aesthetic experiences are culturally universal in humans and their close ancestors.

From a psychological standpoint, art can be seen as “a kind of cognitive play” appealing not merely to immediate sensuous pleasures but also to the “preference for infer- entially rich patterned information” which is especially developed in humans (Boyd 2010:85). Arising less from the specific will to innovate than from “a universal human desire to avoid repetition and boredom” art involves “borrowing and sudden altera- tion, as well as slow changes” (Dutton 2013:273) which engage our evolved appetite for “information that falls into meaningful arrays from which we can make rich inferences” (Boyd 2010:14). Anthropologist Alfred Gell addresses the “cognitive stickiness of patterns” and the mental challenge they represent as an aspect of our aesthetic fascination, especially in artefacts from more primitive cultures. A similar cognitive stickiness is also a perceived quality of games (Hjorth 2011:88), playful systems greatly relying on pattern and variation as source of pleasure (Koster 2005:36).

The kinds of pattern that may engage our cognition are not exclusively visual, spatial or temporal. Broadly, our attention can be excited by information or processes which “signal regularities [...] rather than mere chance” (Boyd 2010:87), with variation upon a global or local structure. These include music, dance and storytelling (Gell 1998:95); as well as games (Koster 2005:36).

In his humane and bittersweet account of the “awakenings” of parkinsonian and cata- tonic patients treated with a miracle drug, neurologist Oliver Sacks observed first- hand the therapeutic value of music and dance in engaging and maintaining patients physically and mentally active (Sacks 2012). On a completely different field, anthropologist Alfred Gell recognizes “an externalized and collectivized cognitive process” in “the spatio-temporal structures of distributed objects in the artefactual realm such as the ouvre of one particular artist [...] or the historical corpus of types of artworks” (Gell 1998:222).

As cognitive play, art engages our attention and constitutes a source of pleasure. Like feeding, sleeping or sex, the gratification resulting of engagement in production and consumption of art indicates an evolutionary origin (Dutton 2013:271). Even so, some may argue that art is not necessarily a successful adaptation, but rather a by-product

78 | ART AND TECHNOLOGY AT PLAY of human mental skills: an artificial stimulus without actual evolutionary benefit or fitness function. In this perspective, art is seen to engage our cognition as fast-food or candy engage our appetite, both being a result of an abundance of stimuli through technical means, too recent to allow time for evolutionary processes of adaptation (Boyd 2010:81, Pinker 1999:525).

Boyd argues instead that art is a behaviour yielding rewards; a “Darwin machine” in its own right, “an evolutionary subsystem effectively designed in this case, for creativity” (Boyd 2010:121). While the outcomes gleaned from art are not genetically transmissible, they are especially beneficial within the context of a social group who can share knowledge, providing not only incentives for social exchange but also discovery and imaginative exploration of materials and processes; it enables science, encouraging “creative habits of mind that help technological discovery”, and in turn benefiting from technology “in a positive feedback loop” (123).

Whether art has evolved from play or simply channels it, it is hard to deny the benefits both can bring. Our social cohesion, empathy and cooperation were refined together with the ability to communicate and express ourselves, to devise and craft better tools, understand and exploit natural resources and processes, and inscribe knowledge onto artefacts so that it may stand the test of time. Art, technology, culture and play appear evolutionarily entwined – hardly separable and, at instances, nearly indistinguishable – for millions of years, in enduring synergy.

4.6 · Interactive Art In art as in play, anything is potential prime matter – something which Duchamp’s fountain illustrates all too well (Kaprow 2003:174).

Lévi-Strauss portrayed artists as bricoleurs, combining ideas, materials and processes from seemingly unrelated sources towards particularly expressive outcomes (Lévi- Strauss 1966). According to the interpretation of Jacques Derrida, the artist as bricoleur can be understood as:

“[...] someone who uses ‘the means at hand,’ that is, the instruments he finds at his disposition around him, those which are already there, which had not been especially conceived with an eye to the operation for which

PLAYFULNESS | 79 they are to be used and to which one tries by trial and error to adapt them, not hesitating to change them whenever it appears necessary, or to try several of them at once, even if their form and their origin are heterogenous” (Derrida 1993:231)

As we have seen, technology can be defined as “knowledge about skilful practices” and has as root the Greek “techne” which can be interpreted as “art, craft or skill” (Lister et al. 2008:87). While the term bricoleur may be seen by some as derogatory, portraying an enthusiast rather than a specialist, this is misleading: all art forms require “the exercise of a specialized skill” (Dutton 2013:273).

Even within the purest forms of performance art, creators employ “technical artistic skills” (ibid.). These include an uncanny, and often philosophical understanding and use of tools and techniques (Gell 1988, Kordjak-Piotrowska 2013) as diverse as the voice, body movement, a paintbrush, perspective or computer-generated graphics.

Skill is also revealed in the exploratory or strategic “misuse” of tools and media as playfulness and critique. This is prevalent in the strategy of détournement employed by the Situationist International movement (Dragona 2010:27). In a contemporary context, the détournement of digital games and their adoption as a medium for artistic expression fosters “innovative forms of new media practice” (Hjorth 2001:31). Through “modding, hacking, exploiting and other strategies of intervention” artists appropriate digital media outside of prescribed modes of use to reveal and subvert their underlying cultural rhetoric (Catlow 2010:8, Montola et al. 2009:276).

Gell portrays the uncanny skills of artists as carrying further the “enchantment of technology,” that is, “the power that technical processes have of casting a spell over us so that we see the real world in an enchanted form” and which is “immanent in all kinds of technical activity” (Gell 1992:44). For him, art and artworks constitute “a vast and often unrecognized technical system” – a “technology of enchantment” (Gell 1992:43).

Artistic exploration can “shift our focus” away from instrumental concerns “toward a deeper understanding of natural processes and social relationships” brought about through technology and associated practices (Thackara 2005:181); revealing hidden fa- cets or unrealized potential through “creative elaboration of the particular dynamic capacities” of technology “and of the ways that through them humans and machines to-

80 | ART AND TECHNOLOGY AT PLAY gether can perform interesting new effects” (Suchman 2007:23).

In making possible new forms of communication and interactivity, digital media lend themselves especially well as both tools for, and subject of inquiry (Lister et al. 2008, Munster 2006). Artistic approaches are able to shed light “on issues of networks, the body, identity, and collaboration” among others, “in a way that methods-driven solutions are not” (Thackara 220). Through creative and critical engagement with new media and its traits, “overlays, fusions, and hybridizations” thereof, art provides insight into the effects of technology on “modern mental life” and technological fantasies (Kaprow 2003:83). Furthermore, artists have increasingly become “aware of the potential of new media for providing intensive sensate engagement” (Munster 2006:180), not least through the possibilities for interactivity that it offers (Daniels 2008); and created pioneering works exploring human-machine interfaces and modalities of interaction “covering all manner of objects and environments, from screen to haptic interaction” (Munster 2006:118).

4.6.1 · Participation and Interactivity Having long depended on “the patronage of the palace and the church” the appreciation of the fine arts seems institutionalized in a way reminiscent of a monastery, with the museum “a direct parallel in mood, appearance, and function to the cloistered, un- attainably grand surrounding art once had” (Kaprow 2003:56). Works of fine art “de- mand for their appreciation physically passive observers, working with their minds” (64) within the museum space, where “the hushed atmosphere” and “the reverence with which one is supposed to glide from work to work” seem to imply that “reverent manners became (and still are) confused with reverence for art” (56).

As an attempt to break free of this limitation, one the main characteristics of contemporary art is the “tendency toward collaborative, participatory practice” (Groys 2008:19). Artists and artist groups “forego their isolated, elevated, privileged position in relation to the audience” and “pointedly stipulate collective, even anonymous, authorship of their artistic activities” (ibid). Bridging “the schism in Western thought that separates the creator from the spectator” (Thackara 2005:181) the artist “invites the participant to make a choice of some kind” (Kelley in Kaprow 2003:xviii). This can be as straightforward as the choice to participate (ibid.) but can extend to great lengths,

PLAYFULNESS | 81 even “requiring that creation and realization, artwork and appreciator, artwork and life be inseparable” as is the case in Kaprow’s Happenings (Kaprow 2003:64). Participation extends the potential of performance as inquiry from the artist to the other participants, and builds on the innate drive for play (Kaprow 2003:177) including “our impulse to craft performances and to engage with the performances of others” (McCon- achie 2011:35). By offering the audience the possibility of actively taking part in the process of conception and/or reception, participatory artworks are engaging embodied beings instead of disembodied spectators “in actions that transform art into experience and esthetics into meaning” (Kelley in Kaprow 2003:xviii).

Some influential schools of thought consider the interpretive experience of a viewer, listener or reader as an active form of participation in or interaction with an artwork (Daniels 2008:29, Lister et al. 2008:50). The present text, however, addresses interactivity in artworks as the ability to actively respond to the recipient’s actions, allowing him/her to intervene in the work’s visual, acoustic or textual form and behaviour (Daniels 2008:29). As it should be understood by now, particular emphasis is put on works which rely on computational technology to achieve that end.

The 1960s saw the birth of the first graphical interfaces and pointer devices on computers, whereas up until then interaction was mostly restricted to textual content and typed commands (Kwastek 2008:18, Dourish 2004:9-13). Ivan Sutherland’s pioneering “Sketchpad” made use of a “light pen” as input device to afford the sense of directly manipulating geometrical shapes on the screen (Kwastek 2008:18). Shortly after, Doug Engelbart and Bill English would devise the now-ubiquitous computer mouse, with a form factor chosen through physical experimentation (Moggridge 2007:17-19). These innovations can be seen to have founded the highly specialized yet interdisciplinary field of human-computer interaction (Kwastek 2008:18). Throughout that same decade, artists experimenting with electronic technologies were creating artworks which “interacted with their environment in one way or another, mostly via light and sound sensors” and were thus dubbed “cybernetic, responsive or reactive” (19). From the early 70s onwards, pioneering artists such as Myron Krueger, Lynn Hershman and Jef- frey Shaw further developed and explored cybernetic aspects in their works (ibid.), prompting periodical events dedicated to these emerging art forms – the oldest of which may well be the Ars Electronica Festival, established in 1979 in Linz, Austria

82 | ART AND TECHNOLOGY AT PLAY (Janko et al. 1996). With the increasing ubiquitousness and accessibility of digital media and especially the internet, continued engagement in cybernetic and reactive artworks led to the establishment of “interactive art” as a genre in the early 1990s (Kwastek 2008:19); with permanent dedicated venues and research institutions such as the Ars Electronica Center (Janko et al. 1996).

Depending on how they are employed according to the artist’s strategy, interactive media offer different degrees of control, authorship or choice in the context of an artwork (Kwastek 2008). Beyond the multiple apprehensions and interpretations afforded in more traditional art forms, interaction opens up a potentially infinite space of possibility where members of the audience can effectively play with the work and construct a unique, personal experience (Salen & Zimmerman 2004:66).

The underlying logic supporting interaction may range from a finite and closed sets of systematic choices to more complex emergent systems of interdependent rules (Daniels 2008, Kwastek 2008, Salen & Zimmerman 2004). While some authors may argue that “no true interaction is possible when one must select from a predefined set of options,” the inner computational logic of an interactive art piece does not necessarily have to be open or emergent in order to convey a meaningful experience (Friel- ing 2008:35-36). Some artworks may not require any systematic openness to be effective, such as Jeffrey Shaw’s Legible City or Mary Flanagan’s giantJoystick. Other works may borrow potential for emergence from non-algorithmic sources, such as the often whimsical and unexpected actions of participants, whose performance is an integral part of the aesthetic (Frieling 2008, Kaprow 2003).

4.6.2 · Procedural Rhetoric By enabling a dialogical relationship between recipient and artwork, interaction can be employed as a powerful form of expression. If we consider, as previously discussed, how action and understanding are tightly coupled (Dourish 2004:17, Suchman 2007:79), then participating in a time-based artwork can offer uniquely intimate and intuitive forms of perception (Kaprow 2003:117), as their time-based or performative aspects “powerfully enhance our capacity to understand processes and systems” (Thackara 2005:181).

Games and other interactive play-spaces are observed to have a “semiotic complexity”

PLAYFULNESS | 83 in their interactive and time-based nature (Lister et al. 2008:298) brought about by their procedurality and the possibility to playfully act upon it. Whereas, for instance, a character in a book is immutably represented, a character in a game can be interacted with and may react differently throughout time as the result of the player’s unique behaviour:

“The book's story would describe how the witch lives in the woods, makes magical potions, and so on. [...] Although both witches might have the same ‘literary’ characteristics, the witch in the book would not possess these traits as procedural qualities, which are triggered as part of an activity of meaningful play. [...] Actively exploiting the witch's witch-like qualities, not just by reading about them but also by playing with them, is what makes her representation so powerful to experience in the context of the game.” (Salen & Zimmerman 2004:436)

Non-interactive media “carry deep in their DNA the fundamental structure of a statement” whereas interactive media “are more like meaning machines, or meaning networks” (Montola et al. 2009:276). Game designer and scholar Ian Bogost characterizes this semiotic complexity as “procedural rhetoric” (Bogost 2010:2). It is worth pointing out that the use of the term “rhetoric” here expands beyond oratory and direct per- suasion, to more generally denote an effective mode of expression, including “writing, speech, or art that both accomplishes the goals of the author and absorbs the reader or viewer” (19). Existing “verbal, written, and visual rhetorics inadequately account for the unique properties of procedural expression” (29) as a type of “symbolic expression that uses process rather than language” and is inscribed “in a medium that actually enacts processes rather than merely describe them” (9).

As with other strategies in expression, procedurality and interactivity can be deployed in a wide range of creative practices, from games to interactive art (Catlow 2010, Daniels 2008, Bogost, Humble 2006). Quite similarly to Dourish’s concept of embodied interaction, the expressive power of procedural media and interactivity rests in the way they closely relate to our understanding of the world by being and acting in it – using the results of our past actions to inform our subsequent ones in a continuous feedback loop, making cause-effect inferences from patterned information and ascribing forms of agency to other actors (Boyd 2010, Pinker 1999, Suchman 2007).

84 | ART AND TECHNOLOGY AT PLAY 4.7 · Strategic Values Playfulness is seen in various forms as a common trait of postmodern art. As such, it would expectedly manifest in one way or another throughout the author’s works. While indeed there is an overtone of playfulness to the whole, it is not for the sake of being postmodern, or necessarily a result thereof; but rather primarily because art is a form of play or – should the reader take offence at the apparent reductionism – a way to harness play as an ingredient for expression.

Nevertheless, playfulness was consciously and intentionally employed as pervading mindset throughout practice, driving experimentation with unusual, even borderline ridiculous scenarios and premises (Headbang Hero, Rambler), as well as the fruitful combination (including hacking and bricolage) of materials, processes or experiences otherwise perceived as unrelated (Rambler, Weltschmerz, Lorm Hand).

Humans, more than other animals, have the drive to play. This manifests through different behaviours and appetites, a few of which the works attempt to engage by providing formalized ludic activities. These include exploration and investigation (Noon, Wcielenie, Tripo), fiction and storytelling (likewise) and fantasy, not least through role-playing (Noon, Wcielenie, Headbang Hero), magic or science-fiction (Noon, Wcielenie, Tripo).

As a procedural medium, an interactive artwork can be powerfully expressive. Enga- ging the participant’s playfulness may reinforce the work’s interactive aspects (Noon, Headbang Hero, Tripo) or critical reception of its message (Headbang Hero, Rambler, Weltschmerz). Ultimately, however, playfulness is an attitude, not a gesture or a rule. As such, it cannot be guaranteed by design, only encouraged and rewarded.

PLAYFULNESS | 85

5. WORKS

This chapter introduces, details and discusses the works which constitute the main focus of the present text. These were produced between 2006 and 2014. Although they do not represent the complete collection of works by the author, or those in which he found himself effectively involved, they are the ones chosen as most relevant for the present purpose; and they all constitute original concepts in whose conception and development the author was meaningfully involved from the beginning.

The works are presented in chronological order. Each begins with a succinct introduct- ory description. This is followed by the contextual and/or inspirational background of the work, which may include textual sources, previous works, other artworks, research projects, cultural phenomena and media – including film, TV series, music albums and videogames.

Following is the narrated development of the concept into actuality, in terms of form, technique and interaction design. Naturally, some designs and creative choices are extremely complex to rationalize, as they are of serendipitous origin, but these do not constitute the norm. Although occasionally technical for relevance or completeness, the description mostly avoids fastidiousness and unnecessary detail.

The final section for each work presents and interprets outcomes based on the experience of producing the work, exhibiting it, observing its reception and conversing with participants.

The collection of works is admittedly very heterogeneous. They deal with different topics, assume different forms and have a different roles – including to entertain, to question, to explore, to empower, to inform and combinations thereof. As such they are not interpreted in the present text as serving an ulterior motive or goal within a total- ity. That was rarely, if ever, their purpose.

Rather, the focus is on the theme and rhetoric of each work in itself. This includes how specific aspects of embodied interaction and/or playfulness empower the work’s efficacy in playing its intended role, whenever reasonable – that is, avoiding lengthy repetition of the previous chapters to highlight what is otherwise pervasive or obvious. Nev-

WORKS | 87 ertheless, the work is considered as a whole, and therefore other aspects of its meaning or reception are highlighted or interpreted, which may or may not be directly (or obviously, at least to the author) related to embodiment or play. The reader is certainly encouraged to interpret connections that have escaped the author. Art is, after all, a very personal experience.

A complete list of exhibitions by work, as well as distinctions, can be found in Ap- pendix (see Appendix A). Also appended are a list of related publications (Appendix B) and press clippings (Appendix C).

88 | ART AND TECHNOLOGY AT PLAY 5.1 · The Gauntlet 2006 - 2008 With help from Christina Heidecker

The Gauntlet is a wearable interface allowing real-world objects to be used in ubiquitous computing applications, with a focus on games. Taking the shape of a stylish bracer, it contains sensors which allow for detection of tagged objects held by the wearer, as well as gestures the wearer is performing. It also has a haptic actuator to convey basic feedback. The wearable was designed to be used together with a mobile device, which runs the software for sensor data analysis, provides audiovisual content and other functionality (e.g. GPS location, network connectivity). The Gauntlet constitutes a user interface providing tangible, intuitive and immersive interaction in mobile gaming environments, untethered from location-specific hardware and software frameworks. As a wearable meant for use in outdoor and public environments, its design takes into account shape, size, weight and placement, among other factors of wearability, as well as its aesthetics. The potential use of the Gauntlet was illustrated and evaluated through an interactive storytelling installation.

5.1.1 · Background and Concept The starting point for what would be the main motivation behind Gauntlet was the author’s work on and observations gathered through the InStory project (Correia et al. 2005) at the Interactive Multimedia Group of the Center of Informatics and Informa- tion Technology, Faculty of Sciences and Technology of the New University of Lisbon.

InStory InStory (Correia et al. 2005) was developed as a platform for mobile storytelling, information access, and gaming activities in partnership with the Cultursintra Founda- tion, making use of the cultural heritage site Quinta da Regaleira in Sintra, Portugal. The system was interacted with by means of a mobile device: a PDA, essentially a palm-sized computer with a touchscreen, Wi-Fi and GPS, at a time when smartphones were a novelty and not nearly as technically (and socially) powerful as they are

WORKS | 89 nowadays. Determining the user’s location through GPS or Wi-Fi triangulation, the system could present context-based multimedia content, allow users to record and share geo-tagged multimedia elements, and deliver an interactive narrative as a playful motivation to explore and discover the site.

While the system was designed to assist and augment the experience, highlighting and complement the beauty of the surrounding environment, the limited interaction afforded by the device seemed to occasionally disrupt it instead. Users did frequently have to shift their focus of attention towards the device and its screen, and the author strongly perceived this as a limitation to be addressed as potential topic of research.

The original motivation behind this first work was then to provide a natural interface complementary to that of screen-based mobile devices (such as PDAs and upcoming smartphones) so that a participant’s gestures and his/her interaction with real, physical objects could be used as part of mobile or ubiquitous computing scenarios. As a result, the development of the Gauntlet concept was focussed on its functional aspects as a human-computer interface.

As a more specific context of use, emphasis was from the outset put on ludic activities, including gameplay (Salen & Zimmerman 2004:303-304, see 4.3). Besides taking advantage of the author’s existing knowledge of digital games and emergent fascination with location-based narratives, this choice also enabled the freedom of experimenting within a bigger breadth of possible interaction modalities and goals, including fantastical scenarios as well as expressive and culturally-constructive activities closer to the domain of interactive art.

The ludic formats of live-action roleplaying and other pervasive or ubiquitous games (particularly those assisted by technology) were found especially informative, providing a more specific set of existing research contexts within which to develop an approach.

Pervasive Games and LARP In live-action roleplaying (LARP) players experience narratives with gameplay elements through live performance. Combining aspects of tabletop roleplaying games, improvisational theatre and historic re-enactments, LARP commonly implies the use of costumes and props in order to foster immersion. In some extreme cases, players will re-

90 | ART AND TECHNOLOGY AT PLAY main in character for long periods of time; and locations, events and players’ actions intermingle with reality or “normal life” in a way as to be indistinguishable by unaware spectators (Jonsson et al. 2007).

The use of objects as props within the game is commonly governed by rules, accounting for how participants signal their in-game actions, how the result is determined and what impact it has in the state of the game or story. In instances, each object used in play has an associated information sheet (a laminate) detailing its in-game properties (Falk & Davenport 2004). In these and other aspects, a LARP can inform the design of other immersive, real-world gaming activities.

LARPs constitute a form of pervasive game, that is, “a game that has one or more salient features that expand the contractual magic circle of play spatially, temporally or socially” (Montola et al. 2009:12). Considering this definition, pervasive games are a broad category, including activities with an emphasis on technology but also those with very little or no technological support (Montola et al. 2009:31-46). Focus was put on technology-supported pervasive games, that is, games where information technology is used to support at least some of the game’s aspects (Montola et al. 2009:164). A great number of technology-supported pervasive games tend to happen within or in relation to academic research on ubiquitous computing (Broll et al. 2006, McGonigal 2006) and give a prominent role to computer systems and human-machine interfaces. Game designer Jane McGonigal refers to these specifically as ubicomp games (McGo- nigal 2006).

Even within this category, games can understandably be quite heterogeneous in terms of interfaces and technological support; but also in terms of gameplay mechanics, complexity, narrative and duration (Broll et al. 2006).

A tentatively basic example is the Bill game, where players collect virtual coins by navigating to specific real-world locations outside of Wi-Fi coverage, and then navigate back to within Wi-Fi coverage to “cash-in” and score points (Chalmers et al. 2005). The game is mediated through a GPS- and Wi-Fi-enabled mobile device, and has the interesting strategy of “seamful” design: instead of “hiding” potential Wi-Fi coverage issues from the players, it rather incorporates these as an essential aspect of gameplay.

As contrast, a rather complex example is Epidemic Menace, supported by an immers-

WORKS | 91 ive narrative using live actors and asymmetric gameplay, where each player has different roles and capabilities matched to different interfaces (Lindt et al. 2006, cf. Fischer et al. 2007). These included an off-the-shelf mobile device but also an innovative wearable (if cumbersome) augmented reality system, overlaying computer-generated graphics onto the wearer’s view of his/her surroundings.

It must be pointed out, however, that pervasive games relying on technology also occur in artistic contexts. The continued work of group Blast Theory is a prime example, starting with the early yet popular Can You See Me Now, which combined real and virtual spaces, pitting on-location “runners” against online players. Blast Theory’s extensive body of works includes other forms with heavier emphasis on narrative, trust and choice, such as Uncle Roy All Around You, A Machine To See With and Day of the Figurines.

One of the recurring issues in technology-supported pervasive games is the effort necessary when setting up the game’s supporting technological framework (Broll et al. 2006, Montola et al. 2009:163-167), making it cumbersome to stage the game repeatedly at different locations. This issue is partly mitigated by the fact that game sessions in ubicomp games (using McGonigal’s term) usually happen within a research context or as part of a larger event.

As such, besides detecting gestures performed and objects touched or held by the wearer, the Gauntlet interface was devised to be usable in a highly mobile context – in a limit case, as an everyday accessory – with the least possible dependency on complex or site-specific frameworks. In order to achieve this, the chosen approach was to place sensors on the user rather than on the environment, housed within a wearable form.

5.1.2 · Design and Development Given that the wearable would house sensors to track held objects and user gestures, the focus being on the user’s hand, the format of a bracer was preferred to that of a glove. A glove would have covered the fingers, which was unnecessary and even un- desirable, especially in everyday use scenarios. A bracer would be both unobtrusive for hand usage, and fit different users more easily (Gemperle et al. 1998).

The first step was to research what sensors and actuators could be used, based on their functionality and availability, and construct a prototype. Informing this process

92 | ART AND TECHNOLOGY AT PLAY was Intel’s research on human activity recognition and tracking, which made use of a custom wearable device with RFID- and motion-sensing capabilities (Smith et al. 2005). This implied an assumption that in a near future most everyday objects would contain RFID tags or a similar unique electronic identifier, something which has not yet come to pass roughly ten years after. Nevertheless, in a hypothetical case any object could potentially be appropriated as a tangible human-computer interface.

Detection of basic gestures would rely on data from an accelerometer (for the hand’s movement), and one or more pressure sensors (for detecting an open/closed hand). Additionally, a digital compass was included, to allow for estimation of the hand’s direction (used mainly when pointing). Detection of objects would be achieved by an RFID module and antenna, which would detect unique RFID tags placed on objects.

The first technical experiments were supported by an Arduino board, at the time an emergent easy-to-use electronics platform aimed mainly at enthusiasts (including artists and designers) and quick-prototyping. At the time, the author’s previous practical experience with electronics was informative yet fairly limited. In its ease of use, the Arduino platform quickly became an essential tool – and would remain so throughout the body of work.

Prototype A prototype was developed (figure 5) with the help of fellow student Christina Hei- decker and demonstrated at the PerGames symposium (Salzburg, 2007) as part of a poster session (Martins et al. 2007). It used off-the shelf sensor modules and a much smaller version of the Arduino board. Electronic components were located in the bracer proper, with the exception of the relatively small RFID antenna, placed at the centre of the palm of the hand, within a band that was made small enough as to not constrain use of the hand, especially the fingers (figures 2 to 4). It lacked a battery and wireless connection, being connected to a computer through a USB cable. The computer displayed only numeric sensor data from the Arduino environment’s console. For a matter of focus, the bracer was designed for right-hand users.

Mock-ups of the components’ shape were made in order to determine placement and manufacture protective casings using synthetic modelling clay (figures 2 and 4). This already included placement of a Bluetooth module, to be added in the future. A rub-

WORKS | 93 ber working glove was modified into a bracer. A cover made of transparent plastic allowed to assess possible shifting of the components during use (figure 5).

At the PerGames symposium the authors had the opportunity to try a glove with RFID and GPS capabilities developed through joint research at the Swedish Institute of Computer Science and the University of Tampere, for use in the technology-assisted pervasive LARP Momentum (Jonsson et al. 2007). This glove was quite cumbersome due to the weight of the batteries, but especially due to the size and placement of the RFID antenna, which covered most of the wearer’s palm and constrained normal use of the hand. Nevertheless, it was meant to used only in a very specific game activity within Momentum; and its overall look and feel fit quite well with the game’s theme and aesthetics.

Having exchanged approaches and gotten fairly positive remarks, work on the prototype was furthered to produce a better-looking, wireless and fully-functional first version.

First Version In relation to the prototype, the main goals for the first complete version of the Gaunt- let were: to add a pressure sensor to the palm of the hand; to make the wearable completely wireless; and to build a bracer form which would balance robustness and look. Additionally, an interactive experience would be developed to demonstrate and test a possible use.

The layout of components remained the same. A Bluetooth module was added, placed as predicted in the prototype. The choice of Bluetooth made it possible to connect the Gauntlet to any other bluetooth-enabled programmable device (e.g. computer and PDA), which would then analyse sensor data to derive the wearer’s actions. A pressure sensor (circular force-sensitive resistor or FSR) was also included, as well as a small vibration motor that provided basic haptic feedback. As both the RFID antenna and pressure-sensor were already taking up the palm of the hand, the actuator was placed on the inner side of the wrist, for good tactile sensitivity. A green LED placed on the wrist provided feedback on the device’s status, lighting up when the device is turned on, and pulsing when an RFID tag is detected. To power the components, a casing for four AAA 1.5 Volts batteries was included, placed on the inner side of the fore-

94 | ART AND TECHNOLOGY AT PLAY Figure 2: Component placement sketch for the Gauntlet.

Figure 3: Gauntlet hardware prototype. Figure 4: Gauntlet hardware placement.

Figure 5: Finished prototype for the Gauntlet.

WORKS | 95 Figure 6: First version of the Gauntlet.

Figure 7: First (top) and second (bottom) versions of the Gauntlet.

96 | ART AND TECHNOLOGY AT PLAY arm.

A few experiments were made to envelop most of the sensor modules in silicone, instead of the previous synthetic clay housings. This would hopefully better protect the components against shock.

The design and construction of the textile part was commissioned to Isabelle Habs- burg (at the time a student in Textile Art and Design at the Kunstuni Linz), following the existing guidelines for shape, using black leather for the outer layer and a mix of black neoprene and rubber foam for the inner layer. The hand strap housing the RFID antenna and pressure-sensor was fastened using velcro, whereas the remaining form used buckles, a choice which further granted the Gauntlet a “goth” style, which would be in line with the upcoming interactive installation (figures 6 and 7).

The interactive installation Noon – A Secret Told by Objects is detailed in its own section (see 5.2). For the present purpose, we may briefly describe it as an interactive narrative where a participant is invited to don the Gauntlet and manipulate found objects, thus accessing their memories and piece together the tragic story of the family to whom they belonged. Besides the handling of objects, the experience also included interactions based on the gesture of pointing. Noon was first exhibited at Campus 2.0 Neoanalog, part of the Ars Electronica Festival 2007. Narrative elements were delivered as text through a PDA enclosed in a small book, which ran all the software and logic, including basic gesture detection.

During the six days of showcasing Noon, the author was able to observe dozens of festival visitors using the Gauntlet, and had the chance to personally and informally discuss the experience with several of them. The Gauntlet was hard to don without help: the straps were hard to fasten single-handedly, especially while the weight of the batteries caused the bracer to slide to one side. Once worn it was comfortable enough, although it was perceived as warm when worn for longer periods, especially as the exhibition took place at the end of a Summer. There was some initial difficulty in placing the hand correctly on objects so that the RFID tags were detected. Some participants found the haptic feedback distracting at first.

One of the most interesting observations, reported by different users, was the sense of “power” that the Gauntlet imparted on the wearer. This has been attributed to the

WORKS | 97 Gauntlet’s look and feel, its narrative role in granting the “special power” of accessing the memories of objects, but also to the fact that it is actually worn by the participant. These findings were presented and published at the 10th International Conference on Human-Computer Interaction with Mobile Devices and Services (Martins et al. 2008a).

Usability Study With the installation Noon as testbed, a formal usability study was conducted. Find- ings were presented and published at the International Conference on Advances in Computer Entertainment Technology in 2008 (Martins et al. 2008b). The study was conducted on a population of sixteen voluntary participants, twelve of them college students and six of them female. The subjects ranged in age from 22 to 49 years old with a mean age of 28.6. Two of the participants were left-handed, but were asked to wear the device on their right-hand, as its form factor and the gesture recognition algorithm running on the PDA had been, for a matter of time and focus, limited to right- hand usage. Twelve of the participants reported they usually play digital games. All participants had their first contact with the wearable device during the test and used it under similar conditions.

The participants’ feedback was generally quite positive. The majority considered it easy to learn and intuitive (within the context of Noon, at least) as well as enjoyable to use. Half of the participants reported their experience as “stimulating”, “pleasant” and “simple”. The wearable’s aesthetic was also found pleasing by a majority, although when inquired if they would wear the device in public on a daily basis the responses were quite heterogeneous. A small yet significant number of participants observed that the device was “warm” and “big”, “writing becomes difficult while using the device”, and that it was “not suitable for precise movements”. These last two observations were important to inform the impact usage for long periods of time – for instance, as a mix of game device and fashion accessory in intermittent game activities.

While the straps and buckle system used to fasten the bracer (figures 6 and 7) was not directly addressed by participants in the usability study, it had already been observed during the Campus 2.0 event that these made the task of donning the Gauntlet harder.

98 | ART AND TECHNOLOGY AT PLAY Second Version Following some of the previous observations, a second version of the Gauntlet was produced, to be lighter, have a smaller form-factor and be easier to put on and take off (figure 7).

Most of the hardware remained the same, with the exception of the RFID reader. The previously used module was the biggest component within the Gauntlet, so an alternative was found in a smaller module, roughly the shape and size of a 2 Euro coin, with an embedded antenna. This module was more expensive and harder to obtain than the previous one, and also implied the use of altogether different RFID tags. However, it allowed to reduce the size of the wearable (figure 7 shows a comparison), as the small circular shape was placed at the palm of the hand. There was some concern that the module might be accidentally broken by users pressing too hard on a surface, but that was never the case. Additionally, a step-up converter module was included, allowing the device to be powered by only 2 AAA 1.5 Volts batteries, further reducing the weight of the wearable in a trade-off with autonomy. In this version the electronic modules were not directly encased in any protective material. Rather, the textile bracer contained inner layers of foam and neoprene padding.

The second Gauntlet was used in a revised version of the Noon installation, presented during the Lange Nacht der Forschung (Long Night of Research) at the University of Art and Industrial Design Linz, on September 26, 2008. It was also featured in a low-fidelity demonstration of Noon at the Creative Showcase of the International Conference on Advances in Computer Entertainment Technology (ACE ‘09) in 2009 (Martins et al. 2009b). For unfortunate reasons, formal usability studies were not conducted on this second wearable.

5.1.3 · Outcomes and Interpretation The Gauntlet (prototype, first and second versions) was the very first electronic device crafted by the author, whose background in computer science and engineering did include some experience with and understanding of electronics, albeit in a more theoretical perspective. As such, developing the Gauntlet also fostered prototyping skills and an interest on the (then emergent, now greatly popular) Arduino platform; as well as in sensors, actuators and their use in interaction aesthetics and creative contexts.

WORKS | 99 Among other things, this encouraged a complementary mode of thinking to the design-driven “how to achieve a specific function or effect with existing technology,” which can be expressed as the exploration-driven “what kinds of function or effect can be achieved through a specific technology,” informing the ideation and development of subsequent projects.

Being the first work in the series, the Gauntlet focussed mostly on technological challenges and functional issues – at least initially. It was developed as an interface for ubiquitous tangible interaction, technical concerns having played a heavy part in its conception.

However, the necessity of working towards a wearable form fostered the curiosity about the possibilities and challenges associated with wearable interfaces. As a result of the work conducted on the different versions of the Gauntlet, the author’s scope of research practice was steadily broadened from the more strictly functional and technical concerns, to include factors such as wearability (Gemperle et al. 1998) and personal, communicative and expressive aspects of clothing and garment (Barnard 2002, Seymour 2008, 2010). This would later manifest especially in the work Rambler (see 5.4). The shift in scope happened throughout development and presentation, supported by the usability study, informal observation and discussion.

Both versions of the Gauntlet were fully-functional wearable interfaces, not only in terms of technical capabilities but also in regard to look and feel (figure 7). Whereas a prototype can focus exclusively on technical challenges and purposefully overlook other aspects of the experience, a fully-functional wearable must take into account its characteristics qua worn item, and how these interact and often clash with its more technical underpinnings. As such, attention had to be paid to how the Gauntlet was to be put on and taken off, how it might fit different wearers, how it would look and feel while worn, and how these influence the perception users have of the device and experiences it mediates.

One of the most interesting observations made by users was their feeling of power imparted by the Gauntlet, heightened by the aspect of being worn. As it refers to the hand, a major focus of direct human action upon the world, and is worn close to the body, it facilitates withdrawal into a ready-to-hand (zuhanden) mode and can perceptually become a part of the body (Gemperle et al. 1998). The Gauntlet was not only

100 | ART AND TECHNOLOGY AT PLAY perceived as a tangible, materialized channel for the “superpowers” granted within the fictional context of Noon, but moreover as an augmentation of the participant’s own sensory and cognitive abilities (Viseu 2005:11). In the way it blends technology with clothing, it effectively works as an amplifier of fantasy (Seymour 2008:12) which positively impacted the sense of immersion within the fictional context of an interactive narrative.

Similarly, the Gauntlet provides a means for embodied interaction. As mentioned, the device’s wearability and placement can facilitate a ready-to-hand mode, but also em- phasizes our cognitive predisposition towards the manipulation of physical objects (Pinker 1999, F. R. Wilson 1999). It illustrates how existing objects may be appropriated or augmented as tangible interfaces (provided they have some form of unique electronic identifier, or afford other means of automatic recognition). Such a system could theoretically provide “infinite affordances” (Montola et al. 2009:17) not only for pervasive games but for human-machine interaction in general. In addition to the deep engagement with objects, the use of gestures as an interaction modality can be seen as building on human empathetic capabilities (Boyd 2010:97, Tomasello 2009:15) and even the precursor to human speech (F. R. Wilson 1999). Pointing, specifically, is a gesture exclusive to humans (Tomasello 2009:15, 73) “a fundamentally embodied activity even in the most straightforward physical sense of the word” (Dourish 2004:147).

While products such as Sony’s EyeToy – similar in concept to Myron Krueger’s work Videoplace dating to the 70s – had already been marketed with some success, gesture- based interfaces became increasingly common in consumer electronics. The Nintendo Wii console came out in late 2006 (the same year in which work on the Gauntlet began), its motion controllers an integral part of the experience that engaged audiences of a wide range of ages, a success which later prompted similar products such as Sony’s PlayStation Move controllers. Smartphones started including accelerometers and gyroscopes, allowing gestures such as rotating the device to be detected by applications. As a result of successful crowd-funding campaigns, devices like the Leap Motion optical sensor or the Myo bracelet (which detects gestures by measuring electric properties of muscles) show continued mainstream enthusiasm about the exploration and inclusion of gestures within human-computer interaction.

WORKS | 101 5.2 · Noon – A Secret Told by Objects 2007-2008 First version with Christina Heidecker

Noon is an interactive installation with ludic aspects, where a participant unveils pieces of a narrative by manipulating objects. As a player, his/her task is to puzzle together the origins of a tragic fire by interacting with the belongings of the deceased. The supporting fiction casts the player within the role of an investigator able to access memories contained within objects. Noon was developed together with the Gauntlet (see 5.1), as a means to showcase and test potential interaction modes for the wearable interface, albeit in a controlled and stationary setting. Drawing on the significance of artefacts as historical media, it constitutes a novel approach to interactive storytelling and explores the notion of augmentation through technology-enabled superpowers.

5.2.1 · Background and Concept Noon was developed to demonstrate and observe usage of the Gauntlet (see 5.1) in an engaging, playful setting. While this was the main motivation behind the work, it constitutes a unique experience on its own.

Together with the Gauntlet, Noon was a direct successor to the author’s contribution in the InStory project (see 5.4.1). Whereas InStory mediated location-based content and interactive fiction as users moved within a physical space, Noon allowed participants to access and interact with fictional content by touching and manipulating physical objects (as opposed to tapping the more generic touchscreen). The experience was, however, limited in physical space. This was a conscious choice, in order to make the installation easier to prepare, attend to and observe in use.

The overall theme borrows heavily from the trope of paranormal investigation often found in contemporary pop-culture fiction. Among the author’s more direct sources of inspiration were: the critically acclaimed TV series The X-Files, where two downtrodden FBI agents investigate cases with a connection to the paranormal; the equally ac-

102 | ART AND TECHNOLOGY AT PLAY claimed series Twin Peaks, portraying the investigation of a murder in a small North American mountain town with a mesmerizing paranormal tinge; the boardgame Cluedo (also know as Clue in some countries) wherein players investigate a murder to determine not only the culprit, bus also the location and object used in the dire deed; and the album Scenes from a Memory by progressive rock band Dream Theater, with its fictional story of a man whose recurring vivid dreams prompt him to undergo hyp- notic regression to a past life.

Paranormal phenomena such as spirits or electronic voice phenomenon (EVP) are often used to characterize behaviours and functions of technology within pervasive games. Among other things, such an approach is useful to explain virtual incorporeal entities in the game as “invisible” forces which can only be perceived and affected through magic or technical means (Montola et al. 2009:89).

5.2.2 · Design and Development The first version of Noon – A Secret Told by Objects was developed in collaboration with Christina Heidecker, who also contributed to the development of the Gauntlet. Noon was specifically designed to be played using the Gauntlet, paired via Bluetooth to a PDA running the game’s logic and providing audiovisual output. The PDA’s limited computational capabilities together with the authors’ lack of specialized knowledge in pattern-detection algorithms led to the constraint of designing an experience around very basic gestures, recognized with a minimum of fidelity. This was especially daunting, since different participants may perform a given gesture in different ways.

Interaction Design The set of gestures which were successfully prototyped consisted of basic movements and rotations of the hand. These of course affected the choice of objects. As such, the installation was from early on meant to feature a cup to be poured, a snowglobe to be shaken and a hammer to be swung. In the same way, game mechanics making use of pointing or opening/closing of the hand would eventually be present.

To offer access to a larger quantity of memories while keeping the number of objects small, a means of navigating through memories in time was devised, using a pointing gesture. The installation would be surrounded by five equally-spaced markers, each representing a successive time period up to a tragic event. By pointing to a marker,

WORKS | 103 the player could shift time for all the objects and associated narrative excerpts. Tech- nically, this was achieved by using the Gauntlet’s digital compass, and by assuming that the player was standing in a more or less fixed location in relation to the markers. To avoid false detections of the pointing gesture, the player’s intention to shift time had to be signalled by first touching a specific object.

Old objects were then sourced from a local flea market, periodically taking place at the main square in Linz. The decision on which objects to acquire for use was made in regards to their physical adequacy, condition and price. The acquired objects and their planned usage were: a snowglobe to be shaken; a hammer to be swung; a cup, to be poured; a clock, to signal the time-shift gesture; and additionally a schoolbook, as well as couple’s picture, with no associated gestures (figure 12).

Narrative As a set of interconnected pieces of text, the narrative was devised with constant shifts between a top-down approach (“what does this object have to reveal throughout time within the larger context?”) and a bottom-up approach (“how can the larger context be structured in order to incorporate this object and interactions with it?”). The first outlines were in place before the acquisition of the actual objects, and were afterwards adapted and refined in function of these, when necessary.

The narrative emerged then, as events taking place in the morning leading up to the tragic demise of the Novak family: a fire that started at noon. As their house burst in flames, Mr. Novak, the two children and the house maid perished in the fire. Mrs. Novak was found barely alive, yet incoherent and insane, a state from which she never recovered. Besides Mrs. Novak, only a few objects were salvaged from the wreckage, miraculously unscathed. As an investigator and spiritual medium donning a “techno- magical” bracer, the player is tasked with retrieving memories from the surviving objects, piecing together parts of the fateful morning, linking people, events and locations to eventually unravel the (intentional) origin of the fire.

Each object was to reveal memories related to its immediate surroundings, and so each was associated with a location within a house: the snowglobe in the hallway, the cup in the kitchen, the hammer in the cellar, the schoolbook in the children’s room and the picture in the parent’s bedroom.

104 | ART AND TECHNOLOGY AT PLAY Refinement What would be the definitive interaction design of Noon was mostly in place at this point. Touched or held objects would yield a narrative fragment, displayed as text on the PDA’s screen, with five different fragments per object corresponding to five distinct time-periods. The player could navigate to a given time-period by touching the clock and pointing towards one of five markers, represented by candles to heighten the im- pression of a magic ritual. For a few objects and at given time-periods, performing a specific gesture would reveal an additional fragment, rewarding the player with further insight into the story.

To facilitate the understanding that certain gestures should be employed, queues were introduced within some of the memories, as verbs in bold text (figure 8, left). For example, the text excerpt corresponding to the snowglobe at the 9:00 time-period reads:

“Mrs. Novak is walking through the house. She enters one of the bedrooms. Ten minutes after she leaves the house, obviously shaken.”

The queues were not always obvious, but when understanding one players felt encouraged to experiment with other combinations of gestures, objects and time-periods.

An additional mode was included which further explored the use of the Gauntlet’s digital compass and the pressure sensor located on the palm. Building on the paranormal theme, the players would occasionally have to track down and capture a poltergeist – a ghost emerging from painful memories, manifesting through noise or force, yet otherwise invisible. A poltergeist would be unleashed when the player attempted to access specific narrative fragments. These were few, but they were also the most revealing ones, retrieved by performing the appropriate gesture with an object. As part of the gameplay, the poltergeist encounters represent an additional obstacle in acquiring the most important pieces of the narrative puzzle, but also a variation on pacing through the sense of risk and urgency. Once a poltergeist was unleashed, the player had to move his/her hand around to find a specific direction where audio noise (from the PDA) and vibration (from the Gauntlet) were loudest/strongest. Once the direction of the poltergeist’s focus was found, the player would have to close

WORKS | 105 his/her hand in order to capture and “dispel” the harmful spirit. If the player failed to achieve this after a short time-span, the screen would become increasingly and permanently unreadable, as if burnt.

As a result, Noon did not explicitly signal “win” or “fail” conditions. The player was assumed to successfully completed the experience if he/she gathers enough information to puzzle together a cause for the Novak tragedy. On the other hand, difficulties in the timely capture of poltergeist resulted in text becoming increasingly obscured and eventually utterly unreadable; in which case the player was assumed to have failed his/her investigation.

The PDA was responsible for analysing the Gauntlet data, managing the game logic and audiovisual presentation. All of these had to be as simple as possible. Each memory was presented as a piece of text, preceded by the time of day and object it referred to (figure 8); and a sound queue (a single, reversed bell toll) was played as positive feedback when retrieving an additional memory by performing the correct gesture. In order to fit within the visual aesthetic and theme of the installation, the PDA was enclosed in a small book (figure 10). instance, by covering the PDA with a page made of highly translucent paper, or eventually using some form of e-paper, which would have greatly increases the technical requirements. Encasing the PDA within the book also prevented users from touching the screen or any of its side buttons, which would have toggled a “debug mode” or disrupted the application, respectively. On the downside, it also made it cumbersome to access the PDA for troubleshooting.

The first version of Noon (figure 9) was exhibited at the Campus 2.0 Neoanalog exhibition, part of the Ars Electronica Festival 2007; and during the Tag der offenen Tür (Open House Day) at the University of Arts and Industrial Design in Linz (April 11, 2008).

Second Version As work on a smaller Gauntlet was underway (see 5.1.2), an overhaul of the installation also took place. The main difference in this version was that a desktop computer and projector were used instead of a PDA.

Utilizing a PDA in the first Noon confirmed the feasibility of pairing the Gauntlet with a

106 | ART AND TECHNOLOGY AT PLAY Figure 9: First version of Noon, with observers trying to view the PDA's screen.

Figure 10: PDA encased in a book, used in the first Figure 11: Participant touches one of the objects. version.

WORKS | 107 Figure 12: Noon installation (second version).

Figure 13: The participant reads a memory from the Figure 14: The participant changes the time-period of snowglobe. memories.

108 | ART AND TECHNOLOGY AT PLAY mobile device for a more ubiquitous kind of use (such as an outdoors location-based game). Encasing the device in a book was also a nice touch, as it was seen to fit well within the fiction and visual aesthetic of Noon.

However, the PDA’s Bluetooth was prone to occasional disconnection, while participants were engaged. When this happened, the author (attending the installation) had to stop the activity, remove the PDA from its case, reset the Bluetooth connection (or even the PDA) and restart the Noon software. This was obviously disruptive and happened more often than deemed acceptable.

The PDA also needed to be charged frequently, roughly every 3 or 4 hours depending on use. While the Gauntlet’s batteries could be exchanged quickly, the PDA would instead be plugged in for charging. To avoid downtime, some participants were allowed to use the PDA without the book-like encasing and plugged to its charger.

Additionally, while the installation was designed for one participant at a time, it was observed that other visitors (such as a participant’s friends) might want to share in the story, and frequently bunched together to look at the PDA’s screen (figure 9).

As a result of these issues and observations, the second version of Noon was developed to run on a desktop computer, providing a more stable and capable computational support, as well as the possibility to project the story in a larger format, for all to see (figure 13). The whole software was rewritten for the Processing environment. Gesture recognition was improved, although no different gestures were introduced.

The content presentation had to be redesigned for a bigger format. This allowed the author to animate the clock transitions during time-shifts (figure 14), as well as the highlighting of text queues for gestures. Stock photos were used to complement the text. Due to time constraints, the presentation was nevertheless kept text-based and most elements static.

The objects remained the same, although all RFID tags had to be replaced, since the RFID module on the second Gauntlet used a different frequency than the first one. The pedestals which previously surrounded the table were also replaced with better- looking candle stands (figures 9 and 12).

This second version of Noon – A Secret Told by Objects was exhibited during the Lange Nacht der Forschung (Long Night of Research) at the University of Art and In-

WORKS | 109 dustrial Design Linz, on September 26, 2008. A low-fidelity demonstration of Noon was presented at the Creative Showcase of the International Conference on Advances in Computer Entertainment Technology (ACE ‘09) in 2009 (Martins et al. 2009). Taking place in the limited booth setting of a conference, the “demo” did not feature a round table or candles; and some heavy objects like the clock were replaced by lighter, smaller ones. In such a setting the immersive atmosphere of the installation was obviously missing. Fortunately, scientific conferences and other research-oriented events have become more accommodating towards the integrity of installations.

5.2.3 · Outcomes and Interpretation The installation Noon – A Secret Told by Objects was developed for showcasing the Gauntlet interface and as such also featured in a usability study, whose results were presented and published at the International Conference on Advances in Computer Entertainment Technology in 2008 (Martins et al. 2008b). While the study focussed on the Gauntlet, it did so in the context of the installation, with some findings pertinent to both the wearable as interface and the installation as a self-contained experience. As such, most of the following observations focus on the installation, whereas those more directly pertaining to the Gauntlet have been introduced above (see 5.1.3).

The installation illustrates the use of existing objects as digital interfaces and media, especially for immersive storytelling and information access. Participants found the installation pleasurable and easy to learn, with the usability study indicating consensus. There was occasional difficulty in successfully performing some of the object usage gestures even after being shown how to execute them, both during the formal tests as during public exhibition. As mentioned, the gesture detection was very basic and mechanistic.

In the way that it relies on tangible interfaces, Noon constitutes an approach to embodied interaction. The active use of objects through gestures resulted in a player performance which bystanders could understand and relate to. This constitutes an aspect of design for accountability, although it is of lesser consequence in this case than it would be in a collaborative activity, such as a hypothetical multiplayer version of the installation.

As opposed to envisioning a mostly immaterial future for technology and data access

110 | ART AND TECHNOLOGY AT PLAY (Bell & Gemmel 2009:31), Noon builds on some of the ways that artefacts are an essential component of human life and culture. Conception, fabrication and use of complex tools and artefacts is seen to have pro-actively developed together with the evolution of the hand, brain, language and heightened collaboration over the span of tens of thousands of years. As a result, humans are cognitively predisposed to deal with objects, which includes assessing their possible origin and intended use (see 3.3.2).

The permanency or resilience of artefacts is an essential affordance, which allows them to be perceived as a sedimentation or palimpsest of past events (Olsen 2010:108, see 3.3.3). In the ways they are inevitably tangled in human affairs, objects can become mnemonic of people, events and groups; or even be considered a form of remote agency or distributed personhood of their creators, owners or donors (Gell 1998, Fowler 2004, see 3.3.3). As objects are resilient, this role may certainly extend beyond an individual’s death (ibid.).

Accessing memories of another through a technical-magical ritual is admittedly science-fiction. Yet fiction can help to speculate upon future applications of technology, and their effect on everyday life (Dunne & Raby 2013). Should it be the case in a near- future that everyday objects are tracked – and recently rekindled enthusiasm about the “internet of things” any indicator – then it is certainly conceivable that artefacts will become a more integral part of a digitally-indexed historical process (Sterling 2005:11); and eventually provide forensic data or even constitute an aspect of an individual’s digital immortality (Bell & Gemmel 2009:17, 139).

Also present throughout Noon is the framing of technological augmentation as magic or “superpowers”. This is supported by the installation’s fiction, gameplay and visual aesthetic; but also strongly reinforced by the fact that the Gauntlet is worn by the participant and perceived as the source of power. This aspect has been previously discussed (see 5.1.3). Furthermore, framing technological applications and possibilities as magic is a form of understanding and eventual domestication. It provides metaphors for talking about technical means and processes in a way that is accessible to the layperson; and can lead to the refinement of existing technologies in social use (see 2.2.2, 2.5.2).

WORKS | 111 5.3 · Headbang Hero 2009 - 2011 With Ricardo Nascimento and Andreas Zingerle

Headbang Hero is a rhythm-based videogame where players score points by shaking their heads in time with heavy metal music. A motion-sensitive wig serves as game interface and prop, motivating performance. As achieving a high score requires repeated and violent movement of the head, the player’s health is put at risk through participation. Besides relying on an unusual wearable interface, Headbang Hero satir- izes the recent overuse of motion controls in videogames; and questions the player’s responsibility towards his/her health or physical integrity when engaged in what would otherwise be a harmless or inconsequential game.

5.3.1 · Background and Concept Between 2005 and 2006 Nintendo introduced the Wii, a videogame console with motion-sensitive game controllers. The previous generation of game consoles and controllers had included Sony’s EyeToy for the PlayStation 2, a camera-based tracking system used to capture players’ movements and likenesses, employed as a means of interacting with games (and strangely reminiscent of Myron Krueger’s pioneering work Videoplace). The Nintendo Wii console however, relied on a physical game controller held by the player, in ways resembling a TV remote and dubbed Wii Remote (or “wiimote” colloquially). Promotional videos showed players of different ages using the Wii Remote as a proxy of different in-game objects, swinging it like a baseball bat or a tennis racquet, for instance. This performative way of interacting with games revealed itself as immediately approachable for casual gamers, in contrast to the traditional and in ways more complex gamepad. As a result, the Wii connected with a much broader audience than its competitors. It quickly became the best-selling console of its generation (which included Sony’s PlayStation 3 and the Microsoft’s Xbox 360) and one of the best-selling consoles of all time (“List of best-selling,” n.d.). Sony would eventually mimic the Wii Remote with its own PlayStation Move controller for the Play-

112 | ART AND TECHNOLOGY AT PLAY Station 3.

Soon enough after the Wii’s launch, reports and videos started appearing of enthusi- astic players who, as result of their playful abandonment when performing wide and vigorous gestures, accidentally hit people and objects nearby, in some cases leading to minor injuries or broken equipment. Additionally, vigorous repetition of gestures was reported as the source of pain and other ailments on some players (a condition playfully dubbed as “wiitis” or “nintendinitis”).

As a subsequent add-on for the Wii, Nintendo released the Balance Board (something like a high-precision scale), with games and software oriented towards exercise and fitness, including physical training programmes, weight monitoring and yoga (Wii Fit).

The same generation of game consoles also saw the emergence of the games Guitar Hero and Rock Band as western pop-culture phenomena (and their eventual dismissal due to “franchise fatigue” after too many sequels). Guitar Hero, and later Rock Band, came bundled with a special game controller in the shape of an electrical guitar. The concept can be said to have originated in the game Guitar Freaks, released in Japan in 1998 but never transitioned to western markets (Forster & Freundorfer 2004:114). At its core, the gameplay consisted of the well-timed press of combinations of buttons on the guitar-shaped controller, in tandem with a song. The buttons were shaped and ar- ranged in a such a way as to convey some of the feeling of playing a guitar, and “playing Guitar Hero” quickly became more than playing a videogame, to include performing, in a theatrical and often over the top way, the role of a rock star, with associated gestures and mannerisms aplenty. While some gameplay mechanics such as tilting the guitar for extra points did encourage this type of performance, the game could as well have been played by pressing buttons on a more typical gamepad or alphanumeric keyboard. However, the tangibility of the guitar-shaped controller and the interaction it afforded within the game’s context conveyed a powerful sense of character, immers- ing players in the game’s fantasy and heightening their experience.

Critics of Guitar Hero saw the game merely as a reductionism of the nuanced art of guitar playing, equating it with timed button presses; and claimed that reductionism was further imparted to players, who were led to feel like skilled performers in the game’s context and as such discouraged from actually learning an instrument (Stuart 2009).

WORKS | 113 Headbang Hero was brainstormed as a humorous, playfully ironic reflection upon the interrelated issues introduced above: the cultural enthusiasm towards motion-controlled videogames; the possibility of associated health benefits as well as risks; the feeling of physical performance augmenting and even overwhelming the formal game mechanics; the question of how powerfully influencing a videogame can be, while seemingly innocuous.

5.3.2 · Design and Development In collaboration with fellow Interface Cultures alumni Andreas Zingerle and Ricardo Nascimento, the core concept emerged quite rapidly. While keeping the visual aesthetic and gameplay mechanic of popular rhythm games Guitar Hero and Rock Band, the player’s role would be intentionally simplified from that of a virtuoso guitar player, to become a headbanging concert-goer instead.

A further and eventually decisive source of inspiration was a seemingly serious study regarding the “risks of mild traumatic brain injury and neck injury associated with head banging” published in the British Medical Journal online (Patton & McIntosh 2008). The methodology employed by the researchers included attending “several hard rock and heavy metal concerts to find the most common style of head banging executed by audience members” (ibid.). The article concludes with advice to reduce the risk of injury associated with headbanging. Among the advice provided we may find the use of “personal protective equipment such as neck braces,” participation in a “formal training programme,” and an occasional change of listening habits to include artists like “Michael Bolton, Celine Dion, Enya, and Richard Clayderman” (ibid.).

The scientific tone of the study, including the presence of formulas and graphs applied to what may be seen as a subculture practice entailing a primitive, occasionally violent way to enjoy noisy music, prompted the inclusion in Headbang Hero of a “personal damage report”, printed out for the player at the end of each song (figure 21). Besides the player’s score, the report would also feature a measure of the damage incurred by playing the game, as well as advice borrowed from Patton and McIntosh’s article.

Prototype A prototype was developed by placing an accelerometer in a novelty wig and reading its data via a wired connection to an Arduino board (figure 15, left). The board relayed

114 | ART AND TECHNOLOGY AT PLAY data to a computer, running a basic program written in Processing. Starting from an existing example for beat detection of an audio track, the program was adapted to compare the timing of beats to the acceleration peaks from the sensor, and award points for good timing. As the concept was though extremely novel, the authors decided to publish a YouTube video as soon as possible, both to gather attention as well as to help establish original authorship of the idea (Martins 2009a). The video was also referred within a tip to the popular gaming culture website/outlet Kotaku, which quickly published a short article about it (Good 2009), eventually catching the attention of other gaming websites and blogs. Comments from the readers showed mixed reactions, with some applauding the idea and excited with the possibility of it becoming an actual product, and others manifesting scepticism or outright disapproval. It must be pointed out that Headbang Hero was never advertised as (or necessarily meant to be) a commercially available product. Rather, the ambiguity was a desirable factor, to allow for interpretation and provoke discussion (Dunne & Raby 2013:40).

Through replies to the publications on YouTube and Kotaku, the authors soon came to realize that a similar concept had been implemented in 2006 by students of the De- partment of Game Design of the Uppsala University in Sweden ("Head Banger" 2006). This earlier work dubbed “Head Banger” was a veritable homage to the rock/heavy- metal fan culture. It took the shape of an arcade videogame cabinet, with beer-hold- ers included for maximum convenience. More importantly, it also featured a wig as game controller, as well as sensors worn on the hands and wrist to incorporate hand gestures such as the “sign of the horns,” associated with rock and heavy metal culture. While this realization could have been discouraging (and initially was, to some degree), the authors nevertheless continued their work on Headbang Hero, focussing on the player’s heroic disregard for his/her own health as distinguishing critical aspect.

The wig was made wireless, with a hardware layout similar to that of the Gauntlet. As such the wig now included a 3-axis accelerometer connected to a small Arduino board, which read the accelerometer data and relayed it to a Bluetooth module. These were powered by two AAA 1.5 Volts batteries, connected to a “step-up” module to regulate and provide the necessary voltage (figure 15, right).

For the first in-house tests, the songs Eye of The Tiger by Survivor and Freak on a Leash by Korn were used due to their clear rhythmic structure, as a way to evaluate and

WORKS | 115 tune-up parameters of the beat detection algorithm. For public exhibition, however, music was sourced from heavy metal albums released under open licences, used for non-commercial purposes and under attribution. Three songs from three different bands, previously unknown by the authors, were chosen for the easy, medium and hard difficulty levels, respectively: Hiljaisina Hetkinä by Synestesia; Lowest Low by Slow Death Factory; and Take The Test by Severed Fifth.

The first public showcasing of Headbang Hero took place during the Dorkbot meeting in Linz (February 26, 2009), kindly organized by Mika Satomi. This version did not yet display a performance report. The wearable failed to work adequately during the presentation, due to apparent Bluetooth connectivity issues. The authors quickly replaced the Bluetooth module with a wired USB serial connection used in testing, and so attendants had the possibility to try the game out once all presentations were over.

First Version The authors finished implementing the temporary logging of data for individual play sessions, which was then used to compose, display and print out the performance report. A new wig was acquired, as the original frequently felt smaller than it should have (as merchandise from Disney’s Hannah Montana series, it was likely meant for younger wearers). Straps were also sewn onto the wig to help keep it in place during the violent head movements, fastened with snap buttons which were deemed fitting to the heavy metal aesthetic. Based on this version, the authors produced a short video introducing and demonstrating the gameplay mechanics, and once again published on YouTube (Martins 2009b).

This was very near a definitive version, in terms of gameplay, interaction design and presentation. The game was started manually by pressing the “space” key (an aspect that was soon to be changed, as we will detail below). A selection menu for the three difficulty settings was presented (figure 19). The player could navigate between choices by tilting the head left or right, upon which an excerpt of the corresponding song would play in a continuous loop; and confirm the selection with a forward shake of the head, thus starting the game proper. The in-game graphical user interface had a concert scene as background (figure 18). On the very centre was displayed the player’s score, in increasing font size as the score became higher. On the top left, the

116 | ART AND TECHNOLOGY AT PLAY name of the song, album and artist. The lower left showed three gauges, displaying as relative measures from left to right: the frequency of headbanging, the estimated amplitude of head swings and the accumulated damage. The damage, while an abstract measure, was calculated from the frequency and amplitude with reference to Patton & McIntosh’s study (2008). Taking up the remaining space right of the gauges was a scrolling graph, showing in real-time the measured total acceleration of the head as a green line, while behind it scrolled translucent blue rectangles representing the time-windows during which headbanging would score points (according to detected beats). Above the gauges and graph, a coloured equalizer bar indicated the player’s prowess, with the according title displayed above: miserable, weak, daring, powerful, heroic or epic. Headbanging prowess increased with the score but also de- creased with mistimed movements or periods of inactivity. Attaining higher prowess levels would cause the crowd to cheer, whereas a significant decrease would instead lead to booing. Spotlights in the background were animated according to beat detection. Tied to the player’s head shaking was the bobbing up and down of the background scene, with a parallax effect to convey some depth. Mistimed movements were signalled by electrical sparks projected from the spotlight sources, with an accompanying sound.

Points were awarded based on the vigorousness of well-timed head movements. On the other hand, if the player’s prowess level dropped below “miserable” or an extended period of inactivity was detected, the song would stop, the crowd would start booing and the message “you suck” would be displayed, after which the game would reset itself.

During actual game play, vigorous head movements and swirling hair made it quite difficult for the player to keep track of what was on the screen. As such, the visual interface provided more of a complementing background for the player’s performance when observed by others, than information directed at the player him-/herself. This was reported as encouraging players to feel the performance and be spontaneous instead of focussing on game-logic and “eye-candy”. Therefore, the audio queues signalling gameplay progression became the primary channel of feedback to the player and were made more audible.

If the player made it to the end of the song, the crowd would cheer. The player’s “per-

WORKS | 117 sonal damage report” was then rendered onto the screen and sent to a printer. From the very first version, the report displayed the total score, as well as graphs of the acceleration, frequency and amplitude of movements and resulting damage over time. It also offered pieces of advice, composed (randomly in part) from a list which included those of Patton & McIntosh’s study (2008) as well as tongue-in-cheek items such as "always listen to what your parents have to say about heavy music", "learn to play a cool instrument such as a flute, sitar or even a shamisen" and "play Headbang Hero for 10 minutes every day”.

Initial Improvements Shortly after, Headbang Hero was showcased during the Tag der offenen Tür (Open House Day) at the University of Arts and Industrial Design in Linz (April 24, 2009). Sim- ilar issues occurred as before, which were initially attributed to Bluetooth connection problems. It was later found that the problem was caused by more than one broken wire in the circuit, which led to occasional disconnections and resetting of the micro- controller. As electronic components and cabling were not supported by any rigid structure, they were subject to mechanical stress. To address this issue and further provide overall stability and durability, the inner harness of a hard hat (or construction helmet) was used as support for the wig. A lining was added to cover the harness, so that it might rest more comfortably on the head, and avoid tangling with the wearer’s hair. The harness was placed between wig and lining in a way that the textile elements could not be separated or the electronics accessed without first breaking the stitching. For convenience, the batteries could be accessed via a hidden fold in the back, fastened with hook-and-loop binding (also known under the manufacturer’s name velcro).

Some participants realized that the game could be “cheated” by simply holding the wig in the hand and shaking it, thus avoiding the tiring performance. As the means to cir- cumvent this, a pressure sensor was added to the inside of the wig, to detect whether it was actually worn and tightly fastened. If the pressure was too low, the game would pause and prompt the player to fasten the wearable, thus avoiding cheating but also giving the player time to adjust the wearable when it occasionally slipped or flew off. This also allowed the game to be started automatically (without manually pressing a key) once a player had fastened the wearable.

118 | ART AND TECHNOLOGY AT PLAY Figure 15: Hardware prototypes for Headbang Hero.

Figure 17: Headbang Hero disclaimer.

Figure 16: Headbang Hero starting screen.

Figure 19: Headbang Hero difficulty selection. Figure 18: Headbang Hero gameplay GUI.

WORKS | 119 Figure 20: Headbang Hero (second version) at the Festival Aucard de Figure 21: Headbang Hero personal Tours. Damage Report.

Figure 22: Player-created score board. Figure 23: Children playing Headbang Hero at the Campus Party Mexico.

Figure 24: Heabang Hero wig, with on/off snap button.

120 | ART AND TECHNOLOGY AT PLAY The songs for easy and medium levels were deemed too long. Most players choosing an easy level did not wish to play through a 3-minutes long track; and felt it unfair to be “booed” and derided by the game once they decided to stop. As such, the tracks were edited to be shorter, especially the one for the easiest setting.

As a further improvement, a disclaimer was added, displayed once the wig was worn and tightly fastened (figure 17, an old version). This was admittedly more for rhetoric than legal reasons, as the disclaimer’s wording and the way in which players were to signal their acceptance were of questionable seriousness:

“When playing Headbang Hero you incur the risk of real physical damage, particularly the occurrence of mild traumatic neck and head injury. The practice of headbanging is strongly disapproved of by medical researchers and concerned parents alike. If you still wish to play Headbang Hero you im- plicitly acknowledge that you do so at your own risk!”

Followed by:

“To accept, please shake your head violently”.

This version of Headbang Hero was first shown at the Speculum Artium Festival, taking place during May of 2009 in Trbovlje, Slovenia. It also featured in The Royal Inter- face Cultures Masquerade Ball, part of the Ars Electronica Festival (September 2009); both the Mexican and Brazilian editions of the Campus Party (November 2009 and January 2010, respectively); and the Creative Showcase of the International Confer- ence on Advances in Computer Entertainment Technology (ACE ‘09) in 2009 (Martins et al. 2009a) where also Noon – A Secret Told by Objects was presented.

This same version of Headbang Hero would also be distinguished as Jury Recommen- ded Work (Entertainment Division) for the 13th edition of the Japan Media Arts Festival (February 2010).

Further Improvements Even with structural support from a hard hat’s harness, the wearable interface would occasionally need repairs, as it was understandably subject to frequent (and encouraged) violent use. In the complete body of the author’s work, the Headbang Hero interface was by far the piece of equipment most often repaired.

WORKS | 121 To mitigate downtime due to technical issues, a second interface was produced, using the same components, materials and structure. This would allow quick replacements during events should the first one break.

The in-game visual elements were also subject to an overhaul (figure 20). The gauges were replaced with shinier chrome versions, and the “damage” gauge given a more prominent size. Amplifier speakers with the Headbang Hero logo were added to the sides of the acceleration/beats graph. The prowess indicator became vertical, with fewer steps, and placed in between the damage gauge and one the leftmost speaker, with the corresponding title (as before, “miserable” to “epic”) indicated in an amplifier head. The layout of the damage report was also significantly improved, with added details taking better advantage of the space within a page (figure 21).

The revised version was exhibited at the music festival Aucard de Tours (June 2010 in Tours, France). The eager engagement of festival goers (and their disregard for their own safety) is observable in the footage captured at the event and published by the authors online (Zingerle 2010).

Later the same year, one of the wearables was entrusted to the organization of the Festival de Arte Digital in Belo Horizonte (Brazil, September 2010). The wearable was shipped back to the authors in very bad condition, with the supporting harness not only partially disassembled but even broken in certain places. While it was never thought that the harness might actually break, even during the most violent usage, the fact is that harder materials tend to break, whereas softer ones can bend, fold or even stretch. As such, a further version of the wearable controller was developed, essentially replacing the hard hat harness with a custom-made flexible black leather cap. Almost on top of the leather cap’s inside was a small foam-padded pocket, housing all the electronics except the battery case, which was located towards the back of the head, in an extension of the pocket with easier access. The electronics were supported by an adequately stable yet light synthetic modelling clay.

An on/off switch and status light were also included. The switch was constructed with resort to a metallic snap button (a technique which would be used also in the work Rambler) and the status light, a small red LED, basically indicated whether the circuit was powered (figure 24).

122 | ART AND TECHNOLOGY AT PLAY Only one such exemplar of the wearable interface was produced, yet it would feature in the Playface Intercult exhibition at the MuseumsQuartier in Vienna (April/May 2011) and later at the Kiblix/MFRU festival in Maribor (November 2013) without needing any repairs.

5.3.3 · Outcomes and Interpretation The concept of using a wig as game controller was initially assumed to be one of the most original and innovative aspects to Headbang Hero. As explained above, this was soon understood not to be the case. Even so, it remained a novel concept to hundreds of players reached through presence at several events, as well as journalists and readers throughout the world. Novelty by itself may not translate into any kind of usefulness, unless it constitutes an improvement or assists in acquiring new perspectives.

Interestingly enough a patent application was submitted in May 2013 by Sony Corpor- ation (the makers of the PlayStation family of videogame consoles, among other consumer electronics) for a “SmartWig” described as a:

“Wearable computing device, comprising a wig [...] at least one sensor for providing input data, a processing unit that is coupled to the at least one sensor [...] and a communication interface [...] for communicating with a second computing device.” (Tobita 2013)

The patent application does not specifically mention any kind of game-related use. It does, however, contain peculiar (and fastidiously detailed) examples of potential applications. These range from using the wig as a haptic feedback device to aid visually- impaired users in navigation, to controlling a slide presentation via buttons embedded into sideburns (ibid.).

Within the context of Headbang Hero, the motion-sensing wig serves as more than mediator of human-computer interaction. Like the Guitar Hero game controller, it facilitates stepping into a role and invites an expressive bodily behaviour. As interaction design, it caters to the pleasure inherent to playful performance (Frasca 2007). Be- sides the agonistic struggle for a high score through the formal rules of the game, the player’s reward can also be found intrinsically and viscerally through engagement (whether cognitive or sensorial), as in Caillois’ category of ilinx or vertigo (Caillois 2001).

WORKS | 123 Furthermore, observing the participant’s display can certainly trigger our natural predisposition to engage in performing with others, seen as a form of collective expression and socially-constructive cohesion (Boyd 2010). While it was quite common for visitors or passersby to gather around Headbang Hero merely to enjoy the ongoing show, it was also occasionally observed that individuals accompanying the player would join in the performance – although this was of no consequence to the gameplay mechanics. This type of collaboration (and occasional physical interaction) is clearly present in the footage from the festival Aucard de Tours (Zingerle 2010).

Similarly to what happened with the Gauntlet, the fact that the game controller is worn imparts a transfer of qualities and abilities, in collusion with the role of wigs as embellishment or costume, facilitating getting into character or assuming a role (Laver 2012:18, 127, 130). Furthermore, it brings physical realism in the weight and displacement of hair. Having had long hair once and attended his fair share of heavy-metal concerts, the author can attest that headbanging is not nearly as motivating without flowing locks.

Showcasing wearable interfaces can be especially daunting, something which had already been experienced with the Gauntlet and Noon, but became especially prevalent with Headbang Hero. As a small team within an experimental context, it is extremely difficult to build a wearable interface that addresses all aspects of its use qua wearable.

In being worn by several attendants throughout, wearables are especially prone to wear-and-tear and eventually become dirty. The shape, choice of materials and fastening must account for the device being worn and taken off frequently (and potentially used during warm weather) and as such it should be kept clean. It can hardly be expected that a wearable interface will retain a pristine condition, especially when it is use implies contact with the wearer’s skin and performing strenuous gestures.

A wig is certainly one of the hardest to maintain, as it becomes dirty and clumped really quick, especially when used to headbang. While the material for the inner lining was chosen for being easy to clean, the wig itself was always a problem, having been completely replaced several times throughout the history of Headbang Hero.

In the way that it mirrors, recombines and playfully questions pop-culture phenom-

124 | ART AND TECHNOLOGY AT PLAY ena, Headbang Hero enacts postmodern artistic expression. It appropriates the format and rhetoric of contemporary videogames, posing perhaps as something innocuous, inconsequential or harmless. We have seen that play and games are commonly characterized as conceptually safe spaces for experimentation, apart from everyday life (see 4.2). In contrast, a Headbang Hero player is risking bodily harm in proportion with the level of engagement. This in itself is not something new within an artistic context, as the artwork PainStation by artist collective //////////fur//// has illustrated, perhaps notoriously so. The PainStation inflicted whippings, burns and shocks on player’s hands as negative feedback for a classic game of Pong, resulting among other things in “bleeding hands and the risk of infections” (“From Art to Arcade,” n.d.).

Headbang Hero does not use a technical feedback process to cause injury. Instead, injury is directly caused by the player’s own reckless abandonment to the performance. During longer exhibition periods, players frequently reported subsequent headaches and neck pain to the authors (usually the day after), although they were advised beforehand. Quite often, though, the very same individuals came back to play once more. Such is the power of games (often allied to the recklessness of youth). There- fore, instead of being faced as innocuous and inconsequential, games should perhaps be approached in knowing their immanent power to captivate and make us forget our ailments – for better and for worse.

The printed “damage reports” were also an interesting choice. In a sense, they can be seen to refer to coin-operated fortune-teller machines which read the user’s heartbeat or palm and produce a printed report. These can often be found at arcades, amuse- ment parks and shopping centres.

It was observed that most players chose to keep their printed reports, at least while at the venue. Each report was personal and unique, especially as it contained the graph plotting of the whole performance. These plottings could be visually understood and compared to some extent: more frequent and violent head movements produced more thickly-filled sections of graph, to the point where the graph’s finer details became irrelevant (figure 21). The report could also have served as advertising material, as players would later show it to their friends.

Perhaps the most interesting aspect regarding the printed reports was that they represented a piece of the game with which players could do whatever they liked, eventu-

WORKS | 125 ally enabling meta-game activities. At the Campus Party Brazil (January 2010) some participants did not take their reports with them, but rather signed (or otherwise identified) them and posted it on the installation’s info panel (figure 22). This behaviour was started by a few, but was quickly perceived as a norm. As such the staff tending to the installation improvised a “top players” signboard, trimmed the posted reports and eventually separated them into difficulty levels. Since the game itself did not keep and display a scoreboard, players authored this aspect themselves by physically arranging an artefact obtained as a result of playing the game.

126 | ART AND TECHNOLOGY AT PLAY 5.4 · Rambler 2009 – 2010 With Ricardo Nascimento

Rambler introduces a pair of sneakers that take microblogging one step further – quite literally so – by posting the wearer’s steps on Twitter. Messages are comprised of repetitions of the elements “tap” and “.” respectively symbolizing the wearer’s steps and time in between these.

The work promotes embodied interaction with online social networking services, connecting natural behaviour, expression and identity through clothing, and the socially- oriented use of digital technology.

At the same time, Rambler is a critical take on compulsive microblogging habits and elicits reflection on the personal nature, amount and usefulness of information generated everyday through online platforms such as Twitter and Facebook.

5.4.1 · Background and Concept Rambler was born from a collaboration with fellow Interface Cultures alumnus Ri- cardo Nascimento, who was also one of the authors of Headbang Hero. A common interest in addressing social networks, especially Twitter, prompted the joint development of a functionally straightforward, innovative and expressive work.

From the outset, the authors considered the possibility of an artefact which interacted with the Twitter platform. This could potentially represent a more tangible alternative to smartphones and computers when interacting with online social networking.

While considering the possible dialectic between one or more artefacts and the social networking service, the seemingly compulsive behaviour of some Twitter users was identified as a potentially interesting aspect to explore as part of the work’s rhetoric and aesthetics.

In 2009 a study was conducted in the U.S. by Pear Analytics, with the goal of determining the types of content posted by Twitter users (“Twitter Study,” 2009). The initial hy-

WORKS | 127 pothesis “was that Twitter was being used predominantly for self-promotion” with a majority of tweets “trying to push a product, service or [...] a distinct ‘Twitter only offer’ of some kind” (ibid.). What the study revealed instead was that the dominant content amounted to “pointless babble”, e.g. tweets such as “I am eating a sandwich now” (ibid.). “Pointless babble” was found to represent 40.55% of content on Twitter, followed by “conversational” messages, representing 37.55%. The remaining categories had each less than 9% of weight; and interestingly enough, there was slightly more spam than news (ibid.).

The authors arrived at the concept of pair of shoes which posted the wearer’s every step on Twitter, embodying an extremely compulsive microblogging behaviour as an automatic, thoughtless act of diffusing large amounts of repetitive and arguably useless personal information. The choice of which specific type of footwear to use fell quite naturally on sneakers, for their adequacy as accessories to the young, easy-going and trendy demographics associated with Twitter (“Twitter Study,” 2009).

The name Rambler was chosen for the double meaning of the word “ramble”. On one hand, to ramble is to take a pleasant, even aimless walk; on the other, it is also to di- gress when talking, losing clarity or even turning aside of the main subject.

5.4.2 · Design and Development As in the case of Headbang Hero, the authors were eager to publish a concept video with a working prototype, to assert the authorship of the potentially innovative concept and to gather attention from the media, since Twitter was somewhat of a hot topic at the time. Although a fully-functional version would only be advertised later, work on the prototype started in late 2009.

Prototype The prototype was developed to confirm technical feasibility and anticipate technical issues or limitations. Instead of starting work on a pair of sneakers with wirelessly interconnected embedded electronics, the authors implemented a single wireless wearable pressure-sensing module.

The system layout followed an already familiar configuration, similar to the Headbang Hero wig and the Gauntlet. It consisted of a force-sensitive resistor connected to an

128 | ART AND TECHNOLOGY AT PLAY Arduino Pro Mini board, which read the sensor’s value and sent it to a Bluetooth module for wireless transmission. The components were powered by two AAA 1.5 Volt batteries connected to a step-up (figure 26 shows a later, more complete version).

Using the (now extinct) Mobile Processing library, an application was developed for the Nokia N95 phone (kindly offered to the author by Nokia in the aftermath of a previous project Wolves & Sheep, together with Thomas Wagner). The application accessed the phone’s Bluetooth services to connect with and obtain sensor data from the wearable module. The data would be analysed to detect steps and eventually to compose a message, by appending “tap” upon each individual step, and “.” for each increasing span of time after (figure 28). The messages were purposefully kept simple.

The adequacy of the solution (including its mechanical robustness) was tested by placing the force-sensitive resistor under the insole of one of the author’s own sneakers, with the remaining electronics fastened to the ankle, and observing the resulting behaviour on the prototype application running on a phone, throughout specific dérives as well as daily activities.

This allowed also to fine-tune the detection of steps from raw data, and to observe that posting the idle symbol “.” at a constant pace tended to produce a lot of data even during periods of inactivity. As such, idle symbols would be added at an increasingly slow rate, so that at the end each subsequent dot represents a longer period of inactivity. Upon further tests the authors could determine with satisfaction that even though text was composed using the same two elements, there was indeed an appre- ciable amount of variation, overall resembling more a discursive morse-code message than the regular pattern of a percussion score.

The following step was to create an account on Twitter used for testing, and register as a developer for the Twitter API, which was relatively straightforward. Then, the mobile phone application was updated to post messages on Twitter, using Mobile Pro- cessing’s REST API.

With the essential elements in place and tested, the authors began working towards a complete fully-functional version.

First Version Ideally, Rambler would consist solely of a pair of sneakers. In such a scenario, at least

WORKS | 129 one of the sneakers would have to wirelessly connect to the internet – either through Wi-Fi hotspots or cellular networks. This would imply more hardware and logic embedded into a single sneaker, which would require more space and would have been harder to configure (for instance, change the pressure threshold for step detection) and troubleshoot (for instance, assess network connection status or access error messages received from the Twitter API). As such, the mobile phone was kept, as a more accessible way to handle these aspects (figure 25).

To avoid overloading the phone’s Bluetooth connection, it was preferred that only one of the sneakers would connect to the phone. The remaining sneaker would relay its pressure readings to its counterpart via a different radio-frequency transceiver. For this purpose, a pair of modules based on Nordic’s Nrf2401 2.4GHz radio transceiver was chosen, for its technical adequacy and the availability of a corresponding library and example for the Arduino environment, facilitating its quick inclusion (figure 26).

A pair of sneakers from a small brand were chosen and acquired in the European shoe size 44, with foresight towards accommodating different wearers. While a shoe can be made to fit a smaller foot size more comfortably with the addition of padding, providing a fit for a larger foot is also a larger problem.

The electronic components were secured to each other by the rigid headers used to connect the through-holes, and wood pieces were used as spacers, with the exception of the pressure sensor and battery case. These were connected with flexible wire. The pressure sensor was attached under the removable insole. The remaining components were placed in the sneaker’s tongue after it was cut open and hollowed out from the foam padding. Each circuit had a small on/off switch, and the tongue was kept closed by a zipper. Foam shapes were soon added for padding and protection from physical stress.

The Rambler text logo was designed using a font and colour scheme similar to those of Twitter – at the time, at least (figures 29 and 30). Twitter’s “bird” logo in particular has seen several redesigns over the years. For Rambler, the authors took the original bird shape (which featured legs) and added a pair of sneakers in orange, which became part of the colour scheme.

The logos (text and bird) were printed as stickers. For the text, each letter was indi-

130 | ART AND TECHNOLOGY AT PLAY vidually cut and placed by hand, which made them harder to properly align. As such, the logo used in the sneakers was redesigned to have a white border, resulting in a single sticker which was also more readable from a distance (figure 29).

The mobile application’s interface was initially very simple, and used more for monitoring (figure 28). It displayed the current message while it was being composed, the state of connection with the wearable, the destination Twitter account (as we frequently switched between the actual “ramblershoes” account and the one used for development and testing) and the amount of characters left (within Twitter’s 140-character limit) until the message was posted. Some consideration was put in the way that the text was split, to avoid breaking a “tap”.

Based on this version of Rambler, a short video was produced, and posted online (Nascimento 2010). The video would soon feature in several online news outlets and blogs, catching the attention of the Portuguese magazine Sábado, who published an interview with the authors (Pereirinha 2010, see Appendix C)

Like Headbang Hero, Rambler was never advertised as a commercially available product, and the tone used in the video even slightly sarcastic. Soon enough, though, the authors would receive emails inquiring about price and availability – which were answered with the explanation that Rambler was still a prototype, and could possibly become a product someday.

Rambler was first presented at the Playful Interface Cultures exhibition, part of the Ars Electronica Festival, taking place during September of 2010 in Linz. The day before the opening significant changes were made to the Twitter API, especially in regards to au- thentication methods. These had been planned and advertised months before, yet the emails sent to the developer account associated with Rambler went unnoticed by the authors. The Rambler application was modified overnight to use the updated Twitter API ME library, developed by Ernandes Mourao Jr. for the (now extinct) Java ME platform.

During exhibitions and other events, the Rambler sneakers would be worn by one of the authors or a performer and were seldom in the same place for long. As an installation, an empty pedestal with the sneakers’ sole prints displayed on top would highlight their absence. This was used to reinforce the necessarily performative nature of

WORKS | 131 the work. Rambler was made meaningful in the actualization of a process of everyday use, a process which generates data which cannot be dissociated from the specific performance of wearing the sneakers like one normally would. Just like the boots in Nancy Sinatra’s 1966 hit, the Rambler sneakers were made for walking, and that’s just what they did. Visitors were encouraged to follow the ramblershoes Twitter account (displayed also on a screen at the exhibition; see figure 30) as well as to spot them out in use throughout the event.

Further Improvements After the Ars Electronica festival, a few improvements were made. To facilitate switching the electronics on or off, the circuit was connected to a metal snap button on the outside of the sneaker’s tongue (figure 27), effectively working as a switch – a technique that was used in the latest version of the Headbang Hero wig (figure 24). Be- sides closing the electrical circuit, the snap button also secured a flap which provided access to the batteries when open. The remaining modules were placed within an inner pocket inside the tongue, and holes drilled on the back of the tongue (facing the ankle) so that the power and status LEDs could be seen when needed, while otherwise remaining mostly concealed.

The mobile application was redesigned to have a black background, with the currently composed message in white text (figure 28, right). The top left had an indicator of the Bluetooth connection status, while on the top right a blue bird would be displayed while a tweet was being posted, and a red exclamation mark added if the posting failed. The bottom displayed two vertical orange bars indicating the pressure on each sensor (an abstract measure). The threshold above which a “tap” was added was indicated independently for each foot as a grey horizontal line. This could be adjusted using the phone’s numerical keypad.

Rambler was subsequently featured at several events, including the Amber Festival (Istanbul, November 2010), Transmediale 11 (Berlin, February 2011) and the Enter Festival (Prague, April 2011).

For the first edition of the Wear It Festival (Berlin, October 2014), the mobile application was ported to run on Android device, mostly retaining the same layout.

132 | ART AND TECHNOLOGY AT PLAY Figure 25: Rambler technical overview.

Figure 26: Rambler hardware schematic, left and right sneakers.

Figure 27: Rambler sneaker showing on/off snap button. Figure 28: Rambler app. Left: first version. Right: Second version (emulator screenshot).

WORKS | 133 Figure 29: One of the Rambler sneakers, with bird logo and text with outline.

Figure 30: Rambler Twitter page (@ramblershoes). Captured 11.03.2010

134 | ART AND TECHNOLOGY AT PLAY 5.4.3 · Outcomes and Interpretation Throughout the last decade online social networks have become an increasingly popular means of mass communication and expression. Their presence is so ubiquitous that governments are starting to voice serious concern about their use as platforms for promoting extremist ideologies, fake news and other forms of misinformation – especially since the later were suggested to have influences the U. S. presidential election (Connoly et al. 2016). Facebook, in particular, has become a powerful entity, not merely in terms of economical capital but more so in social capital, with 1.18 billion users creating, circulating and consuming content on a daily basis (average for September 2016, see “Company Info,” n.d.).

Any given computational system ultimately refers to the material word (Dourish 2004:137). While relying on digital technology and the accompanying computational abstractions, online social networks especially refer to the socio-material world of human affairs and relationships. The underlying digital logic can be said to impose the usage of fixed terms and discontinuous categories when dealing with highly relational, emotional or subjective aspects of human social experience such as “friend”, “like” or “follow”. Facebook’s “like” button (a highly recognizable symbol) was only recently expanded to allow for the expression of other emotional responses to content. Yet it still frames human reactions (even if only immediate, basic ones) as a limited set of six discrete and mutually exclusive options: “like”, “love”, “haha”, “wow”, “sad” and “angry”. Equally poignant is the way we interact with online social networks. Regardless of whether we are searching for an old schoolmate, delighting at the self-referentiality of the latest meme, sending support to a friend going through turbulent times or posting a selfie while “checking-in” at the most exclusive venue in town – we are tapping on a touchscreen, reaching out to thousands by abducting our attention into a few square centimetres of digital real estate.

As a wearable interface, Rambler illustrates a conceptual way of embodied interaction with online social networks which is grounded in human motor behaviour. While it does so within an extreme scenario of use (i.e. posting the user’s every single step) constructed as social comment and critique, it creates a connection between an online space for expression and the world of human experience to which it ultimately refers. A close precedent can be found in the work Seven Mile Boots by Beloff, Berger

WORKS | 135 and Pichlmair, which enabled the wearer to listen to and navigate between online chats by walking around in real spaces ("Seven Mile Boots," n.d.; cf. Pichlmair 2005).

Rambler, however, does not access online content through performance but rather generates it – and it does so in considerable amounts. Even if it is invariably unique and personal, the message remains repetitive and egocentric – regardless of whether the wearer is walking to work, meeting a lover, dancing at a party or taking part in a funeral procession. In this way, while presenting an alternative to interaction with online services, it simultaneously reflects a contextually detached attitude.

Although Rambler was not explicitly presented either as a product or as an artwork, one might tentatively assume that posting every single step on Twitter would be easily perceived as a rhetorical exaggeration belying an agenda of social commentary. While this was the case at art venues – where visitors come prepared to encounter some form of postmodern discourse – online reactions were mixed and often polarized, as expected. Writing about the project, a contributor of the popular technology news outlet Gizmodo seems to simultaneously understand the aspect of critique and feel dis- appointed at the way it is achieved:

“The creators of Rambler think that Twitter's a pretty vapid service filled with useless information. So they went out and tweeted lots of insightful things! No, of course not. They just created the most worthless Twitter account of all time. [...] Rambler throws tainted meat on a big steaming pile of garbage and calls it ‘commentary.’ Next time just clean the garbage up, people!” (Barrett 2010)

As a reply to the article, user Yerzriknot writes that “anyone who's stupid enough to buy a pair of these abominations needs to be sterilized” (ibid.). Replies such as Yerzriknot’s were, of course, all too common in online contexts – more so if an article already introduced the concept with derision, as was Gizmodo’s case.

Many others, however, manifested their enthusiasm, either from (apparent) genuine interest in Rambler as a product or from understanding it as a form of critique through design. As a result, the authors received a few e-mails asking for price and availability, and were eventually in dialogue with a small company interested in developing Rambler as a product, although this did not come to pass.

136 | ART AND TECHNOLOGY AT PLAY This polarization of reactions and opinions is taken as an indication of effectiveness in Rambler’s overall rhetoric, as it sparked individual reflection and collective discussion upon contemporary practices and values. While referring to the sneakers as a product, the author of an article for a geek culture website writes:

“This concept reminds one of the radio collar one puts on animals in the wild to track their every moment. So if one is indeed a social animal who would want the world to know when one walks, stops or dances then put on these Rambler shoes and let them ramble on the Twitter.” (Salazar 2010)

In being appreciated as a potential product which encourages exaggeration by means of a thoughtless act, Rambler is critical design upon the values expressed through the use of social networking services, questioning among other things how much of their popularity is due to their being a useful means of expression, and how much of it arises through postmodern mystification and hype – much like a pair of trendy sneakers.

WORKS | 137 5.5 · Weltschmerz 2011 With Maša Jazbec

Weltschmerz (world-pain or world-weariness) is an installation which prompts the visitor to play Russian roulette with a violent news channel and manifests subsequent physiological reactions on a glistening piece of red, pulsating flesh-like aggregation. The visitor is told to sit on a chair, pick up a revolver, spin the cylinder, put the gun to his/her head and pull the trigger. This effectively “tunes in” the Weltschmerz TV channel which, in spite of being fictive, plays real graphic footage related to violence, human and animal rights.

5.5.1 · Background and Concept The installation was born from a collaboration with Interface Cultures alumna Maša Jazbec, as a way to surface and materialize an ever-present feeling of weariness brought about by the everyday contact with news media.

As such, the work’s biggest motivation and source of inspiration was the authors’ own experience with television, news media and film, especially in the way these graphic- ally mediate forms of violence and cruelty beyond one’s immediate surroundings and social environment.

Inspirational Sources Weltschmerz was informed by the work of philosopher and cultural critic Slavoj Žižek (Žižek 2008), examining the violence inherent to cultural, political and economic systems; and the ways that some forms of violence – subjective, objective or systemic – occlude each other by de-sensitizing our perception or drawing our attention.

Film media also feature among the inspirational sources. Stanley Kubrick’s A Clockwork Orange (1971) depicted the rehabilitation of a violent individual in an ironically violent way, through the use of audiovisual technology within a process of biofeedback. The portrayal of violence and brutality present throughout director Michael Haneke’s

138 | ART AND TECHNOLOGY AT PLAY filmography also motivated the work’s theme, even if less specifically or to a lesser extent. When asked about the recurrence of violent themes in his ouvre, Haneke replies:

“It's simply that violence is a part of our society. [...] But I don't understand why I'm always categorized as a specialist for violence. I deal with lots of social issues, like the question of media in our society. Personally, I can't stand violence. In any standard American mainstream movie, there's 20 times more violence than in any one of my films, so I don't know why those directors aren't asked why they're such specialists for violence.” (Haneke in Bain 2009)

The aesthetics were influenced to some degree by previous works from each author, but more prevalently by the cinematic work of directors David Lynch and David Cronenberg; especially the latter’s science-fiction film Videodrome (1983). The fascinat- ingly bizarre film includes an extreme portrayal of the use of audiovisual mass media for ideological dominance, punctuated by disturbing, sexually charged scenes combining technical apparatuses and living flesh.

Concept Evolution The initial proposal was to visibly manifest the participant’s reaction to real footage of violent events, more specifically by monitoring physiological signs – such as heart rate, skin conductance or respiratory patterns – as a sort of experimental biofeedback process. This led to pondering upon methods for collecting different kinds of physiological data in a way that was either not intrusive, or intrusive in a way that would fit the work’s aesthetics and interaction design.

Additionally, the authors identified a procedural parallel between the practice of zapping (that is, frequently switching channels by means of a remote control) and Russian roulette: a deadly game of chance played with resort to a revolver loaded with a single bullet. Each player takes turns in spinning the gun’s cylinder, pointing at oneself and pulling the trigger. The similarity can be found in the way that zapping is also a repetitive behaviour with an element of chance that can leads to the occasional dose of violence. As a result, it was established that the participant would be invited to incur in a mix of both activities.

Adding to the discomforting context and overall aesthetics, something resembling a

WORKS | 139 piece of raw, even bloody, wet meat would be placed between the participant and the screen or TV set, with a behaviour connected to the measured signals (figure 32). The meat could be interpreted from almost opposite perspectives: as manifesting and even reinforcing the participant’s reactions; or as a filter placed between the participant and the violent images to dissipate emotions.

5.5.2 · Design and Development The artwork’s layout and interaction were at this point mostly settled. The installation was meant to resemble part of a living room setting (figures 31 and 33). It consisted of a chair for the participant to sit on, in front of which was a table. On the table were the revolver and the meat-like aggregation (figure 32). Beyond the table was a TV set or screen. The screen would prompt the visitor to sit down, pick up the revolver, spin the cylinder, put it to his/head head and pull the trigger (figure 35). Once the trigger was pulled, a gunshot sound would be heard, and a randomly chosen video would play, while the participant’s condition would be measured and translated to the motions of the lump of meat (figure 34). Once the video was over, the installation would reset.

The authors chose to measure the participant’s skin conductance or galvanic skin response (GSR), since his/her palm would be in contact with the gun’s handle. There wasn’t a way to ensure this would always be the case – although it was later observed that individuals often kept holding the gun, sometimes even to the temple, while watching a video. As such, pressure sensors within the chair’s cushion were employed for detecting the participant’s potential discomfort through shifting, but also allowed to detect sitting down and getting up, in order to trigger the installation’s life-cycle. Likewise, the GSR sensor also made it possible to know if the gun was being held. A small reflectance sensor (commonly used in line-following robots) was placed within the gun to detect pulling of the trigger, and another to detect the spinning of the cylinder. A circular metal piece placed at the end of the barrel was used as capacitive sensor, to detect placement against the skin. This could also be triggered with any other part of the body, and would be harder to trigger if something got in between, such as clump of hair or a hat, but a vast majority of observed participants did their best to follow the on-screen prompts.

The clump of flesh was built having textile as a base, with silicone poured over, as well

140 | ART AND TECHNOLOGY AT PLAY Figure 31: Weltschmerz installation with projection at Speculum Artium 2011.

Figure 32: Meat-like aggregation and revolver.

WORKS | 141 Figure 33: Weltschmerz installation with TV screen at Ars Electronica 2011.

Figure 34: Weltschmerz participant and observer. Figure 35: Participant performing Russian roulette.

142 | ART AND TECHNOLOGY AT PLAY as other mixed materials. Dark red paraffin was occasionally applied during exhibition to keep it glistening. Under the mountain of flesh were servo motors to produce motion.

The sensors on the gun and actuators within the flesh were connected to an Arduino board. The board was in turn connected to a computer, running software written using openFrameworks. The software analysed the sensor data, first to determine which prompt to show on the screen (that is, depending on whether the participant was sitting down, holding the gun, etc.) and then to control the frequency and speed of the pulsating red meat (via the Arduino). It also randomly selected and displayed one of four videos: the violent handling and killing of cows with a captive bolt gun; the hunting and killing of seals through repeated clubbing and striking with picks; trash being dumped close to children foraging throughout for plastic; and people who barely escaped the Srebrenica massacre, including children. The software also overlaid the “Weltschmerz Channel” logo (a spinning revolver cylinder with a single chambered planet Earth) and an element clearly imitating Euronews’ signature “no comment” ban- ner.

Weltschmerz was first exhibited at the festival Speculum Artium, which took place during April 2011 in Trbovlje, Slovenia. As a 90s TV-set could not be procured in time, a projection was used instead, with the actual content “embedded” within the image of a television set (figure 31). The work was very well-received, and fit quite well within the somewhat shocking or edgy undertone present every year in works from local artists (including co-author Maša Jazbec herself) and occasional curatorial choices.

Further the same year, Weltschmerz was part of the exhibition Unuselesness, part of the Ars Electronica Festival (September 2011 in Linz, Austria). It would also feature as part of Maša Jazbec’s solo show Apparatus:Mikkel=0 at the Galerija Media Nox (Febru- ary-March 2013 in Maribor, Slovenia). The authors opted on using a digital TV screen for both venues (figure 33).

5.5.3 · Outcomes and Discussion Weltschmerz deals with empathy and our reception of mediated violence. The consumption of television programs is remote and sanitized: we don’t even come too close to the picture, let alone touch the television to change the channel or turn it off.

WORKS | 143 As such, the work reinterprets the interaction and relationship with the television set and the violence it mediates as something more personal, performative, fateful and unnerving.

Russian roulette is, by most definitions, a game. But first, it is a game of chance, as the only moves possible are to continue or quit (if altogether allowed); and second, it is a game with potentially deadly outcomes, even the certainty of death, if it is to be played to the end. Television and other mass media are a means of relaying information, but they also have to entertain, captivate or otherwise attempt to satiate the viewer in order to keep an audience. Subliminally appealing to schadenfreude or masochism is “fair game”. So one individual’s “entertainment” may be based on another’s misery, on one’s horrified fascination and the sense of safety by comparison. Weltschmerz highlights the passivity, powerlessness and masochism present in media consumption, while treading boundaries between play and seriousness, and between information and exploitation.

Participants were often quite visibly reluctant towards the premise of putting a revolver to their heads. A few asked beforehand if they would get an electrical shock – perhaps they were familiar with the work PainStation (“From Art to Arcade,” n.d.). Yet reassurance of the contrary did not completely drive all hesitance away. In most cases the issue was the actual performance of the gesture, given its significance – which was not only understandable but also intended. The gun was made of metal and had an almost realistic size and weight, even though it was obtained as a toy from a popular online retailer. One individual, who is known to use unnerving and bizarre atmo- spheric elements in his own work, refused to try the installation, even though he was otherwise curious about it.

While the accuracy (and even adequacy) of the methods for measuring and assessing the participant’s state are far from scientifically accurate, the red meat was nevertheless unnerving as it began to move, sometimes slower sometimes faster; and some form of cause-effect connection was commonly intuited by participants. It can be seen as a method for feedback, manifesting the participant’s potentially masked discomfort visible, in turn inducing more discomfort in a potential positive feedback loop. As a metaphor, the U.S. expression “to spill one's guts” applies to an individual who reveals personal or sensitive information, usually as attempted relief from some sort of pres-

144 | ART AND TECHNOLOGY AT PLAY sure. Alternatively and perhaps contrarily, the meat-like substance can be seen as a filter, condenser or dream-catcher, placed between the participant and the violent footage to absorb and dissipate the participant’s anguish.

Besides the aspects of empathy tested in the relationship of participant and footage, the installation also elicits the empathy of other visitors towards the participant. The actions performed – playing Russian roulette and especially shooting oneself in the head – constitute a powerful (even poisonous) form of accountability, as they are not only easy to understand, but especially easy to relate to. The observer is, after all, watching someone put a gun to the head and pull the trigger (figure 35); and that someone may be a stranger, an acquaintance, a friend or a loved one.

WORKS | 145 5.6 · Wcielenie 2011 With Justyna Zubrycka

Wcielenie (incarnation or embodiment) is a narrative experienced interactively in an urban space. The participant is led through the streets and buildings of the historical Kazimierz neighbourhood in Krakow, listening to the walls by means of a device which borrows the shape and function of a stethoscope.

The story revolves around the murder of a young girl, and her apparent return as a vengeful creature inspired by Slavic folklore. The participant’s role actualizes a deep connection with the story’s main character, revealed during the narrative’s finale.

5.6.1 · Background and Concept Wcielenie was developed together with Justyna Zubrycka, with the intention to use the historical neighbourhood of Kazimierz, located in Krakow, as the stage for a game promoting exploration and discovery.

The essential format of Wcielenie was motivated by the author’s interest in location- based narratives, and previous experience with InStory (see 5.1.1, Correia et al. 2005) and Noon (see 5.2). It also shares aspects with location-based applications such as REXplorer (Ballagas et al. 2007), which employ game forms to uncover historical or context-based information; as well as Christina Kubisch’s work Electrical Walks, in which participants roam the city listening to the surrounding electromagnetic waves; and Blast Theory’s location-based narrative A Machine to See With, which provides an incredibly immersive experience borrowing from the cinematic medium, while using a common mobile phone as interface.

The work was commissioned by the Fresh Zone contest of the third edition of the Art- Boom Festival which took place during June 2011 in Krakow, Poland.

While Kazimierz is often referred to as a Jewish neighbourhood, the authors preferred to avoid the (otherwise obvious) choice of designing an experience focussing specific-

146 | ART AND TECHNOLOGY AT PLAY ally on Jewish religion and culture (of which Kazimierz already has plenty of good content to offer), preferring a more folk-mythological approach. As such, inspiration was initially drawn by a review of legends and creatures from Polish folklore, several of which are also more generally a part of Slavic folklore, with occasional regional variations.

As starting point, the authors chose the myth of the Strzyga (pronounced similarly to “stcheega”). According to folklore, Strzyga are born to human parents and otherwise resemble a normal female human, except they have two hearts, two souls and an additional, hidden set of teeth. Upon death, one of the souls ascends to heaven and the other remains in the body, animating it as a blood-thirsty vampire-like creature.

The narrative outline emerged as an interpretation of the Strzyga myth. Influence from the earlier work Noon – A Secret Told by Objects is especially present in the aspects of supernatural mystery, exploration and investigation (see 5.2).

The plot revolved around Urszula, a young girl who would go missing and then seemingly return as a blood-thirsty Strzyga. The narrative was divided in three acts (figure 43 shows a causal map). The first act introduced Urszula and those more closely related to her, (almost literally) following in her steps up until and including her murder. The second act followed her friend’s investigation of her disappearance; and revealed the characters responsible for her tragic fate, their motivations and growing unease. The third act mostly portrayed the Strzyga’s revenge, followed by a conclusion where the player’s connection to Urszula is diegetically contextualized.

5.6.2 · Design and Development The story was shaped and defined concurrently with the interaction design, and also influenced by the availability and choice of locations in the Kazimierz neighbourhood. This included considerations about accessibility (“can a participant find/enter this place?”), narrative adequacy and realism (“could this scene reasonably happen here?”) and also geographical location (“is this location too close/far from the other?”).

The narrative would be delivered through a device which should be simple enough to procure or produce, so that several could be made available, allowing for more simultaneous participants or for quick replacement in case of malfunction. As such, the narrative was shaped as a series of conversations (although occasional narration was ne-

WORKS | 147 cessary), as auditory memories from urban locations and structures; and the interface borrowed the familiar shape, function and usage of a stethoscope, with which participants could listen to the memories of walls. The device was dubbed Mnemoscope, and its origin explained within the story (figures 40, 41 and 42).

Listening to certain pieces of the story would trigger the availability of others, and through this process the narrative was driven forward (figure 43). As such, memories provided further insight into the motivations of and relationships between characters, but often did so in a way that directed the participant towards a location where the story would overall develop (i.e., trigger the occurrence of new memories in other locations). The choice of where to go next depending on the interpretation of information uncovered so far was intentionally designed to provide a gameplay aspect which is simple and intuitive.

The participant’s interaction and choices did not influence the outcome of the narrative (as was the case in Noon), but rather the way it was experienced, including taking different paths through the story and understanding it in more or less detail, depending on exploration and curiosity.

The scenes were originally written in English. Once completed they were translated and adapted to the Polish language, and excellently interpreted by hired voice actors.

Mnemoscope Three mnemoscopes were made for Wcielenie. Each contained an Arduino board controlling an RFID reader module, which would detect tags placed on the walls, and an audio player module which would reproduce the corresponding audio files (figures 36 and 37). The chosen shape was a simple hemisphere. The flat surface would be placed against the wall to listen to memories, and the round surface facilitated holding the shape in place. The shape’s size allowed plenty of room for the electronics, while remaining comfortable to hold (figure 38). Green LEDs were added, which would pulse when detecting an RFID tag. The earpieces were sourced from actual stethoscopes, tightly coupled to a headphone speaker connected to the audio module. The hardware was powered by a lithium-ion battery.

A simple tilt-switch was placed to estimate motion, used for triggering specific scenes while the participant was walking from one location to another. For instance, while

148 | ART AND TECHNOLOGY AT PLAY Figure 36: Mnemoscope hardware prototype. Figure 37: Mnemoscope hardware.

Figure 38: Mnemoscope hardware placement. Figure 39: Wcielenie map, Kazimierz neighbourhood.

Figure 40: Mnemoscope and Wcielenie map, with RFID tags and indications for placement.

WORKS | 149 Figure 41: Wcielenie participant outside the bookstore.

Figure 42: Wcielenie participant at the Figure 43: Wcielenie narrative map (cheatsheet). playground.

150 | ART AND TECHNOLOGY AT PLAY “following” Urszula on her way home from school a scene leading up to her murder is played. While this did not constitute an accurate location method by any means, some players have enjoyed this behaviour.

Other Assets and Finale To aid in navigation and exploration a foldable map was designed, representing the area relevant for the story, with each location indicated by a unique icon (figure 39). Since the experience could take up to an hour or more, an optional cheatsheet was also made available for participants who wanted to follow the straightest path through the story, limit the amount of deviations from it, or simply make sure they weren’t somehow going in circles (figure 43).

RFID tags were placed at each key location, made more visible by yellow stickers with the respective icon: Urszula’s house, her school, the playground, the church, the alley, the shops, the bookstore and the Banker’s house. The graphic design of the map, RFID tags and cheatsheet was made by Barbara Dzierań, who also contributed to other aspects of information design.

Another tag was placed on a table at the Mleczarnia café, which was not indicated on the map. After the last scene one of the intervening characters would directly address the player with indications to go to the café, where the story’s finale would take place.

At the Mleczarnia café, the character of Jan addresses the participant directly as if he/she were the spirit of Urszula. This implied that the participant had been playing the part of (or embodying) Urszula’s ghost throughout the interactive experience of Wcielenie’s story.

The same character then offers the speculation that this ethereal or immaterial “ghost” of Urszula (embodied by the player) complements the qualities and vengeful- ness of the incarnate Strzyga. This is, of course, based on the folk version of the myth, where a Strzyga is said to have two souls: one which ascends to heaven after death, where the other reanimates the body as a vampiric creature.

Before saying goodbye, Jan tells the participant of a present he placed inside the table’s drawer. The player may open the drawer, to find a necklace with a Möbius strip shaped as an infinity symbol, which he/she could take as a souvenir. The necklace was designed and manufactured in a small but sufficient quantity by Urszula Tarasiuk.

WORKS | 151 5.6.3 · Outcomes and Discussion Wcielenie was designed to lead participants on the exploration of a physical space through the exploration of a story, and vice-versa. By choice, it did not attempt to edu- cate about or directly refer to historical facts. It rather conceptualized Kazimierz as a medium to tell a story instead of telling a story of Kazimierz. While Noon appropriated existing objects to be used as media in a rather stationary setting, Wcielenie appropriated a neighbourhood: conceptually a large navigable object which, upon close in- spection, is full of memories and stories.

Humans perceive the world as a space filled with objects both small and large (Pinker 199:191) and as such are able to navigate a landscape by mentally framing it as a large object and locating its prominent parts, i.e. landmarks, both natural and artificial (377). Furthermore, our curiosity is naturally drawn by features such as meandering paths and hidden places, mysterious features which are perceived as a kind of beauty (ibid.).

The narrative was experienced by less than 20 participants in total, perhaps due to lack of promotion. While the first act was seen as straightforward, most participants did not make it to the third act. Several would eventually quit as they realized that see- ing the story through might take more time than they had; nevertheless they claimed having enjoyed the experience that far.

Very few participants were known to see the ending. One of them visibly and mani- festly engaged the experience, having taken almost two hours, and reportedly making use of the cheatsheet solely to ensure all the content had been listened to.

While listening to a memory, the mnemoscope had to be kept close to the wall (figure 42). This was a conscious design choice, taken for two reasons: it was meant to mimic the use of an actual stethoscope, which cannot be employed at a distance; and it would lead the player to associate story and location more deeply.

However, a few participants reported that they would have welcomed the option of listening to the story while walking to another location (to save time, one claimed). As mentioned above, certain pieces of the narrative were delivered while the player was walking, but these were few and far off. Players who experienced the first of these (Urszula’s death) found it pleasantly surprising and well contextualized.

152 | ART AND TECHNOLOGY AT PLAY At the end of the story it was revealed that the participant embodies an aspect of Urszula which is opposed or complementary to that of the Strzyga. This can be interpreted in a straightforward way as saying that the player represents some good and the Strzyga some form of evil.

However, as the player does not affect the outcome of the story, it can also be interpreted as passivity or even powerlessness: the participant plays the ghost of a memory, reliving tragic events yet unable to affect them, while the Strzyga causes change in the story’s world as an agent of justice or vengeance. Seen like this, the narrative self-referentially breaks the fourth wall, critically addressing its own interaction design for not allowing the player a more active role.

WORKS | 153 5.7 · Tripo 2013 With Vesela Mihaylova Music by Claudio Pina

Tripo introduces an imaginary device inspired by the tin can radio (or string telephone) to deliver three short narratives. Participants are led on the exploration of an indoor space, to find marked hotspots where the device can be used to listen to the memories of walls. The user can switch between stories by rotating the device, as if tuning an old analog radio. Evoking nostalgia and the play of childhood, Tripo is used to augment a space, trigger imagination and encourage exploration.

5.7.1 · Background and Concept Tripo was realized together with Interface Cultures alumna Vesela Mihaylova as an original concept for the Water Tower Art Fest, taking place during June 2013 in Sofia, Bul- garia. As the name hints, part of the festival took place in an old water tower. The authors chose to take inspiration from the destitute charm of this specific venue and portray it as a place that was once inhabited.

This provided the opportunity to borrow knowledge and formal aspects from the previous project Wcielenie (see 5.6), but at the same time to address some limitations and explore different directions. As Tripo borrowed the form of Wcielenie, is was also inspired and informed by the same sources (see 5.6.1).

Although Tripo was developed for the Water Tower Art Fest, the authors also sought to create an experience which would be as simple as possible to transport, set up and enjoy – for the water tower as well as upcoming venues. The work Headbang Hero had been quite successful in this regard. It was very straightforward and self-contained, both as technical system and experience. In contrast, Wcielenie had been made very specifically for the Kazimierz neighbourhood. In order to stage it somewhere else the large script would have to be adapted for available locations; and likely translated to and recorded in another language.

154 | ART AND TECHNOLOGY AT PLAY Figure 44: The Tripo device.

Figure 45: Inside of the "reader" can. Figure 46: Tripo poster with instructions.

WORKS | 155 Figure 47: Tripo participant using the device.

Figure 49: Tripo hotspot (sticker over RFID tag).

Figure 48: Tripo participant exploring a secluded Figure 50: Rotating the device to "tune in" to location. different stories.

156 | ART AND TECHNOLOGY AT PLAY 5.7.2 · Design and Development From the outset, the authors considered using the same technical approach as Wciele- nie (although this was not an absolute must), without repeating the same shape of the device. Different shapes and forms of use were pondered upon, mostly through imaginative exploration, and some would prompt different technical approaches. The choice fell on the tin can radio for its simplicity of shape, construction and use, as well as for its charm and childhood nostalgia (figure 44).

The tin can radio, also known as string telephone, is a self-constructed toy consisting of two empty cans or cups tied with a string at the base. When the cans are pulled apart and the string kept taught, it functions as a crude telephone. Besides evoking a childhood where playthings included less packaged and electronic toys and more artefacts crafted or improvised by oneself, its aesthetic and crude construction were found to be both fitting for the location and peculiar enough in itself.

In the previous work Noon, each object contained memories from different time periods. The time period could be selected by using the clock, and some objects would yield additional memories if used in a specific way.

To keep the number of story locations small, a similar approach was used. One of the few gestures that could be performed while holding the can to the wall was rotating the can. As the can would function as a sort of old radio device, the choice seemed clear to use the same functional design; and so by rotating the can, the participant could effectively tune between narratives (figure 50).

The number of locations was fixed to six and the number of narratives to three, for a total of eighteen narrative pieces. In Bulgarian, the expression “three times six” trans- lates to “tri pŭti shest”, which was abbreviated to Tripo.

In order for the audio narratives to be as language-independent as possible, the authors chose to make them fairly simple and using mostly noises. A list was made of noises that would be reasonably easy to procure, and which could occur or otherwise be heard inside a house. Based on those, the narratives were devised and each adequately split in six pieces.

In one of the narrative pieces a man comes home from the rain, hangs his jacket, pours himself a drink, and sits down listening to the radio. This is the first of six in the

WORKS | 157 story about a man in a stormy night.

In another, a dog comes running and barking at a cat, who retreats to a safe place, screeching and hissing. This piece belongs to the story of the mouse in the house.

The third narrative is not a narrative per se, but rather a song, written and performed on the piano by Claudio Pina. The song constitutes a sequence of parts which is easier to follow temporally, where the other stories might leave room for doubt or interpretation. Even though listening to and interpreting the narratives in a different order is not discouraged, the song remains as a guideline to a feasible ordering of events.

The tin can radio device was made by populating two used cans with electronics. The cans had slightly different shapes: the longer, thinner “reader” can would house most of the hardware (figure 45); and the shorter, broader “speaker” can which would contain a small loudspeaker. Inside the “reader” can were an Arduino board, an audio player module and an RFID reader module, whose antenna was attached to a black plastic lid for the can. There was also enough space left for a 9 Volts battery which, depending on how often the device was used, could last quite long. An audio cable ran from the audio module in the “reader” can to the “speaker” can, which simply contained a small loudspeaker and a black cover made from wood.

On the “reader” can also had an on/off switch and a status LED, which would pulse whenever an RFID tag was detected (figure 44). The use of a pull-switch was initially considered, so that users would have to pull the cans apart to turn on the device (mimicking the way an actual tin can radio works). At the time, however, it was deemed preferable to err on the side of caution and avoid encouraging users to pull on the cable at all. This design was left for a potential future version.

In an ideal scenario, the device would have worked on any wall or part of a wall. As comparison, Christina Kubisch’s work Electrical Walks can be used anywhere to listen to electromagnetic phenomena, although this also means that content is sourced from the environment rather than composed in advance. Among other reasons, we didn’t find a convincing way to detect the placement of the “reader” can against a wall in a way that would exclude other scenarios – such as putting it to one’s own hand or the surface of a small object.

Stickers were designed in order to both cover the tags and make their location visible

158 | ART AND TECHNOLOGY AT PLAY (figure 49). The stickers bear the Tripo logo within a circle of the same diameter as the “reader” can, indicating an adequate position for the correct reading on the underlying tag. A small set of instructions was also produced to ensure participants used the device properly, including turning it off and returning it once they were done exploring (figure 46).

Tripo was always very easy to ship, as the device and tags combined have a relatively small form factor. Setting up was also simple, the most critical choice being the location of the tags – these should be accessible and visible, but not too easy to find (figure 48). If possible, the location of the tags should match the excerpts from the story about man in a stormy night (which audibly include elements such as doors, windows, corridors and stairways). This was admittedly not always possible, and in such cases it was left to the participant to interpret and imagine a connection, if any. In large spaces two of the tags were placed very visibly near the device’s initial location, so the participant had a clear starting point to use the device.

Besides the Water Tower Art Fest (June 2013 in Sofia, Bulgaria), Tripo was also exhibited the same year at the Kiblix/MFRU festival (November 2013 in Maribor, Slovenia) as well as other events throughout the following years. The latest showcase was during the Berlin Design Night (June 2016), a city-wide event, where Tripo motivated the exploration of the IXDS headquarters in Berlin. The device never malfunctioned or needed repairs.

5.7.3 · Outcomes and Discussion Compared with other works, Tripo is relatively simple and straightforward, both as an experience and as technical system. Manifesting an ongoing interest in objects and places as tangible interactive media, it was greatly informed by previous projects Noon – A Secret Told by Objects and Wcielenie (see 5.2 and 5.6), resulting in a design which borrows affordances from existing objects, and fosters imagination and exploration. While similar techniques were employed in previous projects, they were also informed by plenty of experience in exhibition contexts, resulting in a system which remained the most portable and stable work by the author, to date.

The experience was intentionally brief and loosely structured. Part of the challenge was to find the six hotspots, which did motivate people to explore the space and pay

WORKS | 159 attention to features they might otherwise not have noticed (figure 48). A perceived majority of participants actively tried to find all the hotspots, even while having a look at other works in the exhibition.

In contrast, Wcielenie gave players a map, to expedite their navigation, as the story was much longer and the space much bigger (figure 39).

The hotspots also piqued the curiosity of visitors, prompting them to try Tripo. Parti- cipants walking around the space using the device also contributed towards this end, as their performance was understandable and accountable to a degree, without revealing the whole experience.

Participants understood how to use the cans and seldom had any difficulty. Their intended usage was explained in the poster at the device’s starting location, and occasionally be the authors or venue staff. Of those who were not directly explained how to use the device, many referred to previous observation of other participants doing so; and of these only a few did not understand the tuning mechanism as the means to access different stories.

The exclusive use of noises (as opposed to conversation or narration) made the scenes more ambiguous and open to interpretation. Some of the mouse story pieces consisted solely of rustling and squeaking, so they were not as easily rateable to specific events as most of the man’s story.

Admittedly, the story can be more or less adequate to the space, and specific coup- lings of story event and location of the hotspot were occasionally hardly matching. This is a caveat of the work’s design for portability, in contrast with the specificity of Noon and especially Wcielenie.

160 | ART AND TECHNOLOGY AT PLAY 5.8 · Lorm Hand 2013 – 2014 With Tom Bieling (Lead), Andrea Clemens, Inci-Ana Zohrap, Chiara Esposito, Fabian Werfel (Student Helpers) as part of the project Speechless – Interaktiv Inklusiv at the Design Research Lab of the Berlin University of the Arts (UdK)

The Lorm Hand is an installation providing a means for individuals with different degrees of deaf-blindness to reach out from isolation and raise awareness to their cause. Participants are invited to write a message and post it on Twitter by means of a tangible, hand-shaped interface designed to recognize gestures of the Lorm alphabet for the deaf-blind, signed on its surface.

Designed as a tool for empowerment, the Lorm Hand reaches out to individuals through the context of exhibition but also to online communities, raising awareness towards both the possibilities that communication technologies can offer, and the challenges they may pose to people living with disabilities. It can also be used as an educational tool for users to learn and practice the Lorm alphabet.

5.8.1 · Background and Concept The Lorm Hand was developed within the context of the project Speechless, at the Design Research Lab, Berlin University of the Arts. The project was funded by the Fed- eral Ministry of Education and Research (Bundesministerium für Bildung und Forschung, or BMBF) and led by Thomas Bieling. The author of the present text took the role of research associate, aided initially by student helpers Andrea Clemens and Inci-Ana Zohrap, and later by student helpers Fabian Werfel and Chiara Esposito (also a fellow student at Interface Cultures). Interface Cultures alumna Ulrike Gollner had also worked on previous stages of the project, more prominently on the development of a pioneering prototype for the Mobile Lorm Glove (introduced below).

Speechless addressed the empowerment of minorities through access to everyday

WORKS | 161 communication technologies, focussing more specifically on deaf-blind individuals with different degrees of impairment. The first design concept arising from the project was the Mobile Lorm Glove, a wearable, mobile communication and translation device for the deaf-blind (Gollner et al. 2012). The glove recognized and reproduced gestures of the Lorm alphabet used by deaf-blind individuals. The Lorm alphabet is comprised mostly of points and lines which an individual reproduces on another's palm using his fingers (figure 55). While Lorm allows a standardized mode of communication, it also requires being within less than arm’s reach; so a deaf-blind individual may only converse with those very close by, which is a factor contributing to a greater sense of isolation.

The Mobile Lorm Glove could recognize signs “lormed” onto its surface via an array of pressure sensors, and reproduce Lorm signs via a corresponding array of tactile actuators. With support from a mobile device, it was possible not only to converse at a distance, but also with those unfamiliar with the Lorm alphabet, as the software translated between Lorm and plain text.

While the author contributed in the further development and refinement of the Mo- bile Lorm Glove – including shape, functionality and supporting technology – the focus here is on the Lorm Hand, as a concept developed from the ground up during his work on Speechless. The Lorm Hand was originally ideated to be exhibited during the protest march Aktion Taubblind - Taubblinde in Isolationshaft which took place on Octo- ber 4, 2013 in Berlin, culminating at Potsdamer Platz. The march aimed to draw attention to the social isolation and loneliness of deaf-blind individuals as form of solitary confinement.

5.8.2 · Design and Development Taking advantage of previous and ongoing work on the Mobile Lorm Glove, an installation format was devised where visitors could use the Lorm sign language (figure 55) to post messages onto Twitter. This was seen as a way of allowing normally isolated individuals to reach out to a wide audience online and contribute towards raising awareness to their cause. The interface would take the natural shape of a hand onto which visitors could “lorm”, composing a message which could then be posted to a Twitter account by pressing the palm.

162 | ART AND TECHNOLOGY AT PLAY First Version The shape of a hand was crafted through vacuum-forming, based on a mould of an actual hand (figures 51 to 54). The inside of the palm was lined with copper foil split into seventeen distinct areas. Two dedicated modules detected the user’s touch by actively listening for changes in the capacitance in each of these areas. An Arduino board relayed the sensor data an application developed in Processing and running on a laptop. The application included a fairly straightforward pattern-matching algorithm, which compared the activation of sensors over time to pre-recorded patterns, to find the closest match within a limited range. The library Twitter4J, developed by Yusuke Yamamoto, was used to abstract the Twitter API.

While the Mobile Lorm Glove could provide Lorm-like tactile stimuli effectively by being secured to the user’s hand, no adequate solution was found for achieving an equivalent effect on the Lorm Hand which could be designed and implemented in an eleg- ant or timely manner. Instead, a single channel of tactile feedback was used, producing a brief vibration when a sign was recognized, and a crescendo when posting on Twitter.

Nevertheless, a screen would display the message as it was constructed. It must be pointed out that deaf-blind individuals are seldom completely visually- or hearing-impaired. Some can perceive general shapes and colours under specific conditions, and as such are able to read large, high-contrast text, even if slightly slower. Naturally, the visual output was also meant for sighted users. Participants and passersby were encouraged to interact with the Lorm Hand, introducing hands-on a means of communicating with deaf-blind individuals (figure 52).

We included a small tactile push button, located in the wrist, that enabled the sensors (the orange circle on figures 51 and 54). With the sensors off by default, blind users could get a feel of the hand’s shape and position before or in-between “lorming”, without triggering the sign recognition. As there is no consensual sign in the Lorm alphabet corresponding to deletion, we also opted on adding a button to the pedestal surface which deleted a single character when pressed, or the whole message when held.

The hand was attached to a pedestal with an inclined surface, in a way that it could be

WORKS | 163 rotated freely between two positions: palm facing up and palm facing the user. This was prompted by the observation of different positions of the receiver’s hand in “lorming” individuals. A slot was opened on the surface above the hand, so that the laptop could be inserted with only the screen remaining visible. The twitter account LormHand was created specifically for the project, with the associated developer profile and API permissions.

The installation ran wirelessly for one afternoon, powered by the laptop, for which five extra battery packs were provided, and communicating with Twitter via the cellular network. Users were offered short explanations and assistance from the Speechless team, when needed, but otherwise interacted with it freely, in a public environment. Informal usability tests were also conducted in-house. For these we replaced the pedestal by an equivalent structure that could be placed on top of a table and participants used the Lorm Hand while sitting rather than standing.

The Lorm Hand proved easier to use for sighted users than for those with a considerable degree of visual impairment, who frequently had difficulties for different reasons.

Although the function of the enable button was explained, it was often forgotten during use: while released it caused no gesture to be interpreted, whereas while pressed it detected unintentional gestures. This issue improved with continued use in the more relaxed setting of the usability test.

It was also suggested that ascertaining the location and contour of the hand would have been easier if the background had a contrasting colour. Live interpreters for sign language commonly wear black clothes on their upper body and white gloves to make their gestures more visible from a distance.

Another issue was that using a realistic shape with a rigid material meant that users were “lorming” onto an irregular surface, and so occasionally their touch would not register with the sensors – especially when “lorming” fast. This would cause the wrong character (or no character) to be interpreted. One user in particular felt the realism of the shape to be a bit distracting, as it was like “lorming” onto a mannequin’s hand. This can be seen as kind of “uncanny valley” effect.

Proficient “lormers” may sign quite fast and flowingly. While the software could handle a degree of speed in single characters, quick natural transitions between gestures

164 | ART AND TECHNOLOGY AT PLAY Figure 51: The Lorm Hand, first version.

Figure 52: Lorm Hand at the protest Figure 53: In-house testing with a deaf-blind march. participant.

Figure 54: Lorm Hand, first version (on black base instead of pedestal). Figure 55: The Lorm alphabet.

WORKS | 165 Figure 56: Lorm Hand, second version.

Figure 58: Detail of the flexible circuit (second version).

Figure 57: Lorm Hand, third version. Figure 59: Cabling study (third version).

166 | ART AND TECHNOLOGY AT PLAY were not handled properly. So proficient “lormers” had to slow down and sign more mechanistically.

Finally, the gesture for posting on Twitter was usually performed with unnecessary force, so that the hollow shape of the hand suffered some damage and almost broke on one occasion.

In spite of these issues, the Lorm Hand’s presence at the protest march was well-received. Many sighted participants had their first contact with the Lorm alphabet and others got to practice their skills. Most visitors understood the installation as base on a work-in-progress concept and expressed enthusiasm about further developments. Some deaf-blind participants offered to help in testing further prototypes.

A video with footage from the event was published online shortly after (Bieling 2013a), followed by one which more clearly illustrates the use of the Lorm Hand (Bieling 2013b).

Second Version A second version was developed which attempted to address most of the previous issues, with significant changes regarding the shape and construction of the hand, and the touch-sensing method. The development of this version was supported by a grant from the Hochschulwettbewerb (University Competition) 2014 organized by the Fed- eral Ministry of Education and Research.

Concurrently, a version of the Mobile Lorm Glove was being developed which used a custom-designed layout of pressure sensors. These allowed reading a range of pressure values, whereas the capacitive method used on the Lorm Hand reported a binary, on or off state only.

It was hypothesized then that the pressure values could be used by the pattern- matching algorithm to better distinguish intentional touches from unintentional ones. This would also eliminate the need for the unintuitive “enable” button, as hopefully visually-impaired users could feel the hand’s shape using softer touches and sign Lorm characters with more intentional gestures, applying more pressure.

The pressure-sensing surface was implemented according to the same functional principle of commercially available force-sensing resistors (FSR), however with a spe-

WORKS | 167 cific shape and flexibility: as a circuit etched on a flexible substrate, covered by a thin layer of a specially impregnated textile whose electrical resistance varies with pressure.

The sculptor Elisabeth Scharler was enlisted to model a solid and more abstract shape of a hand, with smooth surfaces to allow for unimpeded sliding of the user’s fingers, as well as to better accommodate the flexible array of pressure sensors. To ensure stability and durability, the choice was made not to allow the hand to rotate (which had been seldom used). The hand was fastened to the base using two threaded rods, embedded into the shape while moulding.

The vibration motor providing haptic feedback was placed tightly within a hole drilled at the base of the hand, and the vibration propagated well without becoming distracting. The delete button was kept, but moved to a central position near the wrist.

The flexible circuit part of the pressure-sensing array was secured to the hand (figure 58). The hand was then covered covered with a glove made from a stretchy and smooth synthetic fabric, lined on the palm with the pressure-sensitive resistive material. The use of a textile glove was not related to the Mobile Lorm Glove. It kept the resistive material in place (which could not be directly attached to the flexible circuit), while conforming well to the contours of the hand, and could be easily removed for adjustments and repairs (figure 56).

The pattern-matching algorithm was refactored to consider a range of pressure values for each activated sensor when comparing with known patterns, rather than a sequence of binary (on/off) signals, as was the case with the capacitive sensors on the first version. The possibility for adding new patterns during runtime was also included. As such, if the Lorm Hand would repeatedly not recognize a sign “lormed” by a user (e.g. because of the user’s peculiar way of signing a specific character), the authors could tell the software to “learn” that gesture, that is, to store the pattern resulting from the user’s gesture and associate it with a character or sequence. Besides facilitating the fine-tuning of the Lorm Hand to specific users, this would also allow for the definition of personal “sign-shortcuts” for frequent combinations of characters, words or expressions (something which exists already in the Lorm alphabet, with regional variations). This functionality was also added to the Mobile Lorm Glove, meant for personal use.

168 | ART AND TECHNOLOGY AT PLAY This version of the Lorm Hand was first showcased at the Leben mit Taubblindheit Kongress (Living With Deaf-blindness Congress) which took place during September 2014 in Potsdam (figure 56); and shortly after as part of the VerbaVoice exhibition area at the Medientage München (Munich Media Days) which took place on October 2014 in Munich.

Being among the finalists of the Hochschulwettbewerb, the Lorm Hand placed 2 nd in the award ceremony, on December 12, 2014.

Both versions of the Lorm Hand also featured in a short documentary video about the Speechless project, published online by the Design Research Lab (2015).

The new hand shape was a definite improvement, as it proved more stable and reportedly easier to sign onto. Users inquired about the shape didn’t find it distracting or unpleasant. Whereas previously users were touching hollow plastic which conveyed an artificial feeling, the new Lorm Hand was solid, smooth and covered by a glove – offering a soft and pleasant tactile feeling, but also a sense of humanity, as objects don’t usually wear clothes.

Visually-impaired users which retained some ability to distinguish shapes were able to ascertain the hand’s position to some degree.

The pressure-based gesture-recognition was overall more precise and less sensitive to unintentional contact. It allowed better recognition of or distinction between certain signs which had been challenging with the previous approach – especially the R, K and the gesture colloquially used to separate words.

Although the fabric used for the glove was quite smooth, applying enough pressure with the finger so that the sensors were meaningfully triggered often meant that the fabric would shift or stretch. This was more pertinent in signs with long gestures (such as the L) or ones that comprised several gestures (like the crossing gesture for Ch). The resulting displacement of resistive material surrounding the touched areas, or the pull exerted on edges (especially the fingertips) where the material was already stretched tight, resulted in the triggering of sensors not associated with the gesture. One straightforward way to mitigate this was by recording several patterns for the same character, “lormed” with different speeds and strengths. However, this had to be done frequently, as the glove and lining itself shifted through repeated use.

WORKS | 169 Another issue was the fact that the white glove had to be frequently washed, for reasons that are obvious. While this was foreseen and deemed acceptable, a third version would be built that would do away with removable parts for easier maintenance during longer periods of exhibition.

Nevertheless, this version of the Lorm Hand also made for an interesting piece for more traditional museum contexts. As part of the exhibition Das Netz: Menschen, Ka- bel, Datenströme at the Deutsches Technik Museum (Berlin) in 2015, it was shown inside a display case with the glove partly removed, so that visitors could have a glimpse of the underlying technical aspects.

Third Version The latest version of the Lorm Hand was designed to be easy to set-up and maintain. This would allow it to be shipped and/or remain at venues for a longer time, whereas the previous versions required more constant care by specialized individuals (usually the Speechless team).

To do away with moveable and removable parts, sensors had to be permanently embedded into the hand shape. This posed a risk: if something was done improperly during manufacture or later malfunctioned, it would be impossible to repair without defa- cing the shape. Capacitive sensors were re-introduced, as they are more monolithic and durable when compared with those with mechanical functional principles.

The sculptor John Von Bergen assisted throughout the process, which had to be done with special thoughtfulness and care (figure 59). Any mistakes would imply starting over. Test samples were produced with different modelling materials and glazings, as well sizes of sensors, to assess the best combination for sensor behaviour and surface smoothness. Once the material and glazing were selected, the solid hand shape was moulded. The copper pads were placed, and channels for cabling milled on the hand’s surface, deep enough to prevent the cables’ capacitance from being affected by a user’s touch. Once the cabling was in place, the channels covered by spackling. A thin layer of glazing was the poured over the hand. Throughout the whole process, tests were regularly conducted to determine if the sensor were still working.

A base was produced to contain the modules controlling the capacitive sensors (as in the first Lorm Hand) and a Raspbery Pi, which was much smaller but also computa-

170 | ART AND TECHNOLOGY AT PLAY tionally less powerful than the laptops previously used. This meant the existing software had to be modified and optimized so to run smoothly on the Raspberry Pi. Ports were added for USB, to connect peripherals for configuration and triobleshooting, and HDMI output for connection to a display; as well as a power cord to be plugged to the mains, and power the Lorm Hand.

This version was completed after the official date of conclusion of the funded project, through the continued work of Chiara Esposito and Fabian Werfel. While not officially associated with the Design Research Lab once the project ended, the author did provide guidance on the final steps (especially the software) and a positive assess- ment of the outcome (figure 57).

5.8.3 · Outcomes and Interpretation The design and development of the Lorm Hand was a process which involved a lot of technical experimentation, tied to further developments on the Mobile Lorm Glove. In being so related, the two concepts borrowed from and informed each other not only in technical aspects but also in functional design and usability.

The choice of a tangible shape for the Lorm Hand may seem quite natural, in hind- sight, given that Lorm is signed onto the palm of a receiver’s hand. Alternatives exist without the need of a tangible shape, which are at least technically feasible. One such, tried out to some degree but never fully developed, was resorting to a tablet secured to the palm of the user’s non-dominant hand. The user would “lorm” onto the tablet’s touch-screen surface as if “lorming” onto the non-dominant hand, guided by proprioception of the hand’s position. The biggest advantage of this approach was that it did away with the need to build a custom tangible interface – demonstratedly quite daunting, as it involved a lot of trial and error as well as knowledge on the integration of heterogeneous materials and processes.

The choice of a hand shape has first and foremost facilitated usability through intuitive interaction. Its usage by deaf-blind individuals relies on the performance of a familiar task, that is, “lorming” onto another’s hand. This in turn helps to make the user’s actions accountable, that is, understood by others to some extent. Among other things, this led passer-by to stop and observe individuals currently interacting with the Lorm Hand, making sense of what was going on, and being driven in turn to engage

WORKS | 171 with it – quite naturally and even in an exploratory way, with only occasional need for clarification.

A tangible interface may successfully borrow aspects from an existing object or shape, but trying too hard to resemble the original may introduce an “uncanny valley” effect. A realistic hand would have been potentially disturbing to users, as it might have felt like a corpse’s hand (this was an occasional topic of discussion with users). Therefore a successful design may rely more often on emulating desirable affordances (including aspects of shape and significance), rather than outright copying a shape, material or aesthetic for realism.

Like the Mobile Lorm Glove, the Lorm Hand provides a means of communication to individuals who struggle with isolation, albeit in a limited uni-directional way. The Lorm Hand’s true efficacy, however, resides in its role as a tool for education, empowerment and activism. In the specific way it was designed for the context of public events, interaction and participation, it simultaneously raises awareness in three fronts.

Deaf-blind individuals are made aware of the possibilities for inclusion and expression afforded through technology and interaction design, when information and communication technologies are otherwise almost exclusively designed for a majority of non- handicapped users.

Sighted visitors are made aware of aspects of the life of a deaf-blind individual. While provoking reflection upon an isolated way of life and associated challenges, the Lorm Hand also introduces the Lorm alphabet, as a way of communicating with and mitigate the isolation of people with severe visual- and hearing-impairment.

Online communities comprised of possibly distant individuals are also made aware of the deaf-blind cause, especially their struggle for inclusion and participation. Several of the messages posted on Twitter are written by deaf-blind users expressing themselves to a potentially wide audience. Even when the messages are as short and simple as a “hello there,” their impact is reinforced through their authenticity and personal nature. Although unintentional in design, the occasional misspellings caused by the interface’s shortcomings can also be interpreted as signalling the difficulties deaf- blind people face in accessing online social media.

Besides the role of raising awareness on different fronts through exhibition at public

172 | ART AND TECHNOLOGY AT PLAY events, the Lorm Hand can also be used in educational contexts, as a tool to assist the learning and practice of Lorm sign language; and as such contribute to its diffusion.

The Lorm Hand (like the Mobile Lorm Glove) has frequently featured on media channels dedicated to individuals with disabilities, as an illustrative sign of hope for a better future, where information and communication technologies are shaped to bring their benefits to those who need them the most.

WORKS | 173

6. CONCLUSION

The previous chapter has focussed at length on the creative works which constitute the core of the thesis. Throughout, the author has opted on a more narrative, de- scriptive and interpretive approach, with the intention of framing each work holistic- ally – attending to motivation, technique, process, artefact, reception and insight as interacting aspects of a wholesome entity. Among other things, this serves to reinforce the fact that each work, and the collective body they compose, constitute original creation. They address possibilities, challenges or ideologies immanent in human-machine interactions, in an age of ever-increasing dependency on digital technologies for all aspects of human social life. This is the epoch of smartphones, wearables, “likes” and selfies; the point at which a newly-elected president’s Twitter ramblings have almost immediate (and potentially dire) economic and geopolitical effects (Sims 2016).

6.1 · Summary of Strategies While the works have been presented highlighting significant details, the reader may certainly wish to have a more concise and comparative idea of how Embodied Interac- tion and playfulness support each work. As such, two tables have been included below (see Table 1 and Table 2), followed by a short summary. These highlight prevalent aspects in each work and, while certainly being informative, are not ontologically rigid or exhaustive. From the inquiry’s outset, the intent was not to provide a taxonomy or cover every possible aspect. That is, the works were not undertaken with the objective of extensively charting strategies or aspects of embodiment and playfulness. Rather, the present text surfaces and contextualizes approaches and outcomes which were employed in each work.

CONCLUSION | 175

176 | ART| AND TECHNOLOGY AT PLAY PLAYFULNESS INTERACTION EMBODIED Expression Augment Usability M Engage indset Table Strategic 1: of Embodied Interaction use work. each in storytelling natural rhetoric interactivity magic/sci-fi role-playing narrative exploration/investigation bricolage perspectives critique communication familiar intuitive/spontaneous Table Strategic 2: work. each use of in playfulness • • • Gauntlet Gauntlet • • • • • • • Noon Noon • • • • • • Headbang H. Headbang H. • • • • • • • • Rambler Rambler • • • • Weltschmerz Weltschmerz • • • • • • Wcielenie Wcielene • • • • • • Tripo Tripo • • • • Lorm Hand Lorm Hand 6.1.1 · Embodied Interaction Embodied Interaction has been used throughout the works, resorting to the advantages offered by tangible and wearable interfaces to improve the works’ usability as human-computer interfaces and to confer or refine their roles as means of expression.

Usability was improved through the design of interactions which: make use of natural actions, such as pointing or walking; allow quick, intuitive understanding of use; rely on or encourage spontaneous behaviour; and present configurations or situations that are cognitively or culturally familiar to most users.

Expression was improved by resorting to: the use of objects and places as media for storytelling, relying on their resilience/materiality, shape and symbolism; the chanel- ling of habitual actions into alternate forms of communication; and the appropriation of the meanings and uses of an object as a form of critique.

6.1.2 · Playfulness Playfulness, while pervasive in (and perhaps essential to) creative practice, has not only provided an overall mindset in the practitioner’s identification of and approach to creative opportunities, but also been employed to engage the curiosity and imagination of participants; and to augment the qualities of a work as procedural medium.

A playful mindset allowed the practitioner to assemble (or occasionally stumble upon) unusual, unorthodox perspectives, enabling innovative approaches to what may otherwise be ignored or taken for granted; and similarly, to engage in bricolage with seemingly unrelated materials, processes or phenomena, revealing similarities, com- patibilities and synergies.

A work can engage participants in ways that are intrinsically rewarding by: addressing natural curiosity in exploration of physical spaces; elicit investigation and puzzle- solving; offer participation in a narrative process; appeal to imagination through science-fiction scenarios or by granting magic abilities; and even allow the participant to shift perspectives through role-playing.

Finally, playfulness can augment the interactivity of a work by providing enjoyable obstacles to be actively addressed as part of the process; as well as the work’s rhet-

CONCLUSION | 177 oric, by resorting to humour, using ambiguity to provoke reflection or conveying a message efficiently through enactment and exploration.

6.1.3 · Summary of Works The following is a summary, highlighting the application of strategies in each work. Descriptors in parenthesis refer to entries on Table 1 and Table 2, generalized above (see 6.1.1 and 6.1.2).

The Gauntlet Provides a way to include gestures and manipulation of objects in human-computer interaction (natural, intuitive, familiar). The concept was inspired by pervasive games and live action role-playing games.

Noon Uses common objects (familiar) as a storytelling media and distributed personhood, in their relation to people and actions (storytelling). Appeals to curiosity by inviting investigation (investigation). Wearable and premise impart the role of investigator or spiritual medium (role-playing, magic/sci-fi). Hunting for poltergeist provides an additional challenge and variation in the activity (interactivity).

Headbang Hero Uses a familiar object, albeit in an unusual context (familiar). Interaction relies on a spontaneous performance (spontaneous). Uses a wig as game-controller, in a game addressing the dangers of involving the player’s body in reckless performance (perspectives). The format of a game is used to address a phenomenon related to videogames and gamer culture (rhetoric). Uses procedurality to convey a message through experience, very directly and physically (interactivity).

Rambler Turns an almost thoughtless natural gesture into a means of communication (communication). Interaction is based on a natural behaviour (natural). As a result, the performance can be spontaneous and include dancing or silly walks, for instance (spontaneous). Approaches the interaction with online media through an everyday, subconscious action (perspectives). Combines the physicality and trendiness of sneakers with

178 | ART AND TECHNOLOGY AT PLAY the digital textuality and social aspects of an online social network (bricolage). In being humorous and questionably ridiculous, it is ambiguous and thought-provoking (rhetoric).

Weltschmerz Appropriates the use of a gun (familiar) on oneself, to change a violent TV channel (critique). Combines a gun, a piece of meat, physiological signs and zapping (bricolage). Appropriates a game-like structure as part of expression (rhetoric). Additionally, reaches observers through their empathy with the participant.

Wcielenie Borrows the form and use of a stethoscope (familiar). Uses a neighbourhood, and occasionally walking, to tell a story (storytelling). Engages in exploring a location (exploration), following the interconnected stories of characters (investigation, narrative), with folklore or and spiritual theme (magic). Thrives on the fantastical preposition of using a stethoscope to listen to memories of walls (magic). Player represents a character in the story, although this is only revealed at the end (role-playing).

Tripo Borrows the form and some of the use of the tin can radio (familiar). Re-purposes a destitute space as a way to tell stories (storytelling). Prompts the participant to explore a space (exploration) in order to follow a series of narratives (narrative). Thrives on the fantastical preposition of using something like a childhood toy to listen to the memories of walls (magic). The challenge of searching for hotspots in hidden places complements the device’s interaction design (interactivity).

Lorm Hand Allows deaf-blind individuals to express themselves using a sign language (intuitive/spontaneous). Employs a familiar tangible shape (familiar), as a means of posting on Twitter (communication). Brings together the Lorm alphabet, a sculptural shape, Twitter and deaf-blind activism (bricolage). Technical development involved experimentation with different materials and methods (bricolage).

CONCLUSION | 179 6.2 · Closing Comments This dissertation has provided an account of events spanning almost a decade of creative practice. A lot can happen in a decade – sometimes even in a single day – and a lot can change. The author’s own perspectives as researcher markedly evolved from more monolithic (if sincere) concerns with ubiquity and usability to a more pluralistic view upon the heterogeneity of roles that technology plays in contemporary society, and the different many-to-many relationships that connect people with and through the uncanny abundance and complexity of technical means.

Composing the present text has required recollection and excavation, a personal ar- chaeological undertaking in order to source materials for reference, comparison and interpretation. As an equally pleasurable consequence it also entailed recalling collab- orations, travels, events, expectations and informative disillusionments; and the many, beautifully diverse people encountered along the way. All these have inevitably fuelled creative practice; all these are indexed in one way or another by the author’s work.

While this may sound like a platitude, it helps to justify (or perhaps excuse) the fact that much was left unsaid, barely mentioned or meagerly addressed. An artwork may be experienced and interpreted from many different angles, its expressive power relying strongly on the nigh-boundless recombinant way in which it embodies different forms of knowledge. In covering a total of eight works, seven of which have artistic aspects to them, and three of which – Headbang Hero, Rambler and the Lorm Hand – have a decidedly critical or interventive agenda (and have achieved a reasonable level of popularity), a lot has to be left to interpretation and imaginative speculation. While this constitutes a challenge in the presentation and discussion of interactive artworks within a linear, static, passive textual discourse, it can also constitute an index of successful artistic expression.

Regardless, most (if not all) of the works can be seen to have achieved some evident degree of success in their own goals, in informing their successors and as part of a collective effort. They have reached an audience and enjoyed rapport, manifest in publications, exhibitions, reactions, invitations, coverage and awards (see Appendixes A, B and C).

180 | ART AND TECHNOLOGY AT PLAY It is hoped that this long yet pleasant endeavour may illustrate the value of playing with technology, spontaneously exploring its use as a means to understand the world, including ourselves in it. While contemporary products may be found lacking in addressing our embodied being-in-the-world (Heidegger’s Dasein), they are nevertheless (or perhaps as a consequence) ripe for experimentation, appropriation and détourne- ment. Things will never be truly perfect (or perhaps they already are, and have always been) but that may be a good thing. In the ever-shifting socio-technical panorama, there may always be more for us to play with. And as such, there may always be more to discover, which can only ever be discovered through play.

“God does not play dice with the universe; He plays an ineffable game of His own devising, which might be compared, from the perspective of any of the other players [i.e. everybody], to being involved in an obscure and complex variant of poker in a pitch-dark room, with blank cards, for infinite stakes, with a Dealer who won't tell you the rules, and who smiles all the time.”

- Terry Pratchett, Good Omens: The Nice and Accurate Prophecies of Agnes Nutter, Witch.

CONCLUSION | 181

REFERENCES

ANTONELLI, Paola (2011). “Talk to Me.” Talk to Me: Design and the Communication between People and Objects. Ed. Paola Antonelli. New York: The Museum of Modern Art. 6-17.

APTER, Michael J. (1991). “A Structural Phenomenology of Play.” Adult Play: A reversal Theory Approach. Ed. J. H. Kerr, Ed. M. J. Apter. Amsterdam: Swets & Zeitlinger. 13-29.

ARCHER, Nate (2010, April 6). “Rambler Shoes.” Designboom [online]. Retrieved from: http://www.designboom.com/technology/rambler-shoes/

BAIN, Marc (2009, December 28) “Q&A: Michael Haneke on Violence, The White Ribbon.” Newsweek Europe [online]. Retrieved from: http://europe.newsweek.com/qa-michael-haneke-violence-white- ribbon-75535

BARDZELL, Jeffrey and BARDZELL, Shaowen (2013). “What is ‘Critical’ about Critical Design?” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2013). New York: ACM. 3297- 3306.

BARNARD, Malcolm (2002). Fashion as Communication. Second Edition. New York: Routledge.

BARRETT, Brian Barrett (2010, January 26). “Rambler Sneakers Tweet Every... Single... Step.” Gizmodo [online]. Retrieved from: http://gizmodo.com/5457277/rambler-sneakers-tweet-every-single-step

BARTHES, Roland (2000). Mythologies. Originally published as “Mythologies” by Editions du Seuil in 1957. Translated by Jonathan Cape. London: Vintage Books.

BAUDRILLARD, Jean (1999). Revenge of the Crystal: Selected writings on the modern object and its destiny, 1968-1983. Edited and translated by Paul Foss and Julian Pefanis. London: Pluto Press

BAUDRILLARD, Jean (2005). The System of Objects. First published as “Le système des objets” by Editions Gallimard in 1968. Translated by James Benedict 1996. London: Verso.

BECKER, Becker (2002). Surpassing the spectacle: Global transformations and the changing politics of art. New York: Rowman & Littlefield.

BELL, Gordon and GEMMEL, Jim (2009). Total Recall: How the E-memory Revolution Will Change Everything. Later re-published as “Your Life, Uploaded: The Digital Way to Better Memory, Health, and Productivity”. New York: Dutton.

REFERENCES | 183 BIELING, Tom [claricebouvier] (2013, October 24). “Lorm Hand Demo (DESIGN RESEARCH LAB Berlin)” [video file]. Retrieved from: https://www.youtube.com/watch?v=UoPEgvlT5vA

BIELING, Tom [claricebouvier] (2013, October 30). “Lorm Hand (DESIGN RESEARCH LAB Berlin) [video file]. Retrieved from: https://www.youtube.com/watch?v=WsG0WbcgQq8

BIELING, Thomas, MARTINS, Tiago, JOOST, Gesche (2014). “A Tangible Interface for Information Exchange Addressing Deaf-Blind Users.” Proceedings of the International Conference on Universal Design (UD 2014). Lund: IOS Press. 439-440.

BIELING, Thomas, MARTINS, Tiago, JOOST, Gesche (2017). “Interactive Inclusive – Designing Tools for Activism and Empowerment”. Disability and Social Media. Ed. K. Ellis, Ed. M. Kent. New York: Routledge. 101-118.

BOGOST, Ian (2010). Persuasive Games: The Expressive Power of Videogames. First MIT Press paperback edition. Cambridge: The MIT Press

BOYD, Brian (2010). On the Origin of Stories – Evolution, Cognition and Fiction. Cambridge: Harvard University Press.

BROLL, Wolfgang, OHLENBURG, Jan, LINDT, Irma, HERBST, Iris, BRAUN, Anne-Kathrin (2006). “Meeting Technology Challenges of Pervasive Augmented Reality Games”. Proceedings of 5th ACM SIGCOMM workshop on Network and system support for games (NetGames '06). New York: ACM.

CAILLOIS, Roger (2001). Man, Play and Games. Originally published as “Les Jeux et les Hommes” by Librairie Gallimard in 1958. Translated by Meyer Barash 1961. Chicago: University of Illinois Press.

CHALMERS, Matthew, BARKHUUS, Louise, BELL, Marek, BROWN, Barry, HALL, Malcolm, SHERWOOD, Scott, TENNENT, Paul (2005). “Gaming on the Edge: Using Seams in Pervasive Games.” Proceedings of the Second International Workshop on Gaming Applications in Pervasive Computing Environments (PerGames 2005).

Company Info (n.d.). Facebook Newsroom [online]. Retrieved 16.01.2017 from: http://newsroom.fb.com/company-info/

CONNOLY, Kate, CHRISAFIS, Angelique, MCPHERSON, Poppy, KIRSCHGAESSNER, Stephanie, HASS, Benjamin, PHILLIPS, Dominic, HUNT, Elle, SAFI, MICHAEL (2016, December 2). “Fake news: an insidious trend that's fast becoming a global problem”. The Guardian [online]. Retrieved from: https://www.theguardian.com/media/2016/dec/02/fake-news-facebook-us-election-around-the-world

184 | ART AND TECHNOLOGY AT PLAY CORREIA, Nuno, ALVES, Luís, CORREIA, Helder, ROMERO, Luís, MORGADO, Carmen, SOARES, Luís, CUNHA, José C., ROMÃO, Teresa, DIAS, A. Eduardo, JORGE, Joaquim A. (2005). “InStory: a system for mobile information access, storytelling and gaming activities in physical spaces”. Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology (ACE ‘05). New York: ACM. 102-109.

CROAL, N’Gai Croal (2016, July 15). “Games as a Combination of Art Forms” [video file and transcript]. Retrieved from: http://www.criticalpathproject.com/video/games-as-a-combination-of-artforms/

DANIELS, Dieter (2008). “Strategies of Interactivity”. The Art and Science of Interface and Interaction Design (Vol. 1). Ed. Christa Sommerer, Ed. Lakhmi C. Jain, Ed. Laurent Mignonneau. Berlin: Springer-Verlag. 27-62.

DERRIDA, Jacques (1993). “Structure, Sign, and Play in the Discourse of the Human Sciences.” A Postmodern Reader. Ed. Joseph Natoli, Ed. Linda Hutcheon. Albany: State University of New York Press. 223-242.

Design Research Lab (2015, March 12). “SPEECHLESS (Lorm Hand + Lorm Glove) | Design Research Lab” [video file]. Retrieved from: https://www.youtube.com/watch?v=TW2FoVVrkEg

(Dourish 2004). Where the Action Is: The Foundations of Embodied Interaction. First paperback edition. Cambridge: The MIT Press.

DRAGONA, Daphne. "From Parasitism to Intitutionalism." Artists Re:thinking Games. Ed. Ruth Catlow, Ed. Marc Garrett, Ed. Corrado Morgana. Liverpool: FACT/Liverpool University Press. 26-32.

DUNNE, Anthony and RABY, Fiona (2001). Design Noir: The Secret Life of Electronic Objects. Basel: Birkhäuser.

DUNNE, Anthony and RABY, Fiona (2005). “Towards a Critical Design” [catalogue essay]. Le Design Aujourd’hui. Paris: Centre Pompidou.

DUNNE, Anthony and RABY, Fiona (2013). Speculative Everything: Design, Fiction, and Social Dreaming. Cambridge: The MIT Press

DUNNE, Anthony and RABY, Fiona (2014). “Critical Design FAQ.” Bunny Smash (exhibition catalogue). Tokyo: Museum of Contemporaray Art.

DUTTON, Denis (2010). “A Darwinian Theory of Beauty” [video file]. Retrieved from: http://www.ted.com/talks/denis_dutton_a_darwinian_theory_of_beauty

REFERENCES | 185 DUTTON, Denis (2013). “Aesthetic Universals.” The Routledge Companion to Aesthetics. Third edition. First published in 2000. Ed. Berys Gaut, Ed. Dominic McIver Lopes. New York: Routledge. 267-277.

ERMI, Laura and MÄYRÄ, Frans (2005). “Fundamental Components of the Gameplay Experience: Analysing Immersion.” Proceedings of the 2005 DiGRA International Conference: Changing Views: Worlds in Play.

FALK, Jennica and DAVENPORT, Glorianna (2004). “Live Role-Playing Games: Implications for Pervasive Gaming.” Proceedings of the Third International Conference on Entertainment Computing (ICEC 2004). Berlin: Springer Berlin Heidelberg. 127-138.

FISCHER, Joel E., LINDT, Irma, STENROS, Jaakko (2007). “Evaluation of Crossmedia Gaming Experiences in Epidemic Menace.” Proceedings of the 4th International Symposium on Pervasive Gaming Applications (PerGames 2007). Aachen: Shaker Verlag. 13-21.

FLUEGELMAN, Andrew (1976). The New Games Book. New York: The Headlands Press.

FORSTER, Winnie (Ed.) and FREUNDORFER, Stephan (2003). Joysticks: Eine illustrierte Geschichte der Game-Controller 1972-2004. Utting: Gameplan.

FOWLER, Chris (2004). The Archaeology of Personhood: An Anthropological Approach. London: Routledge.

FRASCA, Gonzalo (2007). Play the Message – Play, Game and Videogame Rhetoric. PhD Thesis. IT University of Copenhagen.

FRIELING, Rudolf (2008). “Toward Participation in Art.” The Art of Participation: 1950 to Now. Ed. Rudolf Frieling, Ed. Boris Groys, Ed. Robert Atkins, Ed. Lev Manovich. New York: Thames and Hudson. 32-49.

From Art to Arcade (n.d.). Retrieved from: http://www.painstation.de/history.html

FUJIMOTO, Kenichi (2006). “The Third-Stage Paradigm: Territory Machines from the Girls' pager Revolution to Mobile Aesthetics”. Personal, Portable, Pedestrian: Mobile Phones in Japanese Life. First paperback edition. Ed. Mizuko Ito, Ed. Daisuke Okabe, Ed. Misa Matsuda. Cambridge: The MIT Press. 77-101.

GELL, Alfred (1992). “The Technology of Enchantment and the Enchantment of Technology.” Anthropology, Art and Aesthetics. Ed. J. Coote, Ed. A. Shelton. Oxford: Clarendon. 40–66.

GELL, Alfred (1998). Art and Agency: An Anthropological Theory. Oxford: Clarendon Press.

186 | ART AND TECHNOLOGY AT PLAY GEMPERLE, Francine, KASABACH, Chris, STIVORIC, John, BAUER, Malcolm, MARTIN, Richard (1998). “Design for Wearability.” Proceedings of the Second IEEE International Symposium on Wearable Computers (ISWC). Los Alamitos (CA): IEEE Computer Society. 116-122.

GILMORE, J.Barnard (1971). “Play: A Special Behavior.” Child’s Play. Ed. R. E. Herron, Ed. B. Sutton-Smith. New York: John Wiley & Sons.

GOLLNER, Ulrike, BIELING, Tom, JOOST, Gesche (2012). “Mobile Lorm Glove: introducing a communication device for deaf-blind people.” Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (TEI 2012). New York: ACM Press. 127-130.

GOOD, Owen (2009, February 8). “Headbang Hero: Rock Out with Your Locks Out”. Kotaku [online]. Retrieved from: http://kotaku.com/5149108/headbang-hero-rock-out-with-your-locks-out

GOODMAN, Nelson (1987). “When is art?” Art and Its Significance: An Anthology of Aesthetic Theory. Ed. S. D. Ross. Albany: State University of New York Press.

GREENFIELD, Adam (2006). Everyware: The dawning age of ubiquitous computing. Berkeley: New Riders.

GROYS, Boris (2008). “A Genealogy of Participatory Art” The Art of Participation: 1950 to Now. Ed. Rudolf Frieling, Ed. Boris Groys, Ed. Robert Atkins, Ed. Lev Manovich. New York: Thames and Hudson. 18-31.

HARAWAY, Donna (1987). “A manifesto for cyborgs: Science, technology, and socialist feminism in the 1980s.” Australian Feminist Studies, Volume 2, issue 4.

Head Banger (2006). Retrieved from: http://game.speldesign.uu.se/projects/arcade-and-physical/head- banger/

HEMMERT, Fabian, HAMANN, Susann, LÖWE, Matthias, WOHLAUF, Anne, JOOST, Gesche (2010). “Weight- Shifting Mobiles: Motion and Tilt in One-Dimensional Gravitational Displays.” Proceedings of the Third International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services (CENTRIC ’10).

HEMMERT, Fabian, GOLLNER, Ulrike, LÖWE, Matthias, WOHLAUF, Anne, JOOST, Gesche (2010). “Intimate Mobiles: Near-Body Telepresence through Tightness, Wetness and Airflow in Mobile Phones.” Proceedings of the First International Workshop on Erotic Life in HCI (Erotic HCI ’10).

HEMMERT, Fabian, LÖWE, Matthias, WOHLAUF, Anne, JOOST, Gesche (2013). “Animate Mobiles: Proxemically Reactive Posture Actuation as a Means of Relational Interaction with Mobile Phones.” Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction (TEI ’13). New York: ACM. 267-270.

REFERENCES | 187 HERZ, J. C. (2001). “Gaming the System: What Higher Education Can Learn from Multiplayer Online Worlds.” The Internet and the University: Forum 2001. 169-191.

HJORTH, Larissa (2011). Games and Gaming: An Introduction to New Media. Oxford: Oxford International Publishers

HLUBINKA, Michelle, BEAUDIN, Jennifer, MUNGUIA, TAPIA, Emmanuel M., AN, John S. (2002). “AltarNation: Interface Design for Meditative Communities.” Extended Abstracts on Human Factors in Computing Systems (CHI '02). New York: ACM. 612-613.

HUIZINGA, Johan (1971). Homo Ludens: A Study of the Play-Element in Culture. Boston: Beacon Press.

HUMBLE, Rod (2006). “Game Rules as Art.” The Escapist Magazine [online], issue 41: “Can A Game Make You Cry?”, 18 April 2006. Retrieved from: http://www.escapistmagazine.com/articles/view/videogames/issues/issue_41/247-Game-Rules-as-Art

HUNT, Jamer Hunt (2011). “Nervous Systems and Anxious Infrastructures.” Talk to Me: Design and the Communication between People and Objects. Ed. Paola Antonelli. New York: The Museum of Modern Art. 48-55.

ISHII, Hiroshi Ishii and ULLMER, Brygg (1997). “Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms.” Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (CHI '97). 234-241.

ITO, Mizuko (2006). “Introduction: Personal, Portable, Pedestrian.“ Personal, Portable, Pedestrian: Mobile Phones in Japanese Life. First paperback edition. Ed. Mizuko Ito, Ed. Daisuke Okabe, Ed. Misa Matsuda. Cambridge: The MIT Press. 1-16.

JANKO, Siegbert, LEPOLDSEDER, Hannes, STOCKER, Gerfried (1996). Ars Electronica Center: Museum of The Future. Linz: Ars Electronica Center.

JONES, Jonathan (2012, November 30). “Sorry MoMA, video games are not art”. The Guardian [online]. Retrieved from: https://www.theguardian.com/artanddesign/jonathanjonesblog/2012/nov/30/moma- video-games-art

JONSSON, Staffan, WAERN, Annika, MONTOLA, Markus, STENROS, Jaako (2007). “Game Mastering a Pervasive LARP – Experiences from Momentum.” Proceedings of the 4th International Symposium on Pervasive Gaming Applications (PerGames 2007). Aachen: Shaker Verlag. 31-39.

KAPROW, Allan (2003). Essays on The Blurring of Art and Life. Expanded paperback edition. Ed. Jeff Kelley. Berkeley: University of California Press.

188 | ART AND TECHNOLOGY AT PLAY KATO, Haruhiro (2006). “Japanese Youth and the Imagining of Keitai”. Personal, Portable, Pedestrian: Mobile Phones in Japanese Life. First paperback edition. Ed. Mizuko Ito, Ed. Daisuke Okabe, Ed. Misa Matsuda. Cambridge: The MIT Press. 103-119.

KLUITENBERG, Eric 2006. The Book of Imaginary Media: Excavating the Dream of the Ultimate Communication Medium. Rotterdam: NAi Publishers.

KOHIYAMA, Kenji (2006). “A Decade in the Development of Mobile Communications in Japan.” Personal, Portable, Pedestrian: Mobile Phones in Japanese Life. First paperback edition. Ed. Mizuko Ito, Ed. Daisuke Okabe, Ed. Misa Matsuda. Cambridge: The MIT Press. 61-74.

KORDJAK-PIOTROWSKA, Joanna, Ed. (2013). Antonisz: Technology for Me Is a Form of Art. Warsaw: National Gallery of Art.

KOSTER, Raph (2005). A Theory of Fun for Game Design. Arizona: Paraglyph Press.

KUSAHARA, Machiko (2006). “Device Art: A New Form of Media Art from a Japanese Perspective.” Intelligent Agent, Volume 6, Number 2, Special Issue: ISEA 2006 Pacific Rim. Retrieved from: http://www.intelligentagent.com/archive/ia6_2_pacificrim_kusahara_deviceart.pdf

KUSAHARA, Machiko (2010). “Wearing Media. Technology, Popular Culture and Art in Japanese Daily Life.” Imaginary Japan: Japanese Fantasy in Contemporary Popular Culture. Ed. Eija Niskanen. Turku: International Institute for Popular Culture.

KWASTEK, Katja (2008). “Interactivity – A Word in Process.” The Art and Science of Interface and Interaction Design (Vol. 1). Ed. Christa Sommerer, Ed. Lakhmi C. Jain, Ed. Laurent Mignonneau. Berlin: Springer- Verlag. 15-26.

LAMONT, Peter and WISEMAN, Richard (1999). Magic in Theory: An Introduction to the Theoretical and Psychological Elements of Conjuring. Hertfordshire: Univ. of Hertfordshire Press.

LAVER, James (2012). Costume and Fashion: A Concise History. Fifth Edition. London: Thanes & Hudson

LAW, John (1992). “Notes on the Theory of the Actor Network: Ordering, Strategy and Heterogeneity.” Systems Practice, Volume 5, issue 4. Springer. 379–393.

LÉVI-STRAUSS, Claude (1966). The Savage Mind. Translated from the French “La Pensée savage” by George Weidenfield and Nicholson Ltd. Chicago: University of Chicago Press.

LINDT, Irma, OHLENBURG, Jan, PANKOKE-BABATZ, Uta, PRINZ, Wolfgang GHELLAL, Sabiha (2006). “Combining multiple gaming interfaces in epidemic menace.” Extended Abstracts on Human Factors in Computing Systems (CHI EA '06). New York: ACM. 213-218.

REFERENCES | 189 LISTER, Martin, DOVEY, Jon, GIDDINGS, Seth, GRANT, Iain, KELLY, Kieran (2008). New Media: A Critical Introduction. Second Edition. New York: Routledge.

“List of best-selling game consoles” (n.d.). Wikipedia [online]. Retrieved from: https://en.wikipedia.org/wiki/List_of_best-selling_game_consoles

MACKENZIE, Noella and KNIPE, Sally Knipe (2006). “Research dilemmas: Paradigms, methods and methodology.” Issues In Educational Research, Volume 16, Number 2. Western Australian Institute for Educational Research. 193-205.

MANN, Steve (1994). “Wearable, Tetherless, Computer-Mediated Reality.” Technical Report #260, M.I.T. Medial Lab Perceptual Computing Section. Cambridge, Massachusetts.

MANOVICH, Lev (2008). “Art after Web 2.0.” The Art of Participation: 1950 to Now. Ed. Rudolf Frieling, Ed. Boris Groys, Ed. Robert Atkins, Ed. Lev Manovich. New York: Thames and Hudson. 66-79.

MARTINS, Tiago [tmsmartins] (2009, February 8). “Headbang Hero - First Prototype” [video file]. Retrieved from: https://www.youtube.com/watch?v=LyvXDcY89NQ

MARTINS, Tiago [tmsmartins] (2009, March 16). “Headbang Hero - Documental Video 1” [video file]. Retrieved from: https://www.youtube.com/watch?v=qf7tXJd-6Po

MARTINS, Tiago, HEIDECKER, Christina, SOMMERER, Christa, CORREIA, Nuno (2007). “Designing a Wearable Interface for Pervasive Games.” Conference Poster. Proceedings of the 4th International Symposium on Pervasive Gaming Applications (PerGames 2007). Aachen: Shaker Verlag. 167-168.

MARTINS, Tiago, SOMMERER, Christa, MIGNONNEAU, Laurent, CORREIA, Nuno (2008). “Gauntlet: A Wearable Interface for Ubiquitous Gaming”. Proceedings of the 10th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '08). New York: ACM. 367-370.

MARTINS, Tiago, ROMÃO, Teresa, SOMMERER, Christa, MIGNONNEAU, Laurent, CORREIA, Nuno (2008). “Towards an Interface for Untethered Ubiquitous Gaming.” Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology (ACE ‘08). New York: ACM. 26-33.

MARTINS, Tiago, NASCIMENTO, Ricardo, ZINGERLE, Andreas, SOMMERER, Christa, MIGNONNEAU, Laurent, CORREIA, Nuno (2009). “Headbang Hero” (creative showcase). Proceedings of the 2009 International Conference on Advances in Computer Entertainment Technology (ACE ‘09). New York: ACM. 446.

190 | ART AND TECHNOLOGY AT PLAY MARTINS, Tiago, SOMMERER, Christa, MIGNONNEAU, Laurent, CORREIA, Nuno (2009). “Noon: A Secret Told By Objects” (creative showcase). Proceedings of the 2009 International Conference on Advances in Computer Entertainment Technology (ACE ‘09). New York: ACM. 454.

MCCONACHIE, Bruce (2011). “An Evolutionary Perspective on Play, Performance, and Ritual.” TDR Winter 2011, Volume 55, Number 4. 33-50.

MCGONIGAL, Jane (2006). This Might Be a Game: Ubiquitous Play and Performance at the Turn of the Twenty-First Century. PhD Thesis. University of California, Berkeley.

MCGONIGAL, Jane (2011). Reality Is Broken: Why Games Make Us Better and How They Can Change the World. London: Jonathan Cape.

MCLUHAN, Marshall (2001). Understanding Media. Routledge Classics first edition. First published in 1964. New York: Routledge.

MENDES, Mónica. ARTiVIS - Arts, real-time video and interactivity for sustainability. PhD Thesis. Faculdade de Ciências e Tecnologia, Universidade Nova de Lisboa.

MIDAL, Alexandra (2011). “Design Wonder Stories: When Speech is Golden.” Talk to Me: Design and the Communication between People and Objects. Ed. Paola Antonelli. New York: The Museum of Modern Art. 92-97.

MILLER, Daniel (2009). The Comfort of Things. First paperback edition. Cambridge, UK: Polity Press.

MOGGRIDGE, Bill (2007). Designing Interactions. Cambridge: The MIT Press.

MORGANA, Corrado (2010). “Introduction.” Artists Re:thinking Games. Ed. Ruth Catlow, Ed. Marc Garrett, Ed. Corrado Morgana. Liverpool: FACT/Liverpool University Press. 7-14.

MONTOLA, Markus, STENROS, Jaakko, WAERN, Annika (2009). Pervasive Games: Theory and Design. Burlington: Morgan Kaufmann/Elsevier

MUNSTER, Anna (2006). Materializing New Media. London: Dartmouth College Press.

NASCIMENTO, Ricardo [ricardo O'nascimento] (2010, April 12). “Rambler Shoes” [video file]. Retrieved from: https://vimeo.com/10868721

New Hacker’s Dictionary, The (August 20, 2000). Retrieved from: http://webyes.com.br/wp- content/uploads/ebooks/book_the_hacker_dictionary.pdf

REFERENCES | 191 NISHIO, Shuichi, ISHIGURO, Hiroshi, HAGITA, Norihiro (2007). “Geminoid: Teleoperated Android of an Existing Person.” Humanoid Robots: New Developments. Ed. Armando Carlos de Pina Filho. InTech Education and Publishing. 343-352.

NORMAN, Donald A. (2002). The Design of Everyday Things. Reprint, 2002. Originally published in 1988 as “The Psychology of Everyday Things”. New York: Basic Books.

OKADA, Tomoyuki (2006). “Youth Culture and the Shaping of Japanese Mobile Media: Personalization and the Keitai Internet as Multimedia.” Personal, Portable, Pedestrian: Mobile Phones in Japanese Life. First paperback edition. Ed. Mizuko Ito, Ed. Daisuke Okabe, Ed. Misa Matsuda. Cambridge: The MIT Press. 41-60.

OLSEN, Bjørnar (2010). In Defense of Things: Archaeology and the Ontology of Objects. Plymouth, UK: AltaMira Press.

O’REILLY, Tim (2005) “What Is Web 2.0 - Design Patterns and Business Models for the Next Generation of Software” [online]. Retrieved from: http://www.oreilly.com/pub/a/web2/archive/what-is-web- 20.html

O’REILLY, Tim and BATTELLE, John (2009). “Web Squared: Web 2.0 Five Years On” [online]. Retrieved from: http://www.kimchristen.com/wp-content/uploads/2015/07/web2009_websquared- whitepaper.pdf

Original Hacker's Dictionary, The (1998). Copy of the 1988 “jargon.txt” file made available by Paul Dourish. Retreived from: http://www.dourish.com/goodies/jargon.html

PATTON, Declan and MCINTOSH, Andrew (2008). “Head and neck injury risks in heavy metal: head bangers stuck between rock and a hard bass.” British Medical Journal [online]. Retrieved from: http://www.bmj.com/cgi/content/full/337/dec17_2/a2825

PENNER, Hans H. (1989). “Rationality, Ritual, and Science.” Religion, Science, and Magic: In Concert and in Conflict. Ed. Jacob Neusner, Ed. Ernest S. Frerichs, Ed. Paul V. M. Flesher. New York: Oxford University Press. 11-24.

PEREIRINHA, Tânia (2010). “Ténis que twittam sozinhos.” Revista Sábado, February 18, 2010. 91.

PICHLMAIR, Martin (2005). “Seven mile boots: implications of an everyday interface.” Proceedings of the 13th annual ACM international conference on Multimedia (MULTIMEDIA '05). New York: ACM. 574-577.

PINKER, Steven (1999). How the Mind Works. London: Penguin Books.

PINKER, Steven (2003). The Blank Slate: The Modern Denial of Human Nature. New York: Penguin Books.

192 | ART AND TECHNOLOGY AT PLAY PLOTKIN, Henry (1994). Darwin Machines and the Nature of Knowledge. Cambridge: Harvard University Press.

REEVES, Stuart, BENFORD, Steve, O’MALLEY, Claire, FRASER, Mike (2005). “Designing the Spectator Experience.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘05). New York: ACM. 741-750.

RODRIGUEZ, Hector (2006). “The Playful and the Serious: An approximation to Huizinga's Homo Ludens.” Game Studies, Volume 6, Issue 1. Retrieved from: http://gamestudies.org/0601/articles/rodriges

SACKS, Oliver (2012). Awakenings. First published in 1973. London: Picador.

SALAZAR (2010, January 27). “Geeky Rambler Twitter Sneakers Tweet Each Step One Takes!” Walyou [online]. Retrieved from: http://walyou.com/blog/2010/01/27/rambler-tweeter-sneakers/

SALEN, Katie and ZIMMERMAN, Eric (2004). Rules of Play: Game Design Fundamentals. Cambridge: The MIT Press.

SCARFE, Neville V. (1962). “Play is Education.” Childhood Education, Volume 39, Issue 3. London: Taylor & Francis. 117-121.

SCHECHNER, Richard (2002). Performance Studies: An Introduction. London: Routledge.

SCHILLER, Friedrich (1967). On the Aesthetic Education of Man, in a Series of Letters. Translated by Elizabeth M. Wilkinson and L.A. Willoughby. New York: Oxford University Press.

SCHÖN, Donald A. (1983). The Reflective Practitioner: How Professionals Think in Action. New York: Basic Books.

SCHRAGE, Michael (2013). “Crafting Interactions: The Purpose and Practice of Serious Play.” Prototype – Design and Craft in the 21st Century. Ed. Louise Valentine. London: Bloomsbury. 19-28.

SCRIVENER, Stephen (2002). “The art object does not embody a form of knowledge”. Working Papers in Art and Design. Retrieved from: http://ualresearchonline.arts.ac.uk/783/

SEAGO, Alex and DUNNE, Anthony (1999). "New Methodolgies in Art and Design Research: The Object as Discourse." Design Issues, Volume 15, Number 2. Cambridge: The MIT Press. 11-17.

SEYMOUR, Sabine (2008). Fashionable Technology: The Intersection of Design, Fashion, Science, and Technology. Vienna: Springer Verlag.

REFERENCES | 193 SEYMOUR, Sabine (2010). Functional Aesthetics: Visions in Fashionable Technology. Vienna: Springer Verlag

Seven Mile Boots (n.d.). Webpage. Retrieved from: http://randomseed.org/sevenmileboots/

SIMS, Alyssa (2016, December 16). “The Stock Market Needs to Gird Itself for Trump’s Tweets.” Slate – Future Tense [online]. Retrieved from: http://www.slate.com/articles/technology/future_tense/2016/12/the_stock_market_needs_to_gird_its elf_for_trump_s_tweets.html

SLAVIN, Kevin (2011). “Reality is Plenty, Thanks: Twelve Arguments for Keeping the Naked Eye Naked.” Talk to Me: Design and the Communication between People and Objects. Ed. Paola Antonelli. New York: The Museum of Modern Art.164-173.

SMITH, Joshua R., FISHKIN, Kenneth P., JIANG, Bing, MAMISHEV, Alexander, PHILIPOSE, Matthai, REA, Adam D., ROY, Sumit, SUNDARA-RAJAN, Kishore (2005). “RFID-based Techniques for Human-activity Detection.” Communications of the ACM, Volume 48, Number 9. New York: ACM. 39-44.

Social Mobiles (2002). Retrieved from: http://p-lin-app-df-01.ideo.com/work/social-mobiles/

SOMMERER, Christa (2015). Interview by Lucía Coll for the Two Cultures Observatory, Mètode. Retrieved from: http://metode.cat/en/The-Two-Cultures-Observatory/Christa-Sommerer-Crec-que-fins-i-tot-les- arts-estan-tornant-se-mes-cientifiques

SOMMERER, Christa and MIGNONNEAU, Laurent (2003). “Mobile Feelings.” Originally published in: CODE The Language of our Time, Ars Electronica 2003, Hantje Cantz Verlag, 258-261. Retrieved from: http://www.interface.ufg.ac.at/christa-laurent/WORKS/CONCEPTS/MobileFeelingsConcept.html

STENROS, Jaakko (2012). “In Defence of a Magic Circle: The Social and Mental Boundaries of Play.” Proceedings of 2012 International DiGRA Nordic Conference (DiGRA Nordic '12).

STENSLIE, Ståle (2010). Virtual Touch - A study of the use and experience of touch in artistic, multimodal and computer-based environments. PhD Thesis. The Oslo School of Architecture and Design.

STERLING, Bruce Sterling (2005). Shaping Things. Cambridge: The MIT Press.

STUART, Keith (2009). “Why Bill Wyman is wrong about guitar games.” The Guardian [online]. Retrieved from: https://www.theguardian.com/technology/2009/sep/16/rock-band-game-theory-column

SUCHMAN, Lucy A. (2007). Human-Machine Reconfigurations: Plans and Situated Actions. Second Edition. New York: Cambridge University Press

194 | ART AND TECHNOLOGY AT PLAY SUTTON-SMITH, Brian (2001). The Ambiguity of Play. Second printing. Cambridge: Harvard University Press.

THACKARA, John (2005). In the Bubble: Designing in a Complex World. Cambridge: The MIT Press.

THOMPSON, Christina (2006). “Art Practice as Research: A Review Essay”. International Journal of Education & the Arts. Retrieved from: http://ijea.asu.edu/v7r3/

THWAITES, Thomas (2012). “The Toaster Project” [online]. Retrieved from: http://www.thetoasterproject.org/page2.htm

TOBITA, Hiroaki (2013). “Wearable computing device.” US Patent Application 13/891,682, 21 November 2013.

TOMASELLO, Michael (2009). Why We Cooperate. Cambridge: The MIT Press.

Twitter Study – August 2009. White paper by Pear Analytics. Retrieved from: http://pearanalytics.com/wp-content/uploads/2012/12/Twitter-Study-August-2009.pdf

VALENTINE, Louise (2013). “Introduction.” Prototype – Design and Craft in the 21st Century. Ed. Louise Valentine. London: Bloomsbury. 1-17.

VINH, Khoi (2011). “Conversations with the Network.” Talk to Me: Design and the Communication between People and Objects. Ed. Paola Antonelli. New York: The Museum of Modern Art. 128-131.

VISEU, Ana (2005). Augmented Bodies: The Visions and Realities of Wearable Computers. PhD Thesis. Ontario Institute for Studies in Education, University of Toronto.

WALDROP, M. Mitchell (2016). “The chips are down for Moore's law.” International Weekly Journal of Science [online]. Retrieved from: http://www.nature.com/news/the-chips-are-down-for-moore-s-law- 1.19338

WEISER, Mark (1991). “The computer for the twenty-first century.” Scientific American, Volume 265, Number 3. 94-104.

WILLIAMS, Robin A. and EDGE, David (1996). “The Social Shaping of Technology.” Research Policy, Volume 25, Issue 6. 865-899.

WILLIAMS, Raymond (1990). Television: Technology and Cultural Form. Second Edition. First published by Fontana, London in 1974. London: Routledge.

REFERENCES | 195 WILLIAMS, Sam (2002). “Appendix B: Hack, Hackers, and Hacking”. Free as in Freedom: Richard Stallman's Crusade for Free Software. Sam Williams. O’Reilly Media. Retrieved from: http://oreilly.com/openbook/freedom/appb.html

WILSON, Frank R. (1999). The Hand: How Its Use Shapes the Brain, Language, and Human Culture. New York: Vintage Books.

WILSON, Margaret (2002). “Six views of embodied cognition.” Psychonomic Bulletin & Review, volume 9, number 4. 625-636.

ZINGERLE, Andreas (2010, Jun 16). “Headbanghero @ Aucard de Tours 2010” [video file]. Retrieved from https://www.youtube.com/watch?v=QGPGxmbzp4o

ŽIŽEK, Slavoj (2008). Violence: Six Sideways Reflections. New York: Picador

ZWICK, Carola (2006). Designing for Small Screens. Lausanne: AVA Publishing

196 | ART AND TECHNOLOGY AT PLAY APPENDIX A · Exhibitions and Distinctions

This appendix contains a list of exhibitions and distinctions for each of the works, ordered chronologically; followed by a list of distinctions, ordered chronologically.

Noon - A Secret Told By Objects

Campus 2.0 Neoanalog, Ars Electronica Festival 2007, Linz, Austria, September 5 to 11, 2007.

Lange Nacht der Forschung (Long Night of Research), University of Art and Industrial Design Linz. Linz, Austria, September 26, 2008.

ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE ‘09). Athens, Greece, October 29 to 31, 2009. Headbang Hero

Speculum Artium 2009. Trbovlje, Slovenia, May 12 to 15, 2009.

The Royal Interface Cultures Masquerade Ball, Ars Electronica Festival 2009. Linz, Austria, September 3 to 8, 2009

ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE ‘09). Athens, Greece, October 29 to 31, 2009.

Festival Musiques Volantes. Metz, France, November 7, 2009.

Campus Party Mexico. Mexico City, Mexico, November 12 to 16, 2009.

Campus Party Brasil. São Paulo, Brasil, January 25 to 31, 2010.

Japan Media Arts Festival 2010 (documentation only). Tokyo, Japan, February 13 to 14, 2010.

Festival Aucard de Tours. Tours, France, June 10 to 13, 2010.

Festival de Arte Digital 2010. Belo Horizonte, Brazil, September 2 to October 3, 2010.

Playface Intercult. quartier21 – MuseumsQuartier. Vienna, Austria, April 20 to May 8, 2011.

Kiblix/MFRU 2013. Maribor, Slovenia, November 6-11, 2013. Rambler

Playful Interface Cultures, Ars Electronica Festival 2010. Linz, Austria, September 2 to 11, 2010.

Amber Festival 2010. Istanbul, Turkey, November 5 to 14, 2010.

404 Festival 2010. Taipei, Taiwan, November 25 to December 8, 2010.

Transmediale 11. Berlin, Germany, February 1 to 6, 2011.

Enter 5 Datapolis. Prague, Czech Republic, April 14 to 17, 2011.

APPENDIX A · Exhibitions and Distinctions | 197 Playface Intercult. quartier21 – MuseumsQuartier. Vienna, Austria, April 20 to May 8, 2011.

Wear It Festival 2014. Berlin, Germany, October 11-12, 2014. Weltschmerz

Speculum Artium 2011. Trbovlje, Slovenia, April 20 to 23, 2011.

Unuselesness, Ars Electronica Festival 2011. Linz, Austria, August 31 to September 6, 2011.

Apparatus:Mikkel=0, Galerija Media Nox. Maribor, Slovenia, February 13 to March 13, 2013 Wcielenie

3rd ArtBoom Festival. Krakow, Poland, June 10 to 24, 2011.

Art of Code Exhibition (device and map with tags). Kungsträdgården, Stockholm, March 22 to 25, 2012.

Iberofest 2016 – Remote Signals (documentation only). Talinn, Estonia, March 14-20, 2016 Tripo

Water Tower Art Fest. Sofia, Bulgaria, June 19-23, 2013.

Kiblix/MFRU 2013. Maribor, Slovenia, November 6-11, 2013. roBOt Festival 07. Bologna, Italy, October 1-5, 2014.

Push Conference 2015 Exhibition. Munich, Germany, October 24-24, 2015.

Speculum Artium 2015. Trbovlje, Slovenia, October 15-17, 2015.

Berlin Design Night – IXDS Exhibition. Berlin, Germany, June 3, 2016. Lorm Hand

Aktion Taubblind - Taubblinde in Isolationshaft. Berlin, Germany, October 4 , 2013.

Leben mit Taubblindheit Kongress. Potsdam, Germany, September 2014.

Medientage München. Munich, Germany, October 2014.

Das Netz. Menschen, Kabel, Datenströme. Deutsches Technik Museum. Berlin, Germany, September - October 2015.

GLOBALE: Exo-Evolution. Zentrum für Kunst und Medientechnologie (ZKM). Karlsruhe, Germany, October 2015 to March 2016.

Culture Interface. Cité du Design. St. Etienne, France, October 2015 to August 2016.

DMY International Design Festival. Berlin, Germany, June 2-5, 2016.

London Design Biennale, German Pavillion. London, United Kingdom, September 7-27 2016.

198 | ART AND TECHNOLOGY AT PLAY Distinctions

Jury Recommended Work (Entertainment Division). Japan Media Arts Festival. For the work Headbang Hero in 2010.

Best of Interface Cultures. Playful Interface Cultures - Ars Electronica Festival. For the work Rambler in 2010.

Winning Selection (2nd Place). Hochsculwettbewerb - Mehr als Bits & Bytes. For the work Lorm Hand in 2014.

“Design Shapes the World” Award. Menschen Bewegen - Lange Nacht der Ideen. For the work LormHand in 2016.

APPENDIX A · Exhibitions and Distinctions | 199

APPENDIX B · Relevant Publications

This appendix contains a list of publications by the author, which were relevant to the context of research. These are ordered chronologically.

2007 MARTINS, Tiago, HEIDECKER, Christina, SOMMERER, Christa, CORREIA, Nuno (2007). “Designing a Wearable Interface for Pervasive Games.” Conference Poster. Proceedings of the 4th International Symposium on Pervasive Gaming Applications (PerGames 2007). Aachen: Shaker Verlag. 167-168.

2008 MARTINS, Tiago, CORREIA, Nuno, SOMMERER, Christa, MIGNONNEAU, Laurent (2008). “Ubiquitous Gaming Interaction: Engaging Play Anywhere.” The Art and Science of Interface and Interaction Design. Ed. C. Sommerer, Ed. L. C. Jain, Ed. L. Mignonneau. Berlin: Springer. 115-130.

MARTINS, Tiago, CORREIA, Nuno, SOMMERER, Christa, MIGNONNEAU, Laurent (2008). “Towards an Interface for Untethered Ubiquitous Gaming.” Proceedings of the ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE ’08). New York: ACM. 26-33.

2009 MARTINS, Tiago, NASCIMENTO, Ricardo, ZINGERLE, Andreas, SOMMERER, Christa, MIGNONNEAU, Laurent, CORREIA, Nuno (2009). “Headbang Hero” (creative showcase). Proceedings of the 2009 International Conference on Advances in Computer Entertainment Technology (ACE ‘09). New York: ACM. 446.

MARTINS, Tiago, SOMMERER, Christa, MIGNONNEAU, Laurent, CORREIA, Nuno (2009). “Noon: A Secret Told By Objects” (creative showcase). Proceedings of the 2009 International Conference on Advances in Computer Entertainment Technology (ACE ‘09). New York: ACM. 454.

APPENDIX B · Relevant Publications | 201 2014 BIELING, Thomas, MARTINS, Tiago, JOOST, Gesche (2014). “A Tangible Interface for Information Exchange Addressing Deaf-Blind Users.” Proceedings of the International Conference on Universal Design (UD 2014). Amsterdam: IOS Press. 439-440.

2017 BIELING, Thomas, MARTINS, Tiago, JOOST, Gesche (2017). “Interactive Inclusive – Designing Tools for Activism and Empowerment”. Disability and Social Media. Ed. K. Ellis, Ed. M. Kent. New York: Routledge. 101-118.

202 | ART AND TECHNOLOGY AT PLAY APPENDIX C · Press Clippings

Gauntlet in Il Sole 24 Ore (IT), 18 September 2008.

APPENDIX C · Press Clippings | 203 Headbang Hero in M! Games (DE), May 2009.

204 | ART AND TECHNOLOGY AT PLAY Headbang Hero in Linz & Linz-Land Rundschau (AT), 6 September 2009.

APPENDIX C · Press Clippings | 205 Rambler in Revista Sábado (PT), February 2010.

206 | ART AND TECHNOLOGY AT PLAY Headbang Hero in Imagine (BR), March 2010

APPENDIX C · Press Clippings | 207 Wcielenie in Architektura & Biznes (PL). September 2011.

208 | ART AND TECHNOLOGY AT PLAY

CC BY-NC-ND 3.0 AT Namensnennung - Nicht-kommerziell - Keine Bearbeitung 3.0 Österreich