EXPLORING A SEMIOTICS OF NEW MEDIA A thesis submitted in partial fulfilment of the requirements of Napier University for the degree of Doctor of Philosophy in Computer Science Shaleph John O'Neill School of Computing Napier University Edinburgh, UK 11/02/2005 In memory of my father 2 Table of contents List of Figures ...............................................................................................2 List of Tables ................................................................................................2 Publications ..................................................................................................2 Acknowledgments ........................................................................................2 Abstract .........................................................................................................2 Introduction ...................................................................................................2 Research Questions ........................................................................................2 Question 1............................................................................................................................. 2 Question 2............................................................................................................................. 2 Question 3............................................................................................................................. 2 The contribution to knowledge made in this thesis .......................................... 2 The Thesis Structure .......................................................................................2 Chapter 1 New media and HCI ............................................................................................. 2 Chapter 2 Semiotic Theory ................................................................................................... 2 Chapter 3 Applied Semiotics ................................................................................................. 2 Chapter 4 The Semiotic Analysis of Three Mobile Phones ................................................... 2 Chapter 5 An Integrated Semiotic Model of Interaction with New Media ............................... 2 Chapter 6 Methods ............................................................................................................... 2 Chapter 7 Interacting with the Hypermedia of Photos hop ..................................................... 2 Chapter 8 Interacting with the Immediacy of Virtual Environments ....................................... 2 Chapter 9 Conclusions .......................................................................................................... 2 Summary .........................................................................................................2 1 New Media and HCI .................................................................................2 1.1 The Avant-Garde and The Birth of New Media ....................................... 2 1.2 The Computer as Medium ...................................................................... 2 1.2.1 An Artistic Medium? .................................................................................................. 2 1.2.2 A Mass Medium? ...................................................................................................... 2 1.2.3 A New Medium? ....................................................................................................... 2 1.3 The Characteristics of New Media .......................................................... 2 1.3.1 From Old to New Media ............................................................................................ 2 1.3.2 The Digitisation and Abstraction of Older Media Forms ............................................ 2 1.3.3 The Convergence of Multiple Media Forms and Technology .................................... 2 1.3.4 The Interactive Production and Interpretation of Meaning ........................................ 2 1.3.5 Two Types of New media ......................................................................................... 2 1.4 Human Computer Interaction .................................................................. 2 1.4.1 Cognitive Psychology and HCI ................................................................................. 2 1.4.2 Conceptual Modelling ............................................................................................... 2 1.4.3 Execution/Evaluation Cycle ...................................................................................... 2 1.4.4 HCI and New Media .................................................................................................. 2 1.5 The Phenomenology of Being ................................................................. 2 1.5.1 Ready-to-hand Vs Present-at-hand .......................................................................... 2 1.5.2 Authentic Vs Inauthentic Being ................................................................................. 2 1.5.3 Phenomenology and HCI .......................................................................................... 2 1.5.4 Embodied Interaction ................................................................................................ 2 1.6 Semiotics ................................................................................................2 1.6.1 Semiotics and HCI .................................................................................................... 2 1.6.2 Semiotics and New Media (Narrowing the focus) ..................................................... 2 2 Semiotic Theory ......................................................................................2 2.1 Basic Semiotic Theory ............................................................................ 2 3 2.1.1 Ferdinand de Saussure ............................................................................................ 2 2.1.2 Saussure's Speech Circuit. ....................................................................................... 2 2.1.3 Syntagmatic and Paradigmatic Dimensions .............................................................. 2 2.1.4 Charles Sanders Peirce ............................................................................................ 2 2.1.5 Icon, Index, Symbol .................................................................................................. 2 2.2 Further Semiotic Theory ......................................................................... 2 2.2.1 Louis Hjelmslev ......................................................................................................... 2 2.2.2 Denotation, Connotation and Metaphor .................................................................... 2 2.2.3 Roland Barthes ......................................................................................................... 2 2.2.4 From Transparent Communication to Opaque Mediation ......................................... 2 2.2.5 Umberto Eco ............................................................................................................. 2 2.2.6 Codes and Social Semiotics ..................................................................................... 2 2.3 The Relevance of Semiotic Concepts to New Media .............................. 2 2.4 Conclusion ..............................................................................................2 3 Applied Semiotics ...................................................................................2 3.1 Computer Semiotics ............................................................................... 2 3.1.1 Semiotic Engineering Research Group (SERG) ....................................................... 2 3.2 Cinema ...................................................................................................2 3.3 Visual Semiotics .....................................................................................2 3.3.1 Visual Grammar ........................................................................................................ 2 3.4 The Semiotics of Graphics ......................................................................2 3.4.1 Graphics and Pictograms ......................................................................................... 2 3.4.2 Tufte ......................................................................................................................... 2 3.5 Maps and Wayfinding ............................................................................. 2 3.6 Architectural semiotics ............................................................................ 2 3.7 Product semiotics ...................................................................................2 3.8 Advertising ..............................................................................................2 3.9 Towards a Semiotics of New Media ........................................................ 2 3.10 Defining what is Relevant to a Semiotics of New Media ....................... 2 3.10.1 Codes ..................................................................................................................... 2 3.10.2 Denotation, Connotation and Metaphor .................................................................. 2 3.10.3 Rhetoric .................................................................................................................