Interactive Media Shaleph O’Neill Interactive Media The Semiotics of Embodied Interaction Shaleph O’Neill, PhD, BA University of Dundee Scotland ISBN 978-1-84800-035-3 e-ISBN 978-1-84800-036-0 DOI: 10.1007/978-1-84800-036-0 British Library Cataloguing in Publication Data Library of Congress Control Number: 2008928135 © Springer-Verlag London Limited 2008 Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use. The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made. Printed on acid-free paper 9 8 7 6 5 4 3 2 1 Springer Science + Business Media springer.com Acknowledgements I sincerely thank Professor David Benyon for his continued support of this project and for his crucial feedback in the early stages of writing this book. I also thank my brother Dr. Nathan O’Neill for his penetrating critique of early drafts of some chapters and his keen editorial eye. Thanks also to the staff at Springer who have supported this project and helped make it happen. A further thanks extends to all my colleagues at the University of Dundee who are involved in running the Interactive Media Design program. Last but not least, a special thanks to my wife Linda for her continued support over the long months it has taken to see this project come to fruition. v Contents 1 Introduction ............................................................................................... 1 1.1 Brave New Media World .................................................................... 1 1.2 McLuhan and the Environmental Thesis ........................................... 2 1.3 The Anti-Determinist View ................................................................ 3 1.4 On-line Social Networks .................................................................... 4 1.5 The Need for New Theory ................................................................. 5 1.6 The Aim of this Book ......................................................................... 6 References ................................................................................................... 7 2 Media, Mediation and Interactive Media ............................................... 9 2.1 The Root of Media? ........................................................................... 9 2.2 Two Different Ways to Think about Media ........................................ 10 2.2.1 Artistic and Creative Media ................................................... 11 2.2.2 Communications Media ......................................................... 13 2.3 The Rise of Interactive Media ............................................................ 14 2.4 Remediation ....................................................................................... 16 2.4.1 Immediacy .............................................................................. 17 2.4.2 Hypermedia ............................................................................ 18 2.5 The Essential Characteristics of Interactive Media ............................ 19 2.5.1 The Technological Convergence of Multiple Media Forms .......................................................................... 20 2.5.2 The Digitization, Abstraction and Simulation of Old Media .......................................................................... 22 2.5.3 The Interactive Authoring and Interpretation of Meaning ..... 23 2.6 Summary ............................................................................................ 25 References ................................................................................................... 26 3 Approaching Interaction .......................................................................... 27 3.1 Interactive Media Today ..................................................................... 27 3.2 Human Computer Interaction ............................................................. 28 3.2.1 The Conventional View of Perception and Cognition ............ 29 vii viii Contents 3.2.2 Mental Modeling .................................................................... 30 3.2.3 The Human Processor Model ................................................. 31 3.2.4 Execution and Evaluation ....................................................... 32 3.2.5 The Changing Face of HCI .................................................... 32 3.3 Phenomenology and HCI ................................................................... 33 3.3.1 Being-In-The-World ............................................................... 34 3.3.2 Ready-To-Hand Vs Present-At-Hand..................................... 35 3.3.3 Computers and Cognition ...................................................... 36 3.3.4 Embodiment ........................................................................... 37 3.4 Semiotics and HCI ............................................................................. 39 3.4.1 Computer Semiotics ............................................................... 39 3.4.2 Semiotic Engineering Research Group (Serg) ...................... 41 3.4.3 Current Semiotic Approaches in HCI .................................... 42 3.5 Summary ............................................................................................ 43 References ................................................................................................... 45 4 Affordance: A Case of Confusion ............................................................ 49 4.1 The Many Faces of Affordance .......................................................... 49 4.1.1 Gibson’s Original Concept of Affordance ............................. 49 4.1.2 Norman’s Concept of Affordance .......................................... 51 4.1.3 Affordance in Technology Design ......................................... 52 4.1.4 Clarifi cation Or Further Confusion? ...................................... 54 4.1.5 Affordance in Information Systems ....................................... 56 4.2 Re-Evaluating Gibson’s Original Concept of Affordance ................. 58 4.2.1 Saving Gibson from ‘The Argument from Illusion’ .............. 59 4.2.2 Distinguishing between Direct and Mediated Perception ............................................................................... 61 4.2.3 Providing an Adequate Theory of Knowledge as Skill Acquisition ................................................................ 62 4.3 Summary ............................................................................................ 64 References ................................................................................................... 65 5 Semiotic Theory ........................................................................................ 67 5.1 Signs and Signifi cation ....................................................................... 67 5.1.1 Peircean Semiotics ................................................................. 68 5.2 Context and Cultural Codes ............................................................... 70 5.3 The Structure of Texts ........................................................................ 73 5.3.1 Layers of Meaning ................................................................ 74 5.4 Communication .................................................................................. 76 5.5 Semiotics and Interactive Media ........................................................ 77 5.6 Summary ............................................................................................ 80 References ................................................................................................... 81 Contents ix 6 Semiotics and Screen Based Interaction ................................................. 83 6.1 The Semiotic Screen .......................................................................... 83 6.1.1 Screen-based Media ............................................................... 83 6.1.2 Graphics, Symbols, and Pictograms....................................... 86 6.1.3 Visual Grammar ..................................................................... 89 6.1.4 Moving Images ....................................................................... 92 6.2 Case Study: Remediating Creativity .................................................. 95 6.2.1 Painting as Interaction ............................................................ 95 6.2.2 Interaction as Painting ............................................................ 97 6.2.3 Zones of Interaction ............................................................... 99 6.2.4 Sense Making During Interaction .......................................... 101 6.2.5 Discussion .............................................................................