Tangible Bits

Beyond Ubiquitous GUIs

2004 Tokyo

1 Tangible Bits

Designing the Seamless Interface between People, Bits, and Atoms

February 2004

2 Ubiquitous Computing Mark Weiser, Xerox PARC, 1991

• Computers should be "transparent." • Computational services are delivered through a variety of computational devices such as Tabs, Pads, and Boards, with the infrastructure to allow these devices to talk with each other.

Mark Weiser at Xerox PARC presented his vision of “Ubiquitous Computing” in 1991. He claimed computers should be transparent and invisible.

He claimed that the most profound technologies are invisible. They weave themselves into the fabric of everyday life until they are indistinguishable from it. His team designed a variety of computational devices including Tabs, Pads, and Boards along with the infrastructure to allow these devices to talk with each other.

Recently, he is introducing a concept of “calm technology” citing the LIVE WIRE by Natalie Jeremijenko as an example.

3 Misunderstood Ubiquitous

• Anytime & Anyplace • Mobile, wireless, broadband, RFID, …. • Multiple Computational Devices / User – c.f. Mainframe (TSS) computing: one computer / many users Personal computing: one computer / user

Mark Weiser at Xerox PARC presented his vision of “Ubiquitous Computing” in 1991. He claimed computers should be transparent and invisible.

Recently, he is introducing a concept of “calm technology” citing the LIVE WIRE by Natalie Jeremijenko as an example.

4 Wieser wrote to me in 1997… Date: Sun, 26 Jan 1997 23:34:10 PST Subject: "Tangible Bits" Dear Hiroshi and Brygg,

I recently had a chance to read your CHI 97 paper "Tangible Bits"! Great work! In my opinion thisi s the k i nd of work that will characterize the technological landscape in the twenty-first century...... My request is that you help me stop the spread of misunderstanding of ubiquitous computing based si mply on its name. Ubicomp was never just about making "computers" ubiquitous. It was always, like your work, about awakening computation mediation into the environment...... I tried to stop using ubiquitous computing because of its misleading implication, but it keeps cropping up again, so I keep returning to it as my umbrella name for lots of work, including Things That Think. Augmented reality was in use for awhile, but again got balkanized in meaning. I have started to talk about Calm Technology as a theme, but it better names a goal than a research project. "Tangible Bits" is very ni ce, and maybe could serve as an overall umbrella, but then you might lose it as the name of your research project! ...... -mark (Dr.) Mark Wei ser Chief Technologist, Xerox PARC © 2004 MIT Media Laboratory, Hiroshi Ishii

5 LIVE WIRE Natalie Jeremijenko, 1995

• 8 foot piece of plastic rope that hangs from a small electric motor mounted in the ceiling. The motor is linked to a nearby LAN, so that passing bits cause twitching of the motor.

• Bits flowing through the wires of a computer network become tangible through motion, sound, and touch.

• Mark Wieser called “Calm Technology”

Natalie Jeremijenko designed a beautiful instrument called Live Wire while an artist in residence at Xerox PARC. It is a piece of plastic cord that hangs from a small electric motor mounted on the ceiling. The motor is linked to the Ethernet, and each passing digital packet causes a tiny twitch of the motor.

Bits flowing through the wires of a computer network become tangible through motion, sound without being obtrusive, taking advantage of peripheral cues. This work encouraged us to think about ambient media as a general mechanism for displaying activities in cyberspace.

6 “The Computer for the 21st Century”

“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”

Mark Weiser July 23, 1952 - April 27, 1999

Mark Weiser wrote as follows in the 1st paragraph of his landmark paper on Ubiquitous Computing published in 1991 Scientific America . “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.” If you learn the task quite well using the right tools, and become the master, the tools cease to be a center of users’ attention. �Tools disappear into the background and you can concentrate on the tasks themselves. This is the definition of transparent or invisible computers by Mark. Invisible does not mean you can not see with your eyes. It is the matter of user’s focus of attention and consciousness. It is the state in which tools do not get in your way and you can concentrate on the task. Mark showed a series of ubiquitous computers including Live Board, Pads, and Badges. All of them come with screens and GUIs. But I do not feel Ubiquitous GUI is the path to achieve invisible interface. Today, I would like to show my approach: Tangible Interfaces.

7 At the Border

Where the land meets the sea, there is a border.

Where the land meets the sea, there is a border. This borderline is constantly moving and changing the shape of the shore. However, it is not only a simple line of a landscape, but also a critical biological locale.

8 Living at the Border

Harsh, but also fertile environment.

Whenever the tides go in and out, the inhabitants in the intertidal zone experience harsh environmental changes. When the tidal animals are exposed, they must withstand the heat and drying. When they are submerged, they must face constant pounding of waves. Living at this border is hard, but it is also a fertile environment for many kinds of life and activity. Where the land meets the sea, countless creatures have evolved diverse and creative approaches for flourishing among the turbulence and constant changes of these competing worlds. Our ancestors came from the sea and landed crossing this border a million years ago.

9 At the Border between Physical and Digital

We live on the border where bits meet atoms. In the flood of pixels from the ubiquitous GUI screens, we are losing our sense of body and places. Pixels impoverish human senses.

We are now crossing another border. Where bits meet atoms, there is another shoreline. Today, we live on the boundary between physical space and cyberspace. The constant pounding by waves of bits from screens distorts our experience of space. People are losing their sense of body and places in the flood of pixels from the ubiquitous rectangular screens. Living on this border is hard because our bodies remain in the physical world, even while we are immersed within digital information. It is also hard because current interfaces are not designed for amphibians. We are half drowning in the sea of bits, squeezing plastic mice instead of breathing and swimming. People are standing at the water's edge at a loss.

10 At the Boundary

Where the sea meets the land, life has blossomed into a myriad of unique forms in the turbulence of water, sand, and wind.

At another seashore between the land of atoms and the sea of bits, we are now facing the challenge of reconciling our dual citizenships in the physical and digital worlds. Windows to the digital world are confined to flat square screens and pixels, or "painted bits." Unfortunately, one can not feel and confirm the virtual existence of this digital information through one's body.

Tangible Bits seeks to realize seamless interfaces between humans, digital information, and the physical environment by giving physical form to digital information and computation, making bits directly manipulable and perceptible.

Hiroshi Ishii | Tangible Media Group | MIT Media Lab

11 • Seamless Interfaces for the Boundaries • Seamless Collaboration Media (88-94) at NTT Human Interface Laboratories • Beyond “Being There” • CSCW (Computer-Supported Cooperative Work) communities: ACM SIGOIS, SIGCHI, IFIP • Tangible User Interfaces (96-00) at Tangible Media Group, MIT Media Laboratory • Beyond “Painted Bits” • HCI (Human-Computer Interactions) communities: ACM SIGCHI and SIGGRAPH

12 My Art Work in 1959

Let me show another art work. This is my drawing on a fridge door in 1959. I loved drawing. Since then, drawing has been a very important medium in which I think, express myself and communicate with my mother.

13 Goal of ClearBoard Design

• Seamless Integration of Shared Workspace and Interpersonal Space • Natural and smooth transition between them using everyday cues

Shared Workspace Interpersonal Space

ClearBord, the shared drawing media I designed, is deeply rooted in my drawing practices and my liking for the whiteboard.

Our goal was to create a collaboration medium that seamlessly integrated the IPS and the SWS.

We designed ClearBoard to allow users to shift easily between these spaces by using familiar everyday cues such as the partner's gestures, head movements, eye contact, and gaze direction.

14 Beyond “Being There” Beyond “Talking Heads”

• Tacit Goal of Telecommunication: Imitation of "Being There" over distance – But this goal is a mirage. – It can never be achieved 100%. • New Goal: Creation of added value that an ordinary face to face meeting cannot provide – e.g. Virtual Shared Workspace

Most current video telephony systems have been designed to see "talking heads," and to create an illusion as if all the participants sit in an imaginative conference room. But this goal to simulate as close as to the real face to face meeting, however, it is something like a mirage. It can never be achieved 100%. The gap between the recognizable reality and virtuality cannot be filled by technology. As long as we try to find the raison d'etre of communication media only in its being "tele", we shall never be able to solve this contradiction. Conventional logic that it saves travel expenses and time should be reviewed. Our goal is to go beyond this model and demonstrate new usage of video communication technologies. To do so, we have integrated groupware technology and real-time video communication technology to go beyond the imitation of physical proximity. Instead of pursuing the illusion of "Being there", we should focus on the creation of added value which cannot be obtained from face to face conversation.

15 Beyond Multimedia

• Multimedia = more than 3 cables behind a computer? • "Multi-ness" is not main issue; rather seamless media integration which hides – low level representations and – complexity of underlying technologies. • Multimedia = a premature label that represents a stage of media evolution from the mono-media to the seamless media.

"Multimedia" is now becoming a big buzz word in the computer and communication industries. As a result, the number of cables behind a computer, the number of features users need to understand, and the number of incompatible data formats are increasing beyond the limits of human cognitive capability.

The communication channels of human beings are inherently multi-modal and seamless. It does not make much sense to decompose the representation of information into primitive data types such as text, video, audio, and graphics, and stress the "multi-ness" of the media. For example, we are speaking, gesturing, and drawing simultaneously in a design meeting. We have great skills to express ideas and understand each other in everyday contexts using all these media as a whole. We believe the multi-ness of media is not the main issue, but how to integrate them into a seamless media hiding the various low level representations, discontinuities among primitive media, and complexity of underlying technologies is the core issue in designing new applications.

"Multi-media" sounds like a premature label that represents a stage of media evolution from the mono-media to the seamless media.

16 Vision Video “Seamless Media Design”

• TED4 Kobe (94), Door of Perception 2 (94)

17 PDA in 1958 - Abacus

When I was 2 years old, I started playing with a PDA called the "abacus". It was more than my mother's financial calculator. The abacus became my musical instrument, imaginary toy train, even back-scratcher. I enjoyed the sound and tactile interaction with this simple physical artifact. The abacus also became a medium of awareness. When my mother used the abacus to manage household accounts, I learned she could not play with me while her abacus made its music. My childhood abacus is suggesting us a future direction of HCI. That is the reason why I started the Tangible Bits project at MIT.

18 Eyes are in charge, but hands are underemployed.

By pointing, by pushing and pulling, by picking up tools, hands act as conduits through which we extend our will to the world. They serve also as conduits in the other direction: hands bring us knowledge of the world. Hands feel. They probe. They practice.

Malcolm McCullough “Abstracting Craft: The Practiced Digital Hand ” 1996

Eyes are fully employed, but hands are underusedHands are under employed. GUIs (Graphical User Interfaces) are designed for human eyes, not for hands. Telepresence, or interpersonal communication, is not the exception. Communication media such as telephones, video conferencing, and e-mail are for human ears and/or eyes, but not for hands. McCullough articulated the importance of hands as follows, “By pointing, by pushing and pulling, by picking up tools, hands act as conduits through which we extend our will to the world. They serve also as conduits in the other direction: hands bring us knowledge of the world. Hands feel. They probe. They practice.” [McCullough, 1996 #1].

19 Drawing Instruments What we have lost in the flood of PCs

David Brewster’s Edinburgh Encyclopedia (1808-1830)

Aesthetics which value haptic interaction with specialized physical objects ... but much richness has been lost.

This slide shows the various architectural drawing instruments used in 19th century. Long before the invention of personal computers, our ancestors developed a variety of specialized physical artifacts to draw geometric shapes. Grasping and manipulating those instruments, users of the past must have developed unique and rich languages, cultures, and aesthetics which value haptic interaction with real physical objects. We began our investigation of "looking to the future of HCI" at a museum by looking for what we have lost in the flood of personal computers. Our intention is to find a way to rejoin the richness of the physical world in HCI.

20 Tangible User Interfaces

• Giving physical forms to digital information & computation, making bits – directly manipulable with our hands (in manipulatory) – perceptible through our peripheral senses (in ambulatory) • Seamless coupling between digital and physical worlds.

Tangible interface is an interface design approach to give physical forms to digital information to make it tangible. It makes bits directly manipulable with our hands, and perceptible through our peripheral senses. The first path to the Tangible Interface is the direct manipulation with hands. Second path is the use of ambient media to make people aware of digital information at the periphery of perception. Pinwheels is such an example. The Tangible Interface explores the way to weave interfaces with the digital world into the fabric of a physical and social context.

21 Center and Periphery Architectural Space as Interface

• to grasp & manipulate bits in the center of user's focus by coupling bits with physical objects and surfaces, and • to be aware of bits at the periphery using ambient display media such as light, sound, airflow, and water movement.

This slide illustrates the framework of “center and periphery” of human perceptions. The ambientROOM was designed to achieve two goals: 1) to grasp & manipulate bits in the center of user's focus by coupling bits with physical objects and surfaces, and 2) to be aware of bits at the periphery using ambient display media such as light, sound, airflow, and water movement. We get information in two main ways. First, we get information from what we are focusing on, where our center of attention is directed. But at the same time, we also get information from ambient sources. We may have a sense of the weather outside from ambient cues such as light, temperature, sound, and air flow from the open windows. We may also have an idea of the activities of our colleagues from the ambient sound and the shadows of passers-by. Our brain is not only capable of processing this ambient information while maintaining a focus elsewhere, but will naturally shift such information to our foreground when our subconscious deems it important. By using ambient media, we can take advantage of our brain's abilities as both a parallel processor and an attention manager. Current HCI research is primary focusing on foreground activity and neglecting the background. We would like to support seamless transitions of the user's focus of attention between the periphery (background) and the center (foreground) of human awareness.

22 Pinwheels: wind of bits Ren, Frei, Dahley, Wisneski, and Ishii, 1997-2000

• Ambient information display spinning in a "wind of bits.”

• Architectural space will be an ambient interface between humans and online digital information.

23 Pinwheels: wind of bits Ren, Frei, Dahley, Wisneski, and Ishii, 1997-2000

Ambient information display spinning in a "wind of bits.” QuickTime™QuickTime™ and and a a decompressordecompressor are needed to see this picture. are needed to see this picture. Architectural space will be an ambient interface.

24 Water Lamp: rain of bits Dahley and Ishii, 1997

QuickTime™ and a decompressor are needed to see this picture.

Water ripple shadow created by a "rain of bits."

25 Foreground --> Background Peripheral Awareness using Ambient Media

PCs

ambient Time-consuming Requires navigation Complex

phone Always on, real-time Peripheral awareness Seamless with environment

26 Orb by Ambient Devices (Media Lab Spinoff) www.ambientdevices.com

• This light glows different colors to help you monitor your portfolio, traffic on your commute, new snow in the mountains, pollen index, etc.

27 Information Gauges 情報計器 courtesy of Ambient Devices

Market 市場 Pollen 花粉 Weather 天気

• Ever-present information gauges log data and show trends in pollen index, weather forecast, snow survey, today’s market, blood pressure, blood sugar, exercise, weight, etc. 常時表示の情報計器がデータを収集記録し表示する。例えば花粉、天気、降雪量、市場、血圧、血糖値、体重など。

27 Ambient Displays Design Principles

• Browser-less interface – Glance-able, requires no navigation and no analysis, simple. • Calm – Non-intrusive, seamless with environment • Persistent connection – Information is continuously updated. • Decision-driven data – Personalized and summarized data feeds to make a decision. • Private – Encoded data

29 Wireless products will diversify ワイアレスプロダクトは進化の初期段階

Electricity • Like wireless, electricity Wireless 電気 started out with one ワイアレス application in mind: the Light Bulb 電球 Mobile Phone 携帯電話 light bulb （電気の最初のアプリケーションは電球） • It took 20 years for the first new products using electricity to be introduced （最初の製品導入まで 20年を要した） • Today electricity is an invisible carrier to virtually every consumer product（今日電気は遍在キャリア） ? © 2004 MIT Media Laboratory, Hiroshi Ishii

29 ambientROOM Architectural Space as Interface

Ripple shadows Ambient sound on ceiling of rain drops

Light projection Clock to on side wall navigate time

Bottles as containers of bits

Open a bottle to release bits into air

Let me summarize what you saw in this video. We demonstrated three types of ambient media. First, ripple shadow display. We made a thin water tank outfitted with a small robot which taps water being "pulled by bits." With each pull, the robot creates a ripple on the surface of water. Light projected onto the water tank from above casts a subtle but poetic image of ripples on the ceiling of the room. Second, subtle light projection on a side wall provide awareness of activities in a remote working place. Third, ambient sound of rain drops or water stream is used to let people be aware of activities in cyberspace such as network traffic.

31 inTouch: Tangible Telepresence Brave, Dahley, Frei, Su, and Ishii, 1998

“Synchronized Distributed Physical Objects” create an illusion of touching the same object using force-feedback technology.

inTouch-0: inTouch-1: inTouch-2: mechanical early electronic distributed mockup prototype prototype

Touch is underdeveloped. Touch is a fundamental aspect especially of interpersonal communication. Physical contact is a basic means through which people achieve a sense of connection and express emotion. Communication through touch, however, has been left largely unexplored in telepresence research. Visual and auditory extension of space was dominant approach of telepresence as represented by ClearBoard. In this summer, we have started the implementation of a distributed prototype via a computer network. The goal of this design is to allow the virtual connection of inTouch-1 to extend over any arbitrary distance.

In experiments using inTouch-1, in conjunction with audio communication, for longer periods of two-person interactions, subjects often noted that the haptic interaction with inTouch-1 was distracting at first, but later fell more into the background of perception. When this happened subjects felt that the interaction became more natural and smooth. Subjects also general felt that the haptic channel presented a “subtle but deep and revealing” addition to audio communication. More specifically users commented that the haptic channel allowed them to get a better sense of the other person’s emotional state and sincerity. Finally, some users expressed feelings of awkwardness when interactions with an acquaintance became more rhythmical, since they associated such movements with more intimate relationships.

32 inTouch: Haptic Interpersonal Communication Medium

illusion of touching the same QuickTime™ and a YUV420QuickTime™ codec decompressor and a object using force-feedback are needed decompressor to see this picture. are needed to see this picture. technology.

Touch is also a fundamental aspect of interpersonal communication. We use touch to achieve a sense of connection, indicate intention, and express emotion. Though often subtle, touch can be a very powerful mechanism through which we express our thoughts and feelings to others. Current telecommunications technology, however, lacks the mechanism for expression through touch. The goal of our work is to enhance real-time remote collaboration and communication by bringing a greater sense of touch and physicality to distributed multi-user interactions.

33 “Ghostly Presence”

traditional remote tangible telepresence collaboration systems

user A’s user B’s shared physical work space physical space physical space

We have primarily described Synchronized Distributed Physiacl Objects as a means for extending the Tangible Interface approach into the realm of distributed CSCW. In our observations of and discussions with users of both PSyBench and inTouch, however, we have found that this approach also suggests a method for providing a new type of awareness of remote users. In traditional telecommunication systems we get the distinct impression that the distant user is in a space that is separate from our own. They are, for example, isolated in the digital world on the other side of our computer screen, where we are unable to touch them or share our physical world with them. In providing distant users with a shared physical space, we begin to see a new form of awareness of the physical presence of the distant user within our own space. As the objects on the surface of PSyBench or the rollers on inTouch move around "on their own", we are compelled to imagine the position and actions of the person causing that movement... much like a player piano compels us to imagine the a real body sitting at the piano bench with arms extending to the keys. And we tend to imagine that presence within our own space; a sort of ghostly presence in our environment which can see and manipulate the same objects that we have access to.

34 ComTouch: A Vibrotactile Communication Device A. Chang, Z. Kanji, A. O'Modhrain, E. Gunther R. Jacob, and H Ishii, 2002

• Goal: Use touch to enhance real-time remote communication • Motivation: Explore how tactile communication language can be developed

The ComTouch project is about designing a touch communication device for mobile use. We aim to enhance interpersonal communication by using touch. Touch conveys a wealth of personal information. Although we know that tactile communication devices do exist, we were interested in further development of touch communication, particularly assessing in what ways touch can enhance audio interaction.

35 Touch to vibration mapping

We chose to implement the touch-to-vibration mapping for the hand as shown here. For each finger, there are three different areas under the finger. Fingertip pressure is converted into vibration frequency and intensity. Because this is such a non-intuitive mapping , we implement a feedback channel so that the sender can feel their squeeze. This vibration is sent to a corresponding partner, who can squeeze back. By pressing harder… stronger and faster vibrations will result.

36 PSyBench Brave, Bulthaup, Ishii, 1998

QuickTime™ and a decompressor are needed to see this picture.

Physically Synchronized Workbench for Remote Collaborative Design (very early crude prototype)

37 Curlybot Frei, Su, ishii, 2000

A toy that can record and playback physical motion.

QuickTime™ and a YUV420 codec decompressor Children establish an are needed to see this picture. affective and body syntonic connection with curlybot, and develop intuitions for concepts such as differential geometry.

© 2004 MIT Media Laboratory, Hiroshi Ishii curlybot is an toy that can record and playback physical motion. As one plays with it, it remembers how it has been moved and can replay that movement with all the intricacies of the original gesture; every pause, acceleration, and even the shaking in the user's hand, is recorded. curlybot then repeats that gesture indefinitely creating beautiful and expressive patterns. === We introduce an educational toy, called curlybot, as the basis for a new class of toys aimed at children in their early stages of development - ages four and up. curlybot is an autonomous two-wheeled vehicle with embedded electronics that can record how it has been moved on any flat surface and then play back that motion accurately and repeatedly. Children can use curlybot to develop intuitions for advanced mathematical and computational concepts, like differential geometry, through play away from a traditional computer.

In our preliminary studies, we found that children learn to use curlybot quickly. They readily establish an affective and body syntonic connection with curlybot, because of its ability to remember all of the intricacies of their original gesture; every pause, acceleration, and even the shaking in their hand is recorded. Programming by example in this context makes the educational ideas implicit in the design of curlybot accessible to young children.

38 Curlybot Frei, Su, ishii, 2000

• Children readily establish an affective and body syntonic connection with curlybot. • They can develop intuitions for concepts such as differential geometry, through play away from a traditional computer.

39 topobo Building Blocks with kinetic memory Hayes Raffle, Amanda Parkes, and Hiroshi Ishii

• made of active( motorized) & passive (static) components • passives geometry based on cubic & tetrahedral crystals • coincident input & output space • actives “programmed” by moving, pushing, twisting units • recorded sequence automatically plays back repeatedly • distributed computation and networking

40 topobo Building Blocks with kinetic memory Hayes Raffle, Amanda Parkes, and Hiroshi Ishii

• made of active (motorized) & passive (static) components • passives geometry based on cubic & tetrahedral crystals • coincident input & output space • actives “programmed” by moving, pushing, twisting units • recorded sequence automatically plays back repeatedly • distributed computation and networking

QuickTimeý Ç² YUV420 ÉRÅ[ÉfÉbÉN êLí£ÉvÉçÉOÉâÉÄ Ç™Ç±ÇÃÉsÉNÉ`ÉÉÇ¾å©ÇÈÇžÇ½Ç…ÇÕïKóvÇ-Ç�ÅB

41 Coincidence of input and output spaces

Principle of Tangible Interface Design

inTouch 98 curlybot 00 topobo 04 interpersonal mathematics and building block with communication expression / narrative kinetic memory

Since 1996, we have been exploring several key design principles of Tangible Interface. One of these principle is the “coincidence of input and output space.” It is critical to blur the boundary of input/output, and representation/control to make information tangible. An initial example was inTouch designed for haptic interpersonal communication. Today, the curlybot will demonstrate a realization of this principle with special focus on mathematical education and narrative expression.

42 Constructive Assembly: Physical/Digital Building Blocks

Blocks, Cubes, Triangles, …

Blocks

[Aish 79] [Anagnostou 89, Frazer 94]

Triangles 99 Physical/ Digital Construction Kit [Anderson 00] MERL

43 Triangles: Physical / Digital Construction Kit Gorbet, Orth and Ishii, 1998

• Physical embodiment of digital information topology

QuickTime™ and a • Tactile feedback of digital decompressor are needed to see this picture. connection from magnetic edge connectors • Physically persistent representation

The Triangles allow users to feel as if they are actually holding and rearranging data itself, with the magnetic connectors providing tactile confirmation when connections are made or broken. Triangles is Not 'digitally enhanced' version of existing object. Triangles is the new language, digital and physical language.

Now let me introduce the Triangles, the final example of Tangible Bits project. The Triangles system is a physical/digital construction kit, which allows users to grasp and manipulate complex digital information with two hands. The system consists of a set of identical flat, plastic triangles, each with a microprocessor inside and magnetic edge connectors, which enable the Triangles to be physically connected to each other. The connectors pass electricity, allowing the Triangles to communicate digital information to each other and to a desktop computer. When the pieces contact one another, specific connection information is sent back to a computer that keeps track of the configuration of the system. The Triangles can be used to make two and three-dimensional objects whose exact configuration is known to the computer.

44 Triangles: Physical / Digital Construction Kit Gorbet, Orth and Ishii, 1998

• Physical embodiment of digital information topology • Tactile feedback of digital connection from magnetic edge connectors • Physically persistent representation

45 Painted Bits (GUI)

General input devices as remote-controllers of intangible representation (pixels on a screen)

output

input pixels sound

remote intangible physical control representation

digital information

46 Tangible Bits (TUI)

Tangible representation as interactive control mechanism to manipulate the information represented in both tangible and intangible forms

physical objects e.g. building model e.g. video projection of digital shadow control tangible intangible physical representation representation

digital information

47 I/O Bulb and Luminous Room Underkoffler and Ishii, 1997 - 1999

• I/O Bulb – High resolution output, two-way information • Luminous Room – Multiple I/O bulbs illuminating architectural space

• Give life to architectural surfaces and physical objects. • Enable direct manipulation of digital world by grasping and manipulating objects with digital shadows.

48 I/O Bulb Video Underkoffler and Ishii, 1997-99

• illuminating light – Holography • distributed illuminating light

• seep – Fluid dynamics

• Urp – Urban simulation

49 Sensetable

Patten, Ishii, Pangaro, 2000

• TUI platform to track multiple objects and their states on a table with video projection

• System Dynamics simulation for Supply Chain Analysis

50 Sensetable: TUI platform James Patten, Dan Chak & Hiroshi Ishii

• TUI platform to track multiple objects and their states on a table with video projection

• Applications – System Dynamics Simulation for Supply Chain Analysis (& Intel and Sloan School) – Music “Audiopad” (James Patten & Ben Recht) – Event Driven Simulation for IP Network Design (& NTT Comware)

51 System Dynamics Simulation for Supply Chain Analysis Patten, Hines, Malone, Murphy-Hoye & Ishii 00-03

QuickTime™ and a decompressor are needed to see this picture.

Collaboration with Intel and MIT Sloan School

52 System Dynamics Simulation for Supply Chain Analysis Patten, Hines, Malone, Murphy-Hoye & Ishii 00-03

QuickTime™ and a Sorenson Video 3 decompressor are needed to see this picture.

Collaboration with Intel and MIT Sloan School

53 IP Network Design Workbench NTT Comware + TMG

• Event-Driven Simulation + NTT Comware’s network design consulting expertise

• TUI supports cooperative direct manipulation of IP Network simulator.

54 IP Network Design Workbench: NTT Comware + TMG (sensetable)

• Based on Event-Driven Simulation Engine and NTT Comware’s NW consulting expertise QuickTime™ and a • This workbench helps YUV420 codec decompressor designers to evaluate the are needed to see this picture. effects of changing topology, bandwidth, server location in real time, to optimize the network performance. • TUI supports cooperative direct manipulation of IP Network simulator. © 2004 MIT Media Laboratory, Hiroshi Ishii

55 IP Network Design Workbench NTT Comware + TMG

• Event-Driven Simulation • TUI supports cooperative direct manipulation of simulator to evaluate the effects of changing topology, QuickTime™ and a YUV420 codec decompressor are needed to see this picture. bandwidth, server location in real time, to optimize the network performance.

Thanks to Mr. Kase, Mr. Hirano, Mr. Narita, Ms. Kobayashi, Mr. Tanaka, and many other NTT Comware people.

56 From Physical World Model to Computational Abstract Model

Principle of Tangible Interface Design

Urp 99 System Dynamics Simulation 03

57 Audiopad James Patten and Ben Recht (Physics & Media)

• A new way to perform electronic music. • Designed to combine the expressive power of traditional musical instruments with the modularity of a computer • Based on the Sensetable proj ect.

58 Audiopad James Patten and Ben Recht (Physics & Media)

QuickTime™ and a H.263 decompressor are needed to see this picture.

• A new way to perform electronic music. • Designed to combine the expressive power of traditional musical instruments with the modularity of a computer • Based on the Sensetable proj ect.

59 Audiopad James Patten, Ben Recht (Physics & Media)

• A new way to perform electronic music. • Designed to combine the expressive power of QuickTime™ and a traditionalH.263 decompressor musical instruments with the modularityare needed to see of this a picture. computer • Based on the Sensetable project.

60 Audiopad James Patten and Ben Recht* (*Physics & Media)

• A new way to perform electronic music. • Designed to combine the expressive power of traditional musical instruments with the modularity of a computer QuickTimeý Ç² • Based on the Sensetable proj ect. YUV420 ÉRÅ[ÉfÉbÉN êLí£ÉvÉçÉOÉâÉÄ Ç™Ç±ÇÃÉsÉNÉ`ÉÉÇ¾å©ÇÈÇžÇ½Ç…ÇÕïKóvÇ-Ç�ÅB

61 CircuiTUI: Real-time circuit simulation and analysis Dan Chak and Hiroshi Ishii

• A middle ground between a laboratory and a text book • Immediate feedback – “Oscilloscope” voltage and current shown by each component • Displays all quantitative and qualitative data • Classroom testing in MIT physics course this fall semester

62 CircuiTUI Visualizations

• “Oscilloscope” voltage and current shown by each component • Wire thickness varies with the amount of current • Positive voltages are blue, negative are orange • Color intensity analogous to voltage intensity • Circuit behavior can be understood immediately

63 Sensetable James Patten & Hiroshi Ishii

• TUI platform to track multiple objects and their states on a table with video projection

• Applications – Music “Audiopad” QuickTime™ and a in collaboration with Sorenson Video decompressor Ben Recht are needed to see this picture. – System Dynamics simulation for Supply Chain Analysis – Chemistry

64 Concrete and Abstract

Principle of Tangible Interface Design

specific / concrete generic / abstract

Phicon of Puck of metaDesk 97 Sensetable 00

65 Actuated Workbench Dan Maynes-Aminzade and Gian Pangaro & Hiroshi Ishii

• Augments Sensetable platform providing an additional physical

dynamic display capability using a QuickTime™ and a Photo - JPEG decompressor grid of electromagnets to move are needed to see this picture. objects on surface of table in two dimensions. • Pucks on table contain LC tag, permanent magnet, and toggle switch. Applications – Synchronization of distributed “Sensetables” in realtime remote collaboration – Clearing up inconsistencies that arise from the computer's inability to move the objects on the table

66 Actuated Workbench Dan Maynes-Aminzade and Gian Pangaro & Hiroshi Ishii 02

QuickTime™ and a QuickTime™ and a H.263 decompressor H.263 decompressor are needed to see this picture. are needed to see this picture.

Function Application

Magnetic forces to move Augment existing “Sensetable” objects on a table in two providing an additional physical dimensions. dynamic display capability.

67 Actuated Workbench Dan Maynes-Aminzade, Gian Pangaro & Hiroshi Ishii 02-03

• Augments Sensetable platform providing an additional physical dynamic display capability using a grid of electromagnets to move objects on surface of table. • Pucks on table contain LC tag, permanent magnet, and toggle switch.

68 Actuated Workbench Dan Maynes-Aminzade and Gian Pangaro & Hiroshi Ishii 02-03

Application 1 • Clearing up inconsistencies that arise from the computer's inability to move the objects on the table

without actuation with actuation

"In the video on the left, we see several people positioning cell phone towers on an overhead map. They can scroll the map by locking down one of the towers and moving it. However, notice how this creates an inconsistency between the physical and the digital states of the system. The user must manually adjust the positions of the other towers on the table to fix it. Users can also rotate and zoom the map, but this creates an inconsistency as well."

"By adding actuation to the system, we can automatically correct these inconsistencies. In the video on the right, we see the same scroll, rotate, and zoom operations, but the positions of the other pucks on the table are automatically updated using magnetic actuation."

69 Actuated Workbench Dan Maynes-Aminzade and Gian Pangaro & Hiroshi Ishii 02-03

Application 2 • Synchronization of distributed “Sensetables” in realtime remote collaboration

Actuation can also be used to synchronize two workbenches in different locations, facilitating remote collaboration. Notice how when a puck is moved on one table, it moves in the same way on the other table. This helps people to work together remotely using the Sensetable.

By placing a camera above the workbench, we can capture silhouettes of the users' hands. We then broadcast these hand silhouettes to the other workbench to show the activity of the remote user.

70 Senseboard Jacob, Ishii, Pangaro, Patten 2001

• TUI platform to allow users to arrange small magnetic pucks bound to digital data and function on a grid of tag readers with video projection

• Application: organizing and grouping information

71 Senseboard Jacob, Ishii, Pangaro, Patten 2001 (video)

QuickTime™ and a decompressor are needed to see this picture.

72 Illuminating Clay Piper, Ratti, and Ishii 2001

• Physical Clay as 3-D Physical Input & Visual Display for intuitive manipulation and understanding of spatial relationships • Landscape Simulation – land erosion, visibility, shadow casting, etc. • 3D Laser Scanner + Video Projector

73 Illuminating Clay Ben Piper, Carlo Ratti & Hiroshi Ishii

• Physical Clay as 3-D Physical Input & Visual Display for intuitive manipulation QuickTime™ and a decompressor and understanding of are needed to see this picture. spatial relationships

• 3D Laser Scanner + Video Projector

74 IR camera video projector System

3D view A ceiling mounted IR camera captures the radiance of the light mode glass beads selection passing through the block sand model to determine the geometry of the surface. stone table The resulting landscape analysis is projected back on to the surface. IR light sources

75 Physical Design Media

Clay Cardboard Wooden Blocks Found Objects Physical Outcomes Stata Center 2002

Frank O. Gehry, Architect

76 Lack of Continuity Between Physical and Digital Representation in Design

Physical Digital

Ease of manipulation Greater precision Clearer communication Easy distribution Aids spatial understanding Quantitative analysis

How can we merge these media?

77 Form Giving + Computational Reflection: Refresh Rate of Iterative Design Cycle

Physical Digital

Upper Stream Lower Stream

Rough and rapid form giving Precise and quantitative with hand for ideation computational reflection

simultaneous form giving + computational reflection

78 Representation of Idea Matters …

… because the mental operations are made possible by the representation. … GUI/CAD is not for ideation.

e.g. • Mathematical representation • Drawings • Physical models • Computational models

79 Media for Design Thinking

• Visual Thinking – sketch

• Tangible Thinking – tactile manipulation of physical representations coupled with digital computation – design + analysis

80 PingPongPlus Ishii, Lee, Wisneski, Orbanes 1999

• Digital augmentation of ping pong play with "reactive table." • Ball tracking using microphone array underneath table. • From competition to collaboration

81 PingPongPlus Ishii, Lee, Wisneski, Orbanes 1999

• Digital augmentation of ping pong play with "reactive table."

QuickTime™ and a YUV420 codec decompressor are needed to see this picture. • Ball tracking using microphone array underneath table.

From competition to collaboration

82 Augmentation of existing objects

Principle of Tangible Interface Design

Another principle is the augmentation of existing objects to develop on top of existing understanding and skills. The bottles demonstrated in SIGGRAPH 99 emerging technologies is such an example. HandSCAPE to be presented by Jay Lee is another example to augment existing tape measure to bridge to the world of digital modeling.

Now I would like to introduce Phil Fre who is presenting and demonstrating curlybot.

83 musicBottles Ishii, Fletcher, Mazalek, Lee, Choo, Berzowska, Paradiso, 98-00

• Glass bottles as "containers" and "controls" for digital information • Seamless extension of metaphors and physical affordances into the digital domain

Jazz Techno Classical Weather

© 2004 MIT Media Laboratory, Hiroshi Ishii musicBottles proposes a new tangible interface using glass bottles as containers for digital information. Imagine an array of perfume bottles. Instead of scent, the bottles have been filled with music classical, jazz, and techno music. Opening each bottle releases the sound of a specific instrument.

84 Origin: Weather Bottle

• Present for my mother • Model: soy sauce bottle in her kitchen

The idea of a bottle interface originated from the concept of a “weather forecast bottle,” which I envisioned as a present for my mother. Upon opening the weather bottle, she would be greeted by the sound of singing birds if the following day’s weather was forecasted to be clear. On the other hand, hearing the sound of rainfall from the bottle would indicate impending rain. Such an interface would be consistent with the everyday interactions with her familiar physical environment, such as opening a bottle of soy sauce. She never clicked a mouse or typed a URL in her life, but opened soy sauce bottles thousands of times.

85 Origin: Weather Bottle

QuickTime™ and a YUV420 codec decompressor are needed to see this picture.

86 Tangible Bits

• Giving physical forms to digital information and computation, making bits – directly manipulable with two hands • Continuity between physical and digital representation in design • Supporting multi-user collaboration and “tangible thinking”

87 Painted Bits (GUI) and Tangible Bits (TUI)

Graphical User Interface • Intangible representation (pixels on a screen) + • Generic input devices as “remote controllers”

Tangible User Interface • Tangible representation as interactive control mechanism to manipulate the information and computation • Continuity between physical and digital representation in design

88 Tangible Bits

• Reconciliation of our dual citizenship in the worlds of bits and atoms. • Interaction Design – informed by sciences (HCI), – materialized by technologies (CS, EE, ME), and – shaped by industrial design, media arts and practical real-world applications.

89 Tangible Bits Design Space

Arts & Sciences & Design Technologies

concept, philosophy, scientific theory, implementation, aesthetics evaluation, analysis

Tangible Interface Design

Interaction Design Computer Sci. Cog. Sci. Industrial / Product Design Mechanical Eng. Soc. Sci. Architectural / Env. Design Electronic Eng. Media Arts / Interactive Arts

90 Design?

Conceptual “eyeglasses” – Inspire people’s imagination and creativity

91 “The Computer for the 21st Century”

“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”

Mark Weiser July 23, 1952 - April 27, 1999

92 Exhibitions

Ars Electronica Center Linz, Austria “Get in Touch” musicBottles SandScape Audiopad

Centre Pompidou Paris, France, 11/19/03-1/6/04 Le Design Interactif PingPongPlus,

93 Acknowledgments

• Tangible Media Group Graduate students and UROPs • Things That Think, Digital Life @ MIT Media Lab • Friends in ACM SIGCHI, SIGGRAPH, IDSA, ICSID • Ars Electronica Center, NTT-InterCommunication Center

We thank Bill Buxton and George Fitzmaurice at the University of Toronto for countless discussions about graspable UI, skill-based design, and foreground & background issues, through which many of the ideas in this paper were developed and shaped. Thanks are also due to Bill Verplank at Interval Research for his insightful comments and discussions on haptic interfaces and tangible media. We appreciate Mark Weiser for his inspiring Ubiquitous Computing work and the introduction of Live Wire to me. We thank TTT (Things That Think), a new consortium at the MIT Media Lab, for its ongoing support of the Tangible Bits project. We also would like to acknowledge the contribution of many hardworking graduate and undergraduate students at MIT for work on the implementation of the Tangible Bits platforms. In particular, we thank graduate students Scott Brave, Andrew Dahley, Matt Gorbet, and many other students for their work on the metaDESK and ambientROOM prototypes. Thanks are finally due to Steelcase, Inc. for their donation of Personal Harbor which became a plat form of the ambientROOM.

94 Thanks!

Hiroshi Ishii Tangible Media Group MIT Media Laboratory http://tangible.media.mit.edu/

95 Thanks!

Hiroshi Ishii Tangible Media Group MIT Media Laboratory http://tangible.media.mit.edu/

Thanks for your kind attention!

96 At the Boundary

Where the sea meets the land, life has blossomed into a myriad of unique forms in the turbulence of water, sand, and wind.

Hiroshi Ishii | Tangible Media Group | MIT Media Lab

97 Ubiquitous Computing Mark Weiser, Xerox PARC, 1991

Mark Weiser at Xerox PARC presented his vision of “Ubiquitous Computing” in 1991. He claimed computers should be transparent and invisible.

Recently, he is introducing a concept of “calm technology” citing the LIVE WIRE by Natalie Jeremijenko as an example.

98 Research?

• Increment – Local optimization – Solve known problems – Reflection-driven • Secure home – Orthodox – Established discipline – Homogeneity

99 Research?

Yes No • Innovation • Increment – Mutation – Local optimization – Identify new – Solve known problems problems – Invention-driven – Reflection-driven • No home • Secure home – Heterodox - heretic – Orthodox – Inter-disciplinary – Established discipline – Diversity – Homogeneity

100 Design

• Highly interdisciplinary – Enjoy identity crisis. – Appreciate diversity.

101 Design

• Highly interdisciplinary – Enjoy identity crisis. – Appreciate diversity. • New vision, no roadmap – Invent new paradigms, rather than do incremental work, because life is short.

102 Design

• Highly interdisciplinary – Enjoy identity crisis. – Appreciate diversity. • New vision, no roadmap – Invent new paradigms, rather than doing incremental work. • Success is a danger – Do not stop. Keep redefining yourself.

103 To the Researchers

• New vision, no roadmap – Invent new paradigms, rather than doing incremental work, because life is short. • Highly interdisciplinary – Enjoy identity crisis. – Appreciate diversity. • Ground to the earth – Learn through real world problem solving • Success is a danger – Do not stop. Keep redefining yourself.

104 Beyond Painted Bits, Toward Tangible Bits GUI TUI

• Beyond GUI (Graphical User Interface) Toward TUI (Tangible User Interface) • Physical objects, surfaces, and spaces as representation and interfaces for computationally mediated information.

Taking advantage of the multi-modality of human interactions with the physical world, we are trying to go beyond “painted bits”, and establish a new paradigm of “Tangible Bits.” Coupling bits with physical objects and phenomena is the key approach, we are trying to make the interfaces ubiquitous and transparent. Ultimately, we believe the architectural space itself should be an interface between people and digital information space

105 Thanks!

Hiroshi Ishii Tangible Media Group MIT Media Laboratory http://tangible.media.mit.edu/

106 Thanks!

Hiroshi Ishii Tangible Media Group MIT Media Laboratory http://tangible.media.mit.edu/

107 Principles of Tangible Interface Design (1)

Coincidence of input and output spaces

inTouch 98 curlybot 00 interpersonal computation / mathematics communication and expression / narrative

108 Principles of Tangible Interface Design (2)

Augmentation of existing objects

Bottles 99 containers

Now I would like to introduce Phil Fre who is presenting and demonstrating curlybot.

109 Oullim

• Aufheben (German) • People, Bits, and Atoms – Body, Cyberspace, and Physical Space • Arts, Design, Sciences, and Technologies – Analysis and Synthesis

110 Real + Virtual, physical | digital but mainly Visual

• Personal Computing

• Virtual Reality – completely immerses a user inside a synthetic environment. • Augmented Reality – allows the user to see the real world, with virtual objects superimposed upon the real world. supplements reality, rather than completely replacing it.

111 Augmented Reality Def. by Ron Azuma Augmented Reality allows the user to see the real worl d, with virtual objects superimposed upon or composited with the real world. supplements reality, rather than completely replacing it. 1. Blends real and virtual, in real environment 2. Real-time interactive 3. Registered in 3-D

• Azuma, A Survey of Augmented Reality. Presence, 6 (4), p.355-385, Aug. 1997 • SIGGRAPH 2001 Course Notes 27, Augmented Reality: The Interface is Everywhere

112 Augmented Reality Computer-Augmented Environments

• How to integrate “real world” and computational media? • Integrating “real world” and computational media by visual overlay of digital information onto real world image with see-through display, hand-held display, or direct video projection • Redefining “Human Interactions” with computational (virtual) artifacts within the social and physical settings of their use – From Desktop Computing to Ubiquitous Computing – From Desktop UI to Ubiquitous UI – From Virtual Reality to Augmented & Mixed Reality

Augmented Reality is a new research stream which tries to integrate the real world and computational media.

There are many important work including Myron Krueger’s pioneering work of VIDEOPLACE in 1985. DigitalDesk by Wellner is one of early work which demonstrated a way to merge physical and digital document by using video projection of computer display onto a real desk with physical documents.

The most common AR approach is the visual overlay of digital information onto real-world imagery with see through head-mounted display, hand-held display, or direct video projection.

113 Eyes are in charge, but hands are underemployed.

114 Bottles as Interface

The core concept utilizes glass bottles as "containers" and "controls" for digital information. Physical manipulation of the bottles - opening and closing is the primary mode of interaction with digital information.

© 2004 MIT Media Laboratory, Hiroshi Ishii musicBottles proposes a new tangible interface using glass bottles as containers for digital information. Imagine an array of perfume bottles. Instead of scent, the bottles have been filled with music - classical, jazz, and techno music. Opening each bottle releases the sound of a specific instrument.

115 Sensetable & Senseboard as Generic TUI Platforms

TUI Applications

Supply Chain Vis. Urban Planning others

TUI Application Program Interface

Sensetable / Senseboard Hardware

116 Tangible User Interfaces

• Giving physical forms to digital information, making bits – directly manipulable with our hands – perceptible through our peripheral senses

• Seamless coupling between digital and physical worlds.

Tangible interface is our challenge to design seamless interface between the land of atoms and the sea of bits for the people. 117 People have developed sophisticated skills for sensing and manipulating their physical environments. However, most of these skills are not employed by the traditional Graphical User Interface. GUIs (Graphical User Interfaces) are designed for human eyes, not for hands. GUIs are in the boxes detached from our physical environment and social context. Tangible interface is an interface design approach to give physical forms to digital information to make it tangible. It makes bits directly manipulable with our hands, and perceptible through our peripheral senses. The Tangible Interface explores the way to weave interfaces with the digital world into the fabric of a physical and social context. The Tangible Interface seeks to build upon the skills of manipulating the physical world, by giving physical form to digital information and situating it in the architectural spaces. The Tangible Interface explores the way to weave interfaces with the digital world into the fabric of a physical and social context.

TouchCounters: Interactive Electronic Labels for Physical Containers (Yarin and Ishii 99)

“Distributed visualization of usage history”: Physical objects and surfaces that display their history of use

118 mediaBlocks: Physical Containers, Transports, and Controls for Online Media Ullmer, Ishii, and Glas, 1998

• Small, electronically tagged wooden blocks • Physical icons (phicons) for the containment, transport, and manipulation of online media. • Rapid “copy” and “paste” from a media source into a media display. • Racks allow navigation and sequencing of collections of media elements.

119 Design Evolution Bill Verplank 1998

Paradigm GUI, Ubiquitous Comp. Principles WYSIWIS (GUI) / Idea WYSIWIS (CSCW) Hunch Hack Prototype Product NLS, Alto, Collab market Lisa, Mac, Navigator PC, Web

120 Tangible Bits

Designing the Seamless Interface between People, Bits, and Atoms

Conference Month Day, 2002

121 Augmented Reality Computer-Augmented Environments • Integrating “real world” and computational media by visual overlay of digital information onto real world image with see- through display, hand-held display, or direct video projection • Examples: – DigitalDesk (Pierre Wellner) – KARMA (Steven Feiner) – Chameleon (George Fitzmaurice) – Wearable Computing (Thad Starner)

Augmented Reality is a new research stream which tries to integrate the real world and computational media.

The most common AR approach is the visual overlay of digital information onto real-world imagery with see through head-mounted display, hand-held display, or direct video projection.

122 Graspable User Interface G. Fitzmaurice, H. Ishii, & W. Buxton, 1995

• Direct control of virtual objects through physical handles called “bricks” • Blend physical and virtual affordances • Distribute cognitive load more evenly from visual to tactile system

This work of Graspable User Interface was done in collaboration with G. Fitzmaurice and Bill Buxton during my one year stay at the University of Toronto in 1994. The Graspable User Interface provide direct control of virtual objects through physical handles called “Bricks.” With Bricks, users can take advantage of our everyday skills in manipulating physical objects.

123