Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials Cover Story by Hiroshi Ishii, Dávid Lakatos, Leonardo Bonanni, and Jean-Baptiste Labrune

Volume XIX.1 | january + february 2012

Association for Computing Machinery cover story

Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials

Hiroshi Ishii MIT Media Lab | [email protected] Dávid Lakatos MIT Media Lab | [email protected] Leonardo Bonanni MIT Media Lab | [email protected] Jean-Baptiste Labrune MIT Media Lab | [email protected]

Graphical user interfaces (GUIs) appearance dynamically, so they let users see digital informa- are as reconfigurable as pixels on tion only through a screen, as if a screen. Radical Atoms is a vision looking into a pool of water, as for the future of human-material depicted in Figure 1 on page 40. interactions, in which all digital We interact with the forms below information has physical mani- through remote controls, such as festation so that we can interact a mouse, a keyboard, or a touch- directly with it—as if the iceberg screen (Figure 1a). Now imagine had risen from the depths to reveal an iceberg, a mass of ice that pen- its sunken mass (Figure 1c). etrates the surface of the water 2 012

and provides a handle for the mass From GUI to TUI beneath. This metaphor describes Humans have evolved a heightened tangible user interfaces: They act ability to sense and manipulate Februar y as physical manifestations of com- the physical world, yet the digital + putation, allowing us to interact world takes little advantage of our directly with the portion that is capacity for hand-eye coordina- made tangible—the “tip of the ice- tion. A tangible user interface berg” (Figure 1b). (TUI) builds upon our dexterity Radical Atoms takes a leap beyond by embodying digital informa- tangible interfaces by assuming a tion in physical space. Tangible hypothetical generation of mate- design expands the affordances

interactions January rials that can change form and of physical objects so they can

38 cover story CoVer STorY

support direct engagement eral attention to make information ment to be directly coupled to digi- with the digital world [1,2]. directly manipulable and intui- tal information, it has limited abil- Graphical user interfaces repre- tively perceived through our fore- ity to represent change in many sent information (bits) through pix- ground and peripheral senses. material or physical properties. els on bit-mapped displays. These Tangible interfaces are at once Unlike with pixels on screens, it is graphical representations can an alternative and a complement difficult to change the form, posi- be manipulated through generic to graphical interfaces, represent- tion, or properties (e.g., color, size, remote controllers, such as mice, ing a new path to Mark Weiser’s stiffness) of physical objects in real touchscreens, and keyboards. By vision of ubiquitous computing time. This constraint can make the decoupling representation (pix- [3]. Weiser wrote of weaving digi- physical state of TUIs inconsistent els) from control (input devices), tal technology into the fabric of with underlying digital models. GUIs provide the malleability to physical environments and making Interactive surfaces is a prom- graphically mediate diverse digital computation invisible. Instead of ising approach to supporting information and operations. These melting pixels into the large and collaborative design and simula- graphical representations and small screens of devices around tion to support a variety of spa- “see, point, and click” interaction us, tangible design seeks an amal- tial applications (e.g., Urp [4], represented significant usability gam of thoughtfully designed profiled on page 41). This genre improvements over command interfaces embodied in different of TUI is also called “tabletop user interfaces (their predeces- materials and forms in the physi- TUI” or “tangible workbench.” sor), which required the user to cal world—soft and hard, robust On an augmented workbench, “remember and type” characters. and fragile, wearable and archi- discrete tangible objects are However powerful, GUIs are tectural, transient and enduring. manipulated, and their movements inconsistent with our interactions are sensed by the workbench. with the rest of the physical world. Limitations of Tangibles Visual feedback is provided onto Tangible interfaces take advantage Although the tangible representa- the surface of the workbench via of our haptic sense and our periph- tion allows the physical embodi- video projection, maintaining

PAINTED TANGIBLE GUI BITS TUI BITS RADICAL ATOMS

THE PHYSICAL WORLD

THE DIGITAL WORLD

a) A graphical user interface only lets b) A tangible user interface is like an iceberg: c) Radical Atoms is our vision for the future of users see digital information through a There is a portion of the digital that interaction with hypothetical dynamic screen, as if looking through the emerges beyond the surface of the water – materials, in which all digital information has surface of the water. We interact with into the physical realm – that acts as physical manifestation so that we can the forms below through remote physical manifestations of computation, interact directly with it – as if the iceberg controls such as a mouse, a keyboard, allowing us to directly interact with the "tip had risen from the depths to reveal its or a touchscreen. of the iceberg.” sunken mass.

interactions January + February 2012 • Figure 1. Iceberg metaphor—from (a) GUI (painted bits) to (b) TUI (tangible bits) to (c) radical atoms.

40 cover story

input/output space coincidence. Urp: An Example of an Early TUI Tabletop TUIs utilize dynamic Urp uses physical scale models of architectural buildings to configure and representations such as video control an underlying urban simulation of shadow, light reflection, wind flow, projections, which accompany and other properties. Urp also provides a variety of interactive tools for query- tangibles in their same physical ing and controlling parameters of the urban simulation. These include a clock space to give dynamic expression to change the position of the sun, a material wand to change building surfaces between brick and glass (thus reflecting light), a compass to change wind of underlying digital informa- direction, and an anemometer to measure wind speed. tion and computation. We call Urp’s building models cast digital shadows onto the workbench surface (via it “digital shadow.” The graphi- video projection). The sun’s position in the sky can be controlled by turning cally mediated digital shadows the hands of a clock on the tabletop. The building models can be repositioned and reflections that accom- and reoriented, with their solar shadows transforming according to their pany physical building models spatial and temporal configuration. Changing the compass direction alters in Urp are such an example. the direction of a computational wind in the urban space. Urban planners can The success of a TUI often relies identify potential problems, such as areas of high pressure that may result in challenging walking environments or hard-to-open doors. Placing the an- on a balance and strong perceptual emometer on the tabletop shows the wind speed at that point in the model. coupling between tangible and intangible (dynamic) representations. In many successful TUIs, it is critical that both tangible and intangible representations are perceptually coupled to achieve a seamless interface that actively mediates interaction with the underlying digital information and appropriately blurs the bound- ary between physical and digital. Coincidence of input and output spaces and real-time response are important elements toward accom- plishing this goal.

Actuated and Kinetic Tangibles: From TUI Toward Radical Atoms Tangible interfaces are deeply concerned with the topic of representation that crosses physical and digital domains. If we rely too much on the digital shadow (intangible representation, such as video 2 012 projection onto the tabletops), it would look like an extended tabletop GUI with multiple point- Februar y ing devices or controllers on the + surface, and then it would lose the advantage of being tangible. The central characteristic of tangible interfaces is the coupling of tangibles (as representation and control) to underlying digital infor- • Urp: A workbench for urban planning and design. Physical building models casting mation and computational models. digital shadows responding to a clock tool that controls the time of the day (bottom

One of the biggest challenges is to left). Wind flow simulations are controlled with a wind tool (bottom right). interactions January

41 cover story

Topobo keep the physical and digital states Topobo is a 3-D constructive assembly system with kinetic memory—the abil- in sync when the information ity to record and play back physical motion. Unique among modeling systems changes dynamically, either by is Topobo’s coincident physical input and output behaviors. By snapping users’ input (direct manipulation of together a combination of passive (static) and active (motorized) components, tangibles) or results of underlying people can quickly assemble dynamic biomorphic forms like animals and computation. Instead of relying on skeletons; animate those forms by pushing, pulling, and twisting them; and ob- the digital shadows (video projec- serve the system repeatedly play back those motions. For example, a dog can tions on the flat surfaces beneath be constructed and then taught to gesture and walk by twisting its body and legs. The dog will then repeat those movements and walk repeatedly. In the or behind tangibles), we have been same way that people can learn about static structures by playing with build- vigorously pushing the envelope ing blocks, they can learn about dynamic structures by playing with Topobo. of the atoms over the last decade beyond their current static and inert states to active and kinetic dimensions, using motors and gears, tiny robots, and shape-memory alloys (SMAs) for prototyping. The evolution from static/inert to active/kinetic tangibles is illus- trated in Figure 2 [5]. Here are three highlights of the threads of evolution from static/passive to kinetic/active tangibles. Embodied kinetic tangibles and kinetic tangible toolkits. Some TUIs employ actuation of tangibles as the central means of computational feedback. Examples include inTouch [6], curlybot [7], and Topobo [8]. This type of actuated TUI does not depend on intangible representation (i.e., video projection/digital shadowing), as active feedback throughout the tangible representation serves as the main display channel. inTouch is one of the early examples of “tangible telepres- ence;” tangibles that involve the mapping of haptic input to haptic 2 012

representations over a distance. One of their underlying mecha- nisms is the synchronization of Februar y distributed objects and the ges- + tural simulation of “presence” artifacts, such as movement or vibration. These gestures can allow remote participants to convey haptic manipulations of distributed physical objects. One outcome is giving remote users

interactions January • Topobo: Kinetic 3-D constructive assembly system. the sense of ghostly presence,

42 STATIC / PASSIVE KINETIC / ACTIVE

Anti-gravity tangibles

levitation of objects

2.5-D deformable / transformable continuous tangibles (digital clay)

transformation of tangible materials

ZeroN levitated tangibles UIST ‘11 Illuminating Clay Sandscape Relief Recompose landscape design tool landscape design tool 2.5-D interactive gesture-controllable using augmented clay using augmented sand shape display 2.5-D shape display CHI ‘02 Ars Electronica ‘02 TEI ‘09, UIST ‘11 UIST ‘11

2-D tabletop discrete tangibles

translation of discrete objects

metaDESK Urp Sensetable PSyBench Actuated PICO concept demo: urban tabletop TUI platform: synchronized Workbench mechanical interven- phicon = container planning objects tracking + actuated computer-actuated pucks tion of actuated pucks + controller workbench video projection workbenches as display & control by users CHI ‘97, UIST ‘97 CHI ‘99, SIGGRAPH ‘98 CHI ‘01 CSCW ‘98 UIST ‘02 CHI ‘07

Kinetic tangible toolkits

Kinetic Sketchup Bosu motion prototyping toolkits with physical transformability TEI ‘09 & DIS ‘10

Embodied kinetic tangibles

tele-sync record and play of motions

Intouch Curlybot Topobo distributed synchronized record & constructive assembly objects (haptic phone) play toy + record & play CSCW ‘98 CHI ‘00 CHI ‘04, ‘06, ‘08

• Figure 2. Evolution of tangibles from static/passive to kinetic/active. cover story

PICO as if an invisible person were PICO is a tabletop interaction surface that can track and move small objects on manipulating a shared object. top of it. It has been used for complex spatial layout problems, such as cellular The use of kinesthetic gestures telephone tower layout. PICO combines the usability advantages of mechani- and movement is a promising appli- cal systems with the abstract computational power of modern computers. It is cation genre. For example, educa- merging software-based computation with dynamic physical processes that are tional toys promoting construction- exposed to and modified by the user in order to accomplish his or her task. ist learning concepts have been Objects on this surface are moved under software control using electromag- explored using actuation technol- nets, but also by users standing around the table. With this method, PICO users can physically intervene in the computational optimization process of determin- ogy, taking advantage of tangible ing cellphone-tower placement. interfaces’ input/output coincidence. Gestures in physical space can illuminate symmetric math- ematical relationships in nature, and kinetic motions can be used to teach children concepts ranging from programming and differential geometry to storytelling. Curlybot [7] and Topobo [8] are examples of toys that distill ideas relating to gesture, form, dynamic movement, physics, and storytelling. Kinetic Sketchup is a toolkit to provide a language for motion prototyping featuring a series of actuated physical programmable modules that investigate the rich • PICO: Mechanical intervention of computationally actuated packs. interplay of mechanical, behavior- al, and material design parameters enabled by motion [9]. Bosu [10] is Relief/Recompose a design tool offering kinetic mem- The Recompose project explores how we can interact with a 2.5-D actuated ory—the ability to record and play surface through our gestures. The table consists of an array of 120 individually back motion in 3-D space—for soft addressable pins, whose height can be actuated and read back simultaneously, materials. Both Kinetic Sketchup thus allowing the user to utilize them as both input and output. Users can interact with the table using their gestures or through direct manipulation. Together, and Bosu were used by designers these two input types provide a full range of fidelity, from low to high precision for motion prototyping to explore a and from hand- (direct manipulation) to body-scale (gestures) interaction. variety of kinetic tangibles. 2-D tabletop discrete tangibles. A variety of tabletop TUIs (interactive surfaces), such as the 2 012

Sensetable system, have been prototyped and partially commercial- ized in the past decade. However, Februar y one limitation of these systems is + the computer’s inability to move objects on the interactive surfaces. To address this problem, the Actuated Workbench [11] and PICO [12] were designed to provide a hardware and software infrastruc- ture for a computer to smoothly

interactions January • Recompose: Direct manipulation and gestural interaction with 2.5-D shape display. move objects on a table surface in

44 cover story

two dimensions, providing an addi- tional feedback loop for computer output, and helping to resolve inform inconsistencies that otherwise see update material arise from the computer’s inability to sync with the

transform digital model to move objects on the table. material 2.5-D deformable/transformable transforms touch or by itself continuous tangibles (digital clay). A gesture fundamental limitation of previous TUIs, such as Urp, was the lack of capability to change the forms of update underlying tangible representations during the model transform interactions. Users had to use pre- digital model of defined, finite sets of fixed-form user gives form the material objects, changing only the spatial user or instructions constraints relationships among them, not conform the form of the individual objects constraints factory themselves. Instead of using pre- constraints defined, discrete objects with fixed • Figure 3. Interactions with Radical Atoms. forms, a new type of TUI system that utilizes continuous tangible materials, such as clay and sand, project [15] explores how removing to control the shape and thus the was developed for rapid form cre- the gravity from the physical world underlying computational model, ation and sculpting for landscape will alter our interaction with it we envision that Radical Atoms design. Examples are Illuminating (see sidebar on page 46). Although should fulfill the following three Clay [13] and SandScape. Later, we focus on the transformable requirements: this type of interface was applied capability of materials in this • Transform its shape to reflect to the browsing of 3-D volumetric article, we envision that the future underlying computational state data in the Phoxel-Space project. of Radical Atoms would incorpo- and user input; Relief was created to explore rate advanced capabilities, such as • Conform to constraints imposed direct and tangible interactions antigravity levitation or syncing by the environment and user with an actuated 2.5-D shape dis- forms among distributed copies input; and play and to provide a kinetic memo- (like inTouch [6]). • Inform users of its transforma- ry of the forms and dynamic trans- tional capabilities (dynamic affor- formation capabilities. Recompose Concept of Radical Atoms dances). added midair gestures to direct Radical Atoms is our vision for Figure 3 illustrates the dynamic touch to enable users to create human interactions with dynamic interactions between the user, forms (sculpt digital clay), and physical materials that are com- material, and underlying digital allowed us to explore a new form of putationally transformable and (computational) model. 2 012

interaction with 3-D tangible infor- reconfigurable. Radical Atoms is Transform. Radical Atoms mation using our bodies [14]. based on a hypothetical, extremely couples the shape of a material Antigravity tangibles. Actuated malleable, and dynamic physi- with an underlying computational Februar y and transformable tangible inter- cal material that is bidirectionally model. The interface must be able + faces have demonstrated a tangible coupled with an underlying digi- to transform its shape in order world that is more dynamic and tal model (bits) so that dynamic to modify the model through the computer controllable, overcoming changes of the physical form can shape of the interface and reflect the limitations of atoms’ rigidity. be reflected in the digital states in and display changes in the com- The most fundamental constraint real time, and vice-versa. putational model by changing its of tangibles comes from the grav- To utilize dynamic affordance shape in synchronicity. ity that governs our interaction as a medium for representation Users can reconfigure and transform

with the physical world. The ZeroN while allowing bidirectional input the interface manually with direct interactions January

45 cover story

ZeroN manipulation, gestural commands, ZeroN is an anti-gravity interaction element that can be levitated and moved or conventional GUIs. The Radical freely by a computer in 3-D space, seemingly unconstrained by gravity. A ZeroN Atoms material should be manu- in movement can represent a sun that casts the digital shadow of physical ally deformable and reconfigurable objects or a planet orbiting based on a computer simulation. The user can place by human hands. We envision or move the ZeroN in the mid-air 3-D space just as they can place and interact “digital clay” as a physically rep- with objects on surfaces. Removing gravity from tangible interaction, the ZeroN resented, malleable material that project explores how altering the fundamental rule of the physical world will is synced with the coupled digital transform interaction between humans and materials in the future. model. Direct manipulation by users’ hands, such as deformation and transformation, of sensor- rich objects and materials should be translated into an underlying digital model immediately to update the internal digital states. Likewise, we envision that “digital and physical building blocks” will allow users to translate, rotate, and reconfigure the structure quickly, taking advantage of the dexterity of human hands, with the changes in configurations reflected in the underlying digital model. This capability will add a new modality of input from physical (users) to the digital world. Material can transform by itself to reflect and display changes in the underlying digital model serving as dynamic physical representations (shape display) of digital information. This is the extension of the concept of current graphical representations on the 2-D screen with pixels to 3-D physical representations with new dynamic matter that can transform its shape and states based on the commands from the underlying digital model. This 2 012

capability will add a new modality of output from the digital to the physical world perceived by users. Februar y Conform: Material transformation + has to conform to the programmed constraints. Since these interfaces can radically change their shapes in human vicinity, they need to conform to a set of constraints. These constraints are imposed by

• ZeroN: Antigravity interaction element enabled by computer-controlled physical laws (e.g., total volume

interactions January magnetic levitation. has to be constant, without phase

46 cover story

changes) and by human common operand, but it constrains the user biguate and identify the best solu- sense (e.g., user safety has to be to reshape the material only at the tion. Though the implementation ensured at all times). scale of his or her hand. Later in of these interfaces is beyond our Inform: Material has to inform history, we started using tools to current-day technological capabili- users of its transformational capa- shape other materials (e.g., wood ties, this vision serves as a guiding bilities (affordance). In 1977 Gibson and metal) to reach a desired principle for our future research [16] proposed that we perceive form. Still, the scope of manipula- explorations. the objects in our environment tion remained on the hand/body Shape-memory clay: Perfect Red. through what action objects have scale. Through the Recompose/ In 2008 the Tangible Media Group to offer, a property he coined as Relief project [14], we explored began to explore Radical Atoms affordance. For example, through how we can combine the low- through a storyboard exercise that evolution we instinctively know precision but extensive gestural asked us to imagine: What interac- that a cup affords storing volumes interaction with high-precision tions are possible with a new kind of liquid. Industrial design in direct manipulation at a fixed of matter capable of changing form the past decade has established scale. Gestures coupled with direct dynamically? a wide set of design principles touch create an interaction appro- Perfect Red represents one to inform the user of an object’s priate for Radical Atoms, since such possible substance: a clay- affordance—for example, a ham- users are able to rapidly reform like material preprogrammed mer’s handle tells the user where dynamic materials at all scales. to have many of the features of to grip the tool. In the case of Context-aware transformation. computer-aided design (CAD) soft- dynamic materials, these affor- Hand tools operate on objects. For ware. Perfect Red is a fictional dances change as the interface’s example, a hammer operates on a material that can be sculpted shape alters. In order to interact nail. We can also imagine writing like clay—with hands and hand with the interface, the user has to tools that can change form and tools—and responds according to be continuously informed about function between a pen, a brush, rules inspired by CAD operations, the state the interface is in, and and a stylus, based on the type including snapping to primary thus the function it can perform. of surface being drawn or written geometries, Boolean operations, An open interaction design ques- upon: a sheet of paper, a canvas, and parametric design. When tion remains: How do we design for or a touchscreen. The way a user Perfect Red is rolled into a ball, it dynamic affordances? holds the tool can also inform how snaps into the shape of a perfect the tool transforms: The way we sphere (primary solids). When two Interactions with Radical Atoms hold a knife and a fork is distinct. pieces are joined, Perfect Red adds Through Radical Atoms we focus If the soup bowl is the operand, the shapes to each other (Boolean on the interaction with a hypothet- then it is likely the operator should addition). Perfect Red also has ical dynamic material rather than be a spoon, not a fork. other behaviors inspired by para- on the technological difficulties Having the ability to sense metric design tools: If you split a in developing such a material. We the grasping hands, surrounding piece in two even halves, then the have explored a variety of applica- operands, and the environment, operations performed on one part tion scenarios in which we interact tools might be able to transform are mirrored in the other. And 2 012 with Radical Atoms as digital clay to the most appropriate form much like CAD software, Perfect for form giving and as a hand tool using contextual knowledge. Red can perform detailed opera- with the ability for context-aware, An umbrella may change its tions using splines projected on Februar y semi-automatic transformation. form and softness based on the the surface of solids. To cut an + Direct touch and gestural interac- direction and strength of wind object in half, for example, all tion. The oldest interaction tech- and rain. A screwdriver could that’s needed is to draw a line nique for form giving is direct transform between a Phillips along the cut and tap it with a touch—humans have been forming and flat head depending on knife. Splines and parametric clay with their hands for thou- the screw it is operating on. behaviors can also be carried out: sands of years. Direct touch offers These context-aware transfor- If you want to drill 10 holes, you high-precision manipulation with mations require a lot of domain simply draw 10 dots and stick a pin

direct haptic feedback from the knowledge and inference to disam- into one of them (Figure 4). interactions January

47 cover story

Tear off a piece of perfect red. Roll a ball of clay to let it snap to a perfect sphere. Smash a ball into a perfect cylinder.

Draw eight dots and stick a pin into Draw a line along the cut Split a piece in two even halves, one of them to drill eight holes. and tap it with a knife. then the operations performed on one part are mirrored in the other.

Carve away shaded area with chisel Insert materials and put two Final rattle created with Radical Atoms. and mirror it in another piece. mirrored pieces together.

• Figure 4. Making an enclosure with Perfect Red (storyboard by Leonardo Bonanni).

The idea of snapping to pri- tion inherent to physical prototyp- micro-robotic technology are open- mary geometries such as sphere, ing with the parametric operation ing new possibilities to material- cylinder, and cube was inspired of computer-aided design tools. Like ize the vision of Radical Atoms. by shape-memory alloys. We software, Perfect Red is only one Nanotechnology. Our vision of extended this notion of “shape of a number of new materials that Radical Atoms requires actuation memory” to the 3-D primary afford unique interactions—what nano-scale (NEMS) and individual forms that can be preprogrammed other materials can you imagine? addressing of elements in the 2 012

into the material. Users can system (quantum computing). conform the constraints of pri- Technology for Atom Hackers Material property changes at the mary volumes by approximating Although the actuated TUIs were molecular level can manifest Februar y them and letting them snap into prototyped using available technol- themselves as drastic property + the closest primary shapes. ogies such as servos, motors, tiny changes at the macroscopic level Perfect Red is imagined as one of robots, electromagnets, and shape- (optical, mechanical, electric, etc.). a number of new materials imbued memory alloys, we envision that Nanosciences aim to scale down with a complex set of responsive Radical Atoms will provide sens- technology to the atomic level. In behaviors. The idea was intended ing, actuation, and communication his legendary 1959 talk, Richard to demonstrate the richness of capabilities at the molecular level. Feynman stated the potential of Radical Atoms in how they can Advances in material science, nan- nanotechnology: “There is plenty of

interactions January combine intuition and improvisa- otechnology, and self-organizing room at the bottom” [17]. Scientists

48 cover story

have made tremendous leaps diffusion, information propaga- exploring novel intelligent materi- toward atom-scale mechanics. tion, Fickian communication). als that could potentially assist Commonplace technology, includ- Since these pioneering theories, the actuation of shape. These ing inkjet printers, already make a number of projects have realized materials could augment com- use of microscopic-scale microelec- basic principles of programmable putational actuation through tromechanical systems (MEMS). matter. Modular self-assembly was material logic [27] by interpolat- Scaling down from micrometers explored by the Miche project [22], ing between actuated points. to nanometers brings its own set which explored how a distributed, Today TUIs are designed in a of difficulties. Devices created at modular system can create struc- heterogeneous manner: Actuation the nanometer scale experience tures of robotic cubes. (structure) and cover (skin) are extreme aging, for example. Light The Millibiology project from designed and implemented sepa- and durable materials like carbon Neil Gershenfeld [23] explores rately. For Radical Atoms we need nanotubes [18] are a promising how synthetic protein chains can new material design principles, alternative to the materials we use reconfigure themselves over six which treat objects as homoge- today. Once technology overcomes orders of magnitude (from Å to neous entities with the ability the difficulties at this level, a wide dm). The project’s main objective to change their properties. range of opportunities will emerge is to explore folding chain configu- A number of materials expe- for hackers at the atomic scale. rations for programmable matter rience a shape-memory effect Throughout the previously through both mechanical design under exernal stimuli due to their described projects, one of the and computer simulation. molecular structures: Shape- key problems we faced was Mechatronics. The field of robot- memory alloys (SMAs) return to addressing individual parts of ics and mechatronics is one of the a preprogrammed shape when the system. Currently, tabletop most vibrant related to dynamic heat is applied, and magnetic actuation (e.g., PICO [12]) solves matter, self-reconfigurable robots, shape-memory alloys experience this problem by hiding actuation and assemblies. Adaptronics by a memory effect under strong completely under the table, but Janocha [24] summarizes the sen- magnetic fields. SMAs inspired the trade-off is half a dimension sor and actuator technologies to our imaginary material Perfect lost to machinery. With advances build systems that adapt to their Red, around which we explored in quantum computers [19], in environments. Self-reconfigurable the interaction techniques for which information is stored by the materials were shown by the form giving, detailed earlier. Other state of individual atoms, system Claytronics project, which aims materials can be actuated by integration (actuation + address- to explore how modular systems driving electric current through ing) is inherent in the design. made up of nodes can be built [25]. them: Electroactive polymers and Computer science. Programmable, Large-scale prototypes have been polymer gels change their size, modular systems based on cellular built to explore the “ensemble shape, and optical properties automata or neural networks have effect,” in which multiple nodes when exposed to high currents. been a focus of computer scien- interact with each other. To scale Optical properties of objects can tists. The theoretical framework down design, the number of nodes change rapidly through certain for programmable matter was pro- has to be increased dramatically. stimuli: Thermochromic materials 2 012 posed by Toffoli and Margolus in In order to scale proportionally, change their color in response to 1991 [20], exploring control theo- assembly may become problem- heat, while halochromic materials ries on how one should “program atic. Kinematic self-replicating change their color in response to Februar y programmable matter.” machines can create nodes and acidity levels. + Later, in 1996, Abelson et al. establish a hierarchy internally [21] described how these systems between them, thus solving assem- Vision-Driven Design Research can be thought of as complex bly problems. The work of Griffith Looking back through the his- biological systems with a myriad and Jacobson [26] shows how prin- tory of HCI, we see that quantum of parallel resources. Their theo- ciples of molecular self-assembly leaps have rarely resulted from ries related fundamental laws can be applied. studies on users’ needs or market from material science to com- Material science. Material research; they have come from the

putational theory (Fitz’s law of computation science has been passion and dreams of visionar- interactions January

49 cover story

ies such as Douglas Engelbart. We believe that vision-driven design is critical in fostering quantum leaps, and it complements needs-driven Vision and technology-driven design by looking beyond current-day lim- its. Tangible Bits is an example of vision-driven research. With Applications / Radical Atoms we seek new guid- Users’ needs ing principles and concepts to help us see the world of bits and atoms with new eyes to blaze a new trail in interaction design. Technologies Figure 5 illustrates the three approaches: technology-driven, needs-driven, and vision-driven design research. The reason why we focus on the vision-driven approach is its life span. We know that technologies become obsolete in about one year, users’ needs change quickly, and applications ~ 100 Years become obsolete in about 10 years. Vision However, we believe the strong vision can last beyond our lifespan.

Applications / Conclusion ~ 10 Years Users’ needs In 1965, Ivan Sutherland envisioned a room where “computers can control the existence of matter” [28], in order to create an immersive visual Technologies ~ 1 Year and haptic sensation. Radical Atoms is our vision of human interactions with dynamic physical materials that can transform their shape, conform to constraints, and inform the users of their affordances. Radical Atoms envisions a • Figure 5. What drives design research, and why we focus on vision-driven design. change in the design paradigm of 2 012

human-computer/material interaction in which we no longer think of designing the interface, but rather Februar y of the interface itself as a mate- + rial. We may call these human- material interactions (HMIs) or material user interfaces (MUIs), in which any object—no matter how complex, dynamic, or flexible its structure—can display, embody, and respond to digital informa-

interactions January tion. Even though we may need to

50 cover story

wait decades before atom hackers Proc. of the SIGCHI Conference on Human Factors 20. Toffoli, T. and Margolus, N. Programmable mat- in Computing Systems: The CHI is the Limit (CHI ter: Concepts and realization. Physica D: Nonlinear (material scientists, self-organizing ‘99). ACM, New York, 1999, 386-393. Phenomena 47, 1-2 (Jan. 1991), 263-272. nano-robot engineers, etc.) can 5. Parkes, A., Poupyrev, I., and Ishii. H. Designing 21. Abelson, H., Allen, D., Coore, D., Hanson, C., invent the enabling technologies kinetic interactions for organic user interfaces. Homsy, G., Knight, Jr., T.F., Nagpal, R., Rauch, E., Commun. ACM 51, 6 (Jun. 2008), 58-65. Sussman, G.J., and Weiss, R. Amorphous comput- for Radical Atoms, we believe the 6. Brave, S., Ishii, H., and Dahley, A. Tangible inter- ing. Commun. ACM 43, 5 (May 2000), 74-82. exploration of interaction design faces for remote collaboration and communication. 22. Gilpin, K., Kotay, K., Rus, D., Vasilescu, I. Proc. of the 1998 ACM Conference on Computer Miche: Modular shape formation by self-disassem- techniques can begin today. Supported Cooperative Work (CSCW ‘98). ACM, bly. The International Journal of Robotics Research New York, 1998, 169-178. 27, 3-4 (Mar. 2008), 345-372. Acknowledgements 7. Frei, P., Su, V., Mikhak, B., and Ishii, H. Curlybot: 23. Gershenfeld, N. et al. The Milli Project; http:// Designing a new class of computational toys. Proc. milli.cba.mit.edu/ The vision of Radical Atoms has been of the SIGCHI Conference on Human Factors in 24. Janocha, H., ed. Adaptronics and Smart Computing Systems (The Hague, The Netherlands, developed through a series of design Structures: Basics, Materials, Design, and Applications Apr.1-6). ACM Press, New York, 2000, 129-136. workshops since 2008. More than two (2nd ed.). Springer, Berlin, Heidelberg, 2007. 8. Raffle, H.S., Parkes, A.J., and Ishii, H. Topobo: A 25. Aksak, B. et al. Claytronics: Highly scalable dozen past and current members of the constructive assembly system with kinetic memory. communications, sensing, and actuation net- Proc. of the SIGCHI Conference on Human Factors Tangible Media Group and colleagues works. Proc. of the 3rd International Conference on in Computing Systems (CHI ‘04). ACM, New York, Embedded Networked Sensor Systems (SenSys in the MIT Media Lab have contributed 2004, 647-654. ‘05). ACM, New York, 2005, 299-299. 9. Parkes, A. and Ishii, H. Kinetic sketchup: Motion to shape the concept and articulate 26. Griffith, S., Goldwater, D., and Jacobson, J. prototyping in the tangible design process. Proc. Self-replication from random parts. Nature 437, 636. the interaction techniques and possible of the 3rd International Conference on Tangible and applications. We would like to thank Embedded Interaction (TEI ‘09). ACM, New York, 27. Oxman, N. and Rosenberg, J.L. Material-based 2009, 367-372. design computation: An Inquiry into digital simu- all listed here for their contributions: lation of physical material properties as design 10. Parkes, A. and Ishii, H. Bosu: A physical pro- generators. International Journal of Architectural Amanda Parkes, Hayes Raffle, James grammable design tool for transformability with soft Computing (IJAC) 5, 1 (2007), 26-44. Patten, Gian Pangaro, Dan Maynes mechanics. Proc. of the 8th ACM Conference on Designing Interactive Systems (DIS ‘10). ACM, New 28. Sutherland, I.E. The ultimate display. Proc. of Aminzade, Vincent Leclerc, Cati York, 2010, 189-198. the IFIP Congress. 1965. Vaucelle, Phil Frei, Victor Su, Scott 11. Pangaro, G., Maynes-Aminzade, D., and Ishii, Brave, Andrew Dahley, Rich Fletcher, H. The actuated workbench: Computer-controlled About the Authors actuation in tabletop tangible interfaces. Proc. of Hiroshi Ishii is the Jerome B. Jamie Zigelbaum, Marcelo Coelho, Peter the 15th Annual ACM Symposium on User Interface Wiesner Professor of Media Arts Schmitt, Adam Kumpf, Keywon Chung, Software and Technology (UIST ‘02). ACM, New and Sciences at the MIT Media York, 2002, 181-190. Lab. Ishii’s research focuses upon Daniel Leithinger, Jinha Lee, Sean 12. Patten, J. and Ishii, H. 2007. Mechanical con- the design of seamless interfaces Follmer, Xiao Xiao, Samuel Luescher, straints as computational constraints in tabletop between humans, digital informa- tangible interfaces. Proc. of the SIGCHI Conference tion, and the physical environment. His team seeks Austin S. Lee, Anthony DeVincenzi, on Human Factors in Computing Systems (CHI ‘07). to change the “painted bits” of GUIs to “tangible Lining Yao, Paula Aguilera, Jonathan ACM, New York, 2007, 809-818. bits” by giving physical form to digital information. Williams, and the students who took 13. Piper, B., Ratti, C., and Ishii, H. Illuminating clay: Dávid Lakatos is a graduate stu- A 3-D tangible interface for landscape analysis. dent in the Tangible Media Group the Fall 2010 Tangible Interfaces Proc. of the SIGCHI conference on Human Factors in at the MIT Media Lab. His course. We thank the TTT (Things Computing Systems: Changing Our World, Changing research focuses on interaction Ourselves (CHI ‘02). ACM, New York, 2002, 355-362. design with future dynamic mate- That Think) and DL (Digital Life) con- 14. Leithinger, D., Lakatos, D., DeVincenzi, A., rials, at the intersection of sortia of the MIT Media Lab for their Blackshaw, M., and Ishii, H. Direct and gestural mechanical engineering, material design, and philosophy. support of this ongoing project. Thanks interaction with relief: A 2.5D shape display. Proc. of the 24th Annual ACM Symposium on User Interface Leonardo Bonanni completed his Software and Technology (UIST ‘11). ACM, New are also due to Neal Stephenson, whose Ph.D. and postdoctoral work with York, 541-548. science fiction novels and talks inspired the MIT Media Lab’s Tangible 15. Lee, J., Post, R., and Ishii, H. ZeroN: Mid-air Media Group, where he taught us, and Ainissa Ramirez of Yale tangible interaction enabled by computer controlled sustainable design and devel- 2 012

University, who advised us about the magnetic levitation. Proc. of the 24th Annual ACM oped interfaces for exploring the Symposium on User Interface Software and Technology lives of objects. He is founder and state of the art of material science. (UIST ‘11). ACM, New York, 2011, 327-336. CEO of Sourcemap.com, the crowdsourced direc- 16. Gibson, J.J. The theory of affordances. In tory of supply chains and carbon footprints.

EndNotes: Februar y

Perceiving, Acting, and Knowing: Toward an Jean-Baptiste Labrune is a

Ecological Psychology. Wiley, Hoboken, 1977. + 1. Ishii, H. and Ullmer, B. Tangible bits: Towards researcher at the MIT Media Lab seamless interfaces between people, bits and 17. Transcript of a talk presented by Richard P. specializing in human-machine atoms. Proc. of CHI’97. ACM Press, New York, 1997, Feynman to the American Physical Society in creativity. He is also lab director 234-241. Pasadena on December 1960; http://calteches. at Bell Labs in their application 2. Ishii, H. Tangible bits: Beyond pixels. Proc. of the library.caltech.edu/1976/1/1960Bottom.pdf domain, where he leads research 2nd International Conf. on Tangible and Embedded 18. Saito, R., Dresselhaus, G., and Dresselhaus, projects involving interdisciplinary Interaction (TEI ‘08). ACM, New York, 2008, 15-25. M.S. Physical Properties of Carbon Nanotubes. teams from the human sciences, design, and technology. 3. Weiser, M. The computer for the 21st century. University of Electro-Communications, Tokyo, 1998. Scientific American 265 (Sept. 1991), 94-104. 19. Nielsen, M.A. and Chuang, I. Quantum 4. Underkoffler, J. and Ishii, H. Urp: A luminous- Computation and Quantum Information. Cambridge DOI: 10.1145/2065327.2065337