Squidy: A Zoomable Design Environment for Natural User Interfaces Werner A. Konig Abstract Human-Computer Interaction Group We introduce the interaction library Squidy, which University of Konstanz eases the design of natural user interfaces by unifying Universiti:itsstrasse 10, Box 073 relevant frameworks and toolkits in a common library. 78457 Konstanz, Germany Squidy provides a central design environment based on [email protected] high-level visual data flow programming combined with zoomable user interface concepts. The user interface Roman Radle offers a Simple visual language and a collection of Human-Computer Interaction Group ready-to-use devices, filters and interaction techniques. University of Konstanz The concept of semantic zooming enables nevertheless Universiti:itsstrasse 10, Box 073 access to more advanced functionality on demand. 78457 Konstanz, Germany Thus, users are able to adjust the complexity of the [email protected] user interface to their current need and knowledge. Harald Reiterer Keywords Human-Computer Interaction Group Natural user interface, design environment, zoomable University of Konstanz user interface, multi modal interaction, Squidy Universiti:itsstrasse 10, Box 073 78457 Konstanz, Germany ACM Classification Keywords [email protected] H.S.2 [Information Interfaces and Presentation]: User Interfaces - Graphical user interfaces, Input devices and strategies, Interaction styles, Prototyping; 0.2.2 [Software Engineering]: Design Tools and Techniques - User interfaces. 4561 Introduction and interaction designers have to face the following One of the emerging research fields in Human­ challenges: Computer Interaction is concerned with the design and They need practical knowledge on different layers, development of so-called post-WIMP user interfaces ranging from hardware prototyping, drivers, [8], such as tangible interaction, surface computing, protocols and signal processing, to application gaze-based interaction as well as gestural and voice programming interfaces and the final application. input (see Fig. 1-3). These technologies provide a They have to deal with different more or less richer set of interaction modalities than traditional monolithic tools as well as the according program­ mouse and keyboard input and break up with standard ming language and development environment. WIMP-based user interface concepts (Window, Icon, Menu, Pointing device). The recently coined term The resulting complexity restricts rapid prototyping "Natural User Interfaces" (NUl) highlights the intention and fast design iteration processes. This may also behind these novel interaction styles: they build upon reduce comparability of the realized interaction users' pre-existing knowledge of the everyday, non­ techniques and concepts. digital world [4] and hence lead to a more natural and reality- based interaction. Few development environments are available which address these issues by supporting some novel input However, the design and development of NUls is not devices (e.g. physical turntables, mixing desks, multi­ only conceptually but also practically a very challenging touch surfaces and simple vision tracking). Two task. In contrast to the design of traditional graphical examples are MAX/MSP [5] and vvvv [9]. Both are user interfaces, it involves both software and hardware graphical development environments for music and components [2]. Yet, conventional development video synthesis and are widely used by artists to realize environments (e.g. MS Visual Studio/.Net, Adobe Flash, interactive installations. Using the concept of visual Java) fall short of supporting uncommon input devices dataflow programming both tool kits provide a simple and appropriate data processing (e.g. computer vision) way of defining and organizing the desired function­ as well as the handling of multi point and multi-user alities. However, the visual representation of each applications (e.g. for multi-touch interaction). To primitive variable, parameter, connection, and low-level Figure 1: Digital pens benefit from address this issue, a broad variety of heterogeneous instruction (e.g. matrix multiplication) lead to complex users' pre-existing knowledge and and very specialized toolkits and frameworks have and scattered user interfaces, even for small projects. thus offer a very natural mode of evolved over the last few years (e.g. Apple iPhone SDK, MAX/MSP and vvvv offer the possibility to encapsulate interaction e.g. for digital sketching consecutive instructions in so-called "abstractions" or and prototyping. Microsoft Surface SDK, NUIGroup Touchlib, GlovePIE). Researchers and interaction designers therefore have to "subpatches". This approach helps to reduce the size of choose between these different tool kits, depending on the visual dataflow graph, but introduces additional the compatibility with the specific hardware being in complexity by the hierarchical organization. Moreover, use. This situation makes the design and development the multiplicity of the provided tools and add-ons as of NUIs a lot more demanding. Particularly, researchers well as the tight coupling with visual interface 4562 components and rendering further increase the as device and data filter configuration. Squidy thereby complexity that users have to deal with. ICON Input hides the complexity of the technical implementation Configurator [1], its successor MaggLite [3] and the from the user by providing a simple visual language Openlnterface Framework [7] are further development and a collection of ready-to-use devices, filters and environments for post-WIMP user interfaces. They also interaction techniques. This faci litates rapid prototyping employ the concept of visual dataflow programming and fast iterations. However, if more functionality and and share the complexity issues with the frameworks profound customizations are required, the visual user described above. They partially reduce the visual interface provides these on demand by using the complexity by encapsulating advanced functionalities in concept of semantic zooming. Thus, users are able to pre-defined modules which are textually developed with adjust the complexity of the user interface to their a conventional programming language. Thus, the user current need and knowledge (ease of learning). Figure 2: Multi-touch surface can apply these "black-box" modules in the visual user augmented with physical tokens interface. However, one has to leave the environment User Interface Concept reduces the gap between real-world in order to develop or change a module in source code. The basic concept which enables the visual definition of and digital-world interaction . The user is therefore forced to switch between multiple the dataflow between the input and output is based on development environments in favor of a decreased the pipe-and-filter concept (Fig. 6) . This offers a very visual complexity. simple, yet powerful visual language to design the interaction logic. The user thereby selects the input To sum up, all of these development environments device or hardware prototype of choice as " source", ease the implementation of NUls by supporting some e.g. a laser pOinter, connects it successively with filter hardware devices and by providing a visual user nodes for data processing such as compensation of ~. interface to design and realize the desired interaction hand tremor or gesture recognition and routes the ~ '~ .. techniques. However, the complexity caused by refined data to the " sink". The filter nodes may crowded user interfaces (Iow-level visual dataflow transmit, change, delete data objects, or generate ~ ../' ~...-: programming) or by the need of additional tools, addit ional ones (e.g. if a gesture is recognized). The .-.~ ......... " programming languages and compilers is still a major "sink" ca n be any output modality or device such as a research issue. Furthermore, the demands on the vibrating motor for tactile stimulation or LEDs for visual designers' expertise are still very high, since they have feedback. Squidy also provides a mouse emulator as an output node to offer the possibility of controlling Figure 3: Well -known devices such to understand and ro ute each primitive variable/ data as an omnipresent laser pointer even when using " black-box" modules. standard WIMP-applications with unconventional input enhanced with button module, LEDs devices. Multipoint applications (e.g. for mult i-touch and vibration motor provide flexible Squidy - Zoomable Design Environment surfaces or multi-user environments) and remote input from any distance. Users We address these issues with our interaction library connections are supported by an output node which benefit from a more natural and conven ient po inting experience. " Squidy" which unifies various device toolkits and NUl transmits the interaction data either as TUIO messages frameworks in a common library and provides a central or as basic OSC messages over the network. TUIO is a user interface for visua l dataflow management as well widely used protocol for multipoint interaction based on 4563 the more general OpenSound Control protocol (OSC). The internal dataflow between the nodes in Squidy • X UJ El eJ ~-c:.=IIeclia Aocm / laserp<>nte" Ii P""""",,, consists of a stream of single or multiple grouped data objects of well-defined data types (Fig. 4) based on the ~Pc ___ oorn prim itive virtual devices introduced by Wallace [10]. In contrast to the low-level approaches used in related 001--