XRDSCrossroads The ACM Magazine for Students SUMMER 2010 ISSUE 16.4 The Future of Interaction Profi le: Hiroshi Ishii Looks for What’s Tangible Computers that Connect Directly to the Brain! Pen-Based Computing: Old Dog, New Tricks

XRDS.ACM.ORG The Imagine Cup is the world’s premier student technology competition. While competing for cash and prizes, students gain real life experiences, make new friends, and turn their ideas into reality. The brightest young minds join together and, using technology, take on the toughest problems facing our world today. www.imaginecup.com

© 2010 Microsoft Corporation. All rights reserved. Microsoft and the Microsoft logo are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries.

Imagine Cup 2010_ACM Advert_March v2.indd 1 3/18/2010 2:28:02 PM Crossroads The ACM Magazine for Students SUMMER 2010 VOL.16 • NO.4

The Future of Interaction

begin features end

04 LETTER FROM THE EDITOR 09 FEATURE 50 HELLO WORLD In Search of a Natural Gesture Real-Time Face Detection with 05 InbOx By Johnny Chung Lee Webcam By Dmitry Batenkov 06 UPDATES 14 FEATURE About the Redesign Pen-Based Computing 52 LAbZ By James Stanier By Gordon Kurtenbach Microsoft Research, Redmond, Washington 07 bEnEFIT 21 FEATURE By Tom Bartindale Free Software from the Microsoft Interactive Surfaces and Tangibles Developer Academic Alliance By Sergi Jordà, Carles F. Julià, 53 bACK By Daniel Gooch and Daniel Gallardo The Evolution of a Mouse By James Stanier 07 ADVICE 30 FEATURE Marketing Your Ideas Interfaces On the Go 54 EVEnTS By Daniel Lemire By Desney Tan, Dan Morris, and T. Scott Saponas 55 POInTERS

36 FEATURE 34 ACROnYMS My New PC is a Mobile Phone By Patrick Baudisch 56 bEMUSEMEnT and Christian Holz

42 FEATURE From Brains to Bytes By Evan Peck, Krysta Chauncey, Audrey Girouard, Rebecca Gulotta, Francine Lalooses, Erin Treacy Solovey, Doug Weaver, and Robert Jacob

49 PROFILE Hiroshi Ishii: Tangible Bits By David Chiu

XRDS • summer 2010 • Vol.16 • No.4 1 World Class Journals & Magazines Original froM aCM Research Publications that 50+ Journals top computing EDITORIAL BOARD EDITORIAL STAFF PUBLICATIONS & Magazines Editor-in-Chief Director, group BOARD professionals Chris Harrison, Publishing Co-Chairs Comprehensive, Carnegie Mellon Scott E. Delman Ronald F. Boisvert depend on! University, USA and Holly E. Rushmeier Managing Editor & Timely & Objective Departments Chief Senior Editor at ACM HQ board Members Tom Bartindale, Jill Duff y Jack Davidson, Nikil Coverage Newcastle University, Dutt, Carol Hutchins, UK Production Manager Ee-Peng Lim, Catherine Lynn D’Addesio Technological C. McGeoch, M. Tamer Feature Editors Ozsu, Vincent Y. Shen, Ryan K. L. Ko, Art Direction Shostak Studios, Mary Lou Soff a, Ricardo Developments Nanyang Technological Mitch Shostak, Baeza-Yates University, Singapore Corey Kuepfer Firsthand James Stanier, Advertising University of Sussex, UK SUBSCRIBE Gene Paulsson Perspectives Subscriptions ($19 Inbal Talgam, gene@pentagon-usa. per year) are available The Weizmann Institute com by becoming an Industry of Science, Israel Copyright Permissions ACM Student Member News Sumit Narayan, Deborah Cotton www.acm.org/ University of [email protected] membership/student Connecticut, USA Marketing & Non-member Department Editors Communications subscriptions: Daniel Gooch, Manager $70 per year University of Bath, UK Brian Herbert http://store.acm.org/ acmstore www.acm.org/pubs David Chiu, Public Relations The Ohio State Coordinator ACM Member Services University, USA Virginia Gold P.O. Box 11450 New York, NY 10286- Rob Simmons, 1405 USA Carnegie Mellon ACM +1 800-342-6626 Crossroads_quarter_page_Pubs_Ad.indd 1 4/21/10 12:10:08 University,PM USA Association for +1 212-626-0500 Michael Ashley- Computing Machinery Postal Information Rollman, Carnegie 2 Penn Plaza, Suite 701 Application to mail at Mellon University, USA New York, NY 10121- Periodical Postage 0701 USA Dmitry Batenkov, rates is pending at New +1 212-869-7440 The Weizmann Institute York, NY and additional of Science, Israel mailing offi ces. Copy Editors CONTACT Off ering# XRDS0163 Erin Claire Carson, General feedback: ISSN# 1528-4981 University of California- [email protected] (print) Berkeley, USA ISSN# 1528-4982 For submission (electronic) Andrew David, guidelines, please see University of Minnesota, http://xrds.acm.org Copyright ©2010 by USA the Association for Letters to the Editor: Computing Machinery, Web Editor [email protected] Inc. Permission to make Malay Bhattacharyya, Connect with us on digital or hard copies Indian Statistical Facebook: of part of this work for Institute, Kolkata, India http://tinyurl.com/ personal or classroom XRDS-Facebook use is granted without fee provided that XRDS ADVISORY Comment via Twitter: copies are not made or BOARD #xrds distributed for profi t or Bernard Chazelle, commercial advantage Princeton University, and that copies bear USA this notice and the full citation on the fi rst page Lorrie Faith Cranor, or initial screen of the Carnegie Mellon document. Copyrights for University, USA components of this work Noam Nisan, owned by others than The Hebrew University ACM must be honored. of Jerusalem, Israel Abstracting with credit is permitted. To copy William Stevenson, otherwise, republish, post Apple, Inc., USA on servers, or redistribute Jeff D. Ullman, requires prior specifi c Stanford University, permission and a fee. USA To request permissions, please contact [email protected].

2 XRDS • summer 2010 • Vol.16 • No.4

LETTER FROM THE EDITOR modalities, researchers are develop- ing new ways for us to get information The Future of Interaction into these rich platforms—what we generally refer to as input. This topic is particularly close to Interfaces Everywhere my heart and forms the core of my present PhD research. I think about ways to enable (small) mobile devices e’ve been a bit preoccu- to “steal” (large) everyday surfaces for pied with interfaces as of input. Consider, for example, a cell late. For one thing, you’re phone sitting on a desk. Why reach holding the launch issue No longer do we think over and press some diminutive but- Wof XRDS, ACM’s magazine for interfac- ton to silence an incoming call when ing with the student population. In of computing solely you could simply issue a finger ges- many ways, this is a brand-new publi- as sitting in front of ture right on the (large) table in front cation—new format, new content, and of you? Or imagine a music player new vision. But it’s also an evolution a desktop computer strapped to your upper arm while out of Crossroads, and as such, it’s staying with a keyboard and for a jog. Why reach over to interact true to a 15-year legacy of student-cen- with some tiny scroll wheel when you tric content. mouse. Computing could use the nearly two square feet occurs in cars, while of skin surface area on your lower arm Introducing XRDS for finger input? You’ve probably already noticed we’ve we’re walking or donned a fresh new look, created by riding the subway...” It’s All Happening Now world-renowned design firm Penta- Chris Harrison That might sound like science fiction, gram and the dedicated staff at ACM but these projects have already been headquarters. However, you’ll quickly published at UIST (the Symposium on discover cosmetic changes are only a User Interface Software and Technolo- small part of this redesign. You might gy) and the annual SIGCHI conference­ have also noticed we’ve put on a bit (or the Special Interest Group on Com- of weight. On the next 50 or so pages, puter-Human Interaction), two pre- you’ll discover a dozen new columns, leads for feature articles, writing col- mier ACM conferences that you, as an things like “Advice” (page 7), a tutorial umns, or reporting from conference ACM student member, get discounted called “Hello World” (page 50), “Labz” floors sounds exciting, we want to hear entrance to, by the way. (page 52), and much more. from you. That’s just the tip of the input ice- These columns, headed up by a ded- Email us ([email protected]). Join our berg. We’ve got six feature articles icated group of departments editors discussions on our Facebook group from top researchers covering every- (see masthead on page 2), were careful- page (http://tinyurl.com/XRDS-Face- thing from tangible tabletops and pen ly selected and designed to get useful book). Chatter with us via Twitter by input, to micro-device interactions, information into your hands and help using “#xrds” in any tweet. and brain-computer interfaces. In- you connect with the organizations, trigued? Keep reading… opportunities, and other students that Interfaces for Input matter most to you. These columns We’ve decided to kick off XRDS with will be recurring, so you will be able to an issue dedicated to a highly relevant jump right to the information you find and rapidly evolving subject: interfac- Chris Harrison most useful in every issue. es for input—where they are now, and Chris Harrison is a Our goal is to make XRDS the pre- where they’ll be going soon. PhD student in the mier news and information platform Advances in electronics, both in Human Computer for students interested in computer computational power and reduced Interaction Insti- science, computer engineering, infor- cost, have allowed computers to per- tute at Carnegie mation systems, interaction design, vade almost all aspects of our lives. Mellon University. and similar fields. This is one of ACM’s No longer do we think of computing He is a Microsoft chief missions, and we intend for this solely as sitting in front of a desktop Research PhD Fellowship recipient, and magazine to be a bold step toward ful- computer with a keyboard and mouse. has worked as several industry labs, filling that promise. Computing occurs in cars, while we’re including IBM Research, AT&T Labs, Interested in helping us realize this walking or riding the subway, at a kiosk and Microsoft Research. More about his vision? XRDS is not only a magazine in the airport, and even on interactive background and work is available at for students, but also run by students. tabletops and walls. To fully unleash www.chrisharrison.net. That means we need you! If digging up the true potential of these computing

4 XRDS • summer 2010 • Vol.16 • No.4 inbox scope of data aggregation in the true voice of customers. distributed databases over Once you’re in Due to its great business Cloud Computing the cloud. value, opinion mining has Although I am by no means Bibudh Lahiri, the cloud, how received a great amount an expert in the field, I Iowa State University, U.S., do you get out? of interest in the past read David Chiu’s article Facebook decade. There has been an “Elasticity in the Cloud,” Teo Fernandez explosion in research, both (issue 16.3, Spring 2010) and I was really excited to see in academia and industry, found it provocative and this issue (issue 16.3)! and various firms like topical. Cloud computing Haven’t had a chance to Lexalytics currently offer seems to be the dominant read it yet... but it’s on the opinion-mining services. paradigm of computing agenda for this week. :) Am and we’re optimistic that Denzil Correa, today and the elasticity that having to design a data programming for the cloud Facebook Mr. Chiu writes about is center for a networks class will part of the toolkit our dead on. Keep up the good I am in and the “cloud” is students graduate with. work! part of that design so this Shaun Kane, XRDS, International Clint Benjamin, Email was very timely for me. University of Washington, Awesome! chegou minha Thanks for putting this one U.S., Email ACM Crossroads. Achei que The article “Cloud together. nem fosse vir :) Computing in Plain Mary Melloy, Translation: Awesome! My English” (issue 16.3) was a Facebook Future Issues ACM Crossroads arrived. I great introduction to the I’d be glad to see a separate thought it was not going to topic, and the references The security issue of issue on “opinion mining/ come :) were very interesting cloud computing (“State sentiment analysis.” Henrique Pereira (ikkebr), overall. I think that the of Security Readiness,” Opinion mining/sentiment Santa Mari, Brazil, Twitter pervasiveness of cloud issue 16.3) raises the most analysis is a challenging computing now and the concern for myself but not text mining and natural ImagineCupの広告がある enormous impact it has as described in the article. language processing ACM Crossroads Spring 2010 on our lives will force us to If cloud computing grows problem. The main aim Translation: There is rethink how we use it going as predicted, what is in of sentiment analysis an announcement for forward and to find the place from preventing the is to discover opinions ImageineCup in ACM sweet spot between local companies controlling embedded in social media, Crossroads Spring 2010. and cloud computing that the cloud from changing news, and other online A graduate student (naka4), we haven’t quite figured policies or charging new avenues, and hence hear Japan, Twitter out yet. fees? Once you’re in the Hesham Wahba, cloud, how do you get out? How to contact XRDS Send a letter to the editors, an amazing photo that highlights your Email Teo Fernandez, work or something interesting on your campus, or other feedback by email (xrds@acm. org), on Facebook by posting on our group page (http://tinyurl.com/XRDS-Facebook), via Email Twitter by using #xrds in any message, or by post to ACM Attn: XRDS, 2 Penn Plaza, Suite 701, New York, New York 10121, U.S. Reading ACM Crossroads, it’s crazy to think how much of the computing we do is in Photo spotlight the cloud now, compared to, say four years ago. #xrds Jason(jwiese), Pittsburgh, Twitter

I liked the issue on clouds, especially the article on MapReduce research perspectives We introduced cloud (“Clouds at the Crossroads: computing and MapReduce Research Perspectives,” into our undergrad issue 16.3). I would like to database programming see more specific issues course this year. After being addressed, e.g., the initial conceptual what challenges come up hurdle, the students in offering database as a became very comfortable service (e.g., Amazon S3) applying these techniques Students at the 33rd ACM ICPC Asia Hefei Regional

Photo by Jim Grisanzio(www.flickr.com/jimgris) over the cloud, and the to computing problems, competition in China.

XRDS • summer 2010 • Vol.16 • No.4 5 begin

updates About the Redesign The Path to the New XRDS wo short years ago, ACM members saw Communications of the ACM undergo Ta remarkable transfor- mation, in both editorial content and artwork. A few months ago, a major reshuffle took place at Crossroads, too. The goal was to reshape the editorial team into a diverse, interconnected, and highly energized crew who could reignite this magazine and push it screaming forward into the new decade. In November 2009, a few Crossroads editors, including myself, assembled at ACM headquarters in New (l-r) Luke Hayman, Ryan K. L. Ko, Tom Bartindale, Chris Harrison, Scott Delman, and James Stanier. York, with Jill Duffy, senior editor at ACM and this effort was to return to that mind. We hope that they choosing the final look and magazine’s new managing founding vision. will spark you to enroll feel was finding something editor, and Scott E. Delman, As we incorporated this in a new course or grad that would make you, the group director for ACM’s change, we realized a more program, or seek out a fresh lifeblood of this magazine, publications department, important one was taking path in your career. want to be involved. This to completely revamp the place just beneath the Despite the huge revamp is your publication, and we magazine. We spent days surface. We were moving of content, probably the want you to help shape its analyzing Crossroads’ away from “student journal” most noticeable change is future. strengths and weaknesses, and toward “the ACM the design. While in New We want to know what while thinking about how magazine for students.” York, we met with Luke you think. Talk to us via we wanted to change it. It’s a subtle distinction to Hayman from Pentagram our Facebook group (http:// Looking through early some, but this publication design firm. We told him tinyURL/XRDS-Facebook), issues, which date back is for you and should be we wanted Crossroads to post on Twitter using to 1994, we saw that the exciting for you to read. feel inviting, contemporary, #xrds, or email us directly magazine’s founders While Crossroads will and young, and for articles at [email protected]. assigned a theme to each continue to accept student- printed on these pages to You may have noticed issue, a topic that was of the submitted articles, the look beautiful. After several that the name has changed utmost interest to computer editorial team now also weeks, Hayman and his as well. How did I forget? science students, and which invites feature articles colleague Rami Moghadam This isn’t Crossroads also gave the magazine by inspiring people in expertly put into place the anymore. Welcome to a sense of cohesion. Step computer science, written design you see before you. XRDS. —James Stanier number one in our redesign especially with students in A key motivator in

6 XRDS • summer 2010 • Vol.16 • No.4 5% iPhone Users by Age 17% 13% 13-17 5% 1980 18-24 13% 136 36% 29% 25-34 29% David Bradley of IBM invents 35-54 36% Years since the QWERTY keyboard “Control-Alternate-Delete” 55+ 17% layout was invented.

benefit advice Microsoft Developer Marketing Your Ideas Academic Don’t Sell Yourself Short Alliance ou found a new algorithm? Why trading quality for quantity. It is an If you’re a student ACM should we care? Somehow, you unfortunate trade, as there is a glut of member, there are dozens of must attract our attention. poor research papers, and too few high perks at your fingertips: free Richard Hamming, one of quality ones. Continuous writing, edit- courses, serious discounts, and Ythe founders of ACM, said it eloquent- ing, and rehearsal should be an inte- exclusive opportunities that ly: “The world is supposed to be wait- gral part of your activities. XRDS doesn’t want you to miss. ing, and when you do something great, One of the most exciting they should rush out and welcome it. Reach Out to Your Audience benefits is the Microsoft But the fact is everyone is busy with Scientists and engineers are most suc- Developer Academic Alliance. their own work.” cessful when their work is most avail- It’s an agreement between Marketing may seem like a dirty able. Torvalds and Berners-Lee initiat- Microsoft and the ACM to offer word to engineers and scientists, but it ed Linux and the Web, respectively, by you free (repeat: free) software. is a necessary evil. emails sent to mailing lists. Perelman The Alliance offers finished proving the Poincaré conjec- development software, Take Your Time ture by posting eprints on arXiv, an such as Visual Studio 2008 Young scientists tend to rush their pre- open electronic archive. Professional, as well as sentations. They work four months to a But posting your content and giv- Microsoft Project, Visual year on a project, yet they wait until the ing talks is hardly enough. You have C++2005 Express Edition, last minute before writing their paper to use good marketing. If you want and several other software and rehearsing their presentation— people to attend your talks, make packages. Coders will love this when they rehearse it at all. sure your title tells them why they benefit, especially those using We all have seen Steve Jobs, the CEO should attend. Think about your audi- Microsoft platforms. Who of Apple, show up for a presentation in ence. They want to know whether they wants to shell out for the latest jeans, without slides or special effects. should continue reading your paper IDE when they can download it But consider how Jobs has a clear theme or come to your talk. Convince them for free? Some of the offerings and a precise outline. He never rambles. that you have something remarkable are niche (heard of Office He has smooth transitions from topic to to tell them. Avoid jargon, acronyms, Groove? Anyone?) but seeing topic. His talks only appear laid back. In and long sentences. as it’s free, you can experiment fact, they are precisely choreographed Do not underestimate email. It is without spending recklessly. and thoroughly rehearsed. the most powerful medium at your dis- The download system is What about reports and research posal. Yet, you have to use it wisely. To straightforward and gives you papers? Rushing their publication is get famous people to read your emails, an installation key, allowing study their work. Show appreciation the software to be installed on for their results. Think of reasons why multiple machines. they might find your question or pro- The Academic Alliance posal interesting. belongs to a bigger scheme Do not underestimate For more advice, be sure to read or called the Student Academic email. It is the most listen to Richard Hamming’s 1986 talk Initiative Program, which “You and Your Research.” The tran- includes some offerings from powerful medium at script and audio file are online. Sun and CA, too. See http:// your disposal. Yet, www.acm.org/membership/ Biography membership/student/sai_ you have to use it Daniel Lemire is a professor of computer science at the University of Quebec at Montreal (UQAM). He has a blog at general/. — Daniel Gooch wisely.” daniel-lemire.com, where he sells his ideas every week.

XRDS • summer 2010 • Vol.16 • No.4 7 Knowledge, Collaboration, ACM & Innovation in Computing Resources for Student Members

XRDS: The ACM Magazine for Students ACM Local Chapters XRDS: Crossroads is the ACM magazine for students. Re-launched and Hundreds of local chapters worldwide function as geographical hubs fully redesigned, XRDS provides students with what they need to suc- of activity for both ACM members and the computing community at ceed in their current academic and future professional careers. Each large, offering seminars, lectures, and the opportunity to meet peers issue is packed with information about careers in computing, inter- and experts in many fields of interest. 500+ Student Chapters enable views and profiles of leaders in the field, and highlights from some of all members to network with their peers with similar interests. the most interesting research being done worldwide. Learn more: XRDS.acm.org ACM Conferences Through its Special Interest Groups, ACM sponsors more than 120 ACM Online Books and Courses conferences annually throughout the world, many of which offer The ACM online books program includes full access to 500 online books unique opportunities for students to get involved. from Books24x7®. In addition, student members get full access to 3,200 online courses in multiple languages and 1,000 virtual labs through ACM’s online course program. These courses are open to all ACM Stu- dent Members on a wide range of technical and business subjects. ACM Publications ACM publishes, distributes, and archives original research and firsthand perspectives from the leading professionals in computing and information technology, helping computing students and Communications of the ACM professionals get the information they need when they need it. Communications of the ACM, the flagship publication of ACM, is the leading print and online magazine for the computing and information technology fields. Industry leaders ACM Special Interest Groups use Communications as a platform to present ACM’s 34 Special Interest Groups (SIGs) represent virtually every and debate various technology implications, major area of computing. ACM SIGs offer a wealth of resources public-policies, engineering challenges and including conferences, proceedings and newsletters covering a broad market trends, and its new website features array of technical expertise and providing firsthand knowledge of the additional content, blogs and functionality. latest development trends.

ACM Digital Library ACM Electronic Services The ACM Digital Library is the definitive online resources for ACM helps student members stay informed with convenient electronic computing professionals and students, providing access to ACM’s vast services, including: TechNews, ACM’s online tri-weekly IT news digest; collection of publications and bibliographic citations from the universe CareerNews, ACM’s online bi-monthly IT news digest; Student Quick of published IT literature. Takes , ACM’s quarterly student newsletters; and MemberNet, ACM’s Member Newsletter; plus an email forwarding address and filtering service with a free acm.org email address and high-quality Postini ACM Professional Development spam filtering; and much more! ACM Student Members receive full access to software and course- ware through the ACM Student Academic Initiative (SAI). ACM has developed unique relationships with several partners to offer valuable ACM-W resources specifically for Student Members - at no additional cost! ACM-W is ACM’s membership group dedicated to encouraging the engagement of women in activities in all areas of computing. ACM’s Career & Job Center, powered by JobTarget®, offers members Join ACM-W at: https://women.acm.org/joinlist the opportunity to view and apply for a wide variety of highly targeted technology jobs. ACM also provides free mentoring services through MentorNet®, the leading organization promoting e-mentoring relationships. www.acm.org/membership/student the number of images we can mean- ingfully process per second, and the size of our fingers has not significantly In Search changed. Yet the capacity for technolo- gy to supply us with content has grown in step with Moore’s Law. In the 1980s and 1990s, the consum- of a Natural er appetite for faster technology could not be satiated. But in recent years, the information supply has started to fundamentally surpass the ability of Gesture many people to absorb it. This creates a situation of computational surplus, which, economically, should dictate a While computing has advanced exponentially, substantial drop in cost. almost explosively, since the 1970s, input devices Just a few years ago, a $100 laptop have only just begun to change. Why? was considered a magnificent dream that would change the world. Now, it is By Johnny Chung Lee possible to find a reasonably capable DOI: 10.1145/1764848.176853 netbook for $100 on a good coupon day or even “free” with a network ser- vice contract. While it would have cer- n many articles discussing the future of computing, tainly been possible to manufacture you are very likely to find either a reference to, or $100 worth of computation in 1990, very few people would have found it a motivational assumption based on, a continued satisfactory. That’s not the case today. projection of Moore’s law. This article will make no The average consumer’s demand for Iattempt to deviate from that steady tradition. more powerful technology has simply not kept up with the exponentially in- The regularity of references to “the face, trackpads, flat-panel displays, and creasing supply. law” probably extends from the fact touch screens, we still fundamentally Some have referred to this stall that human behavior and clever uses operate our computers using a single in performance demand as the era for technology are notoriously dif- 2D pointing device and a keyboard. of “good enough computing.” While ficult to predict, but the technology Yet in just the past two or three “good enough” might suggest even fur- itself has demonstrated an unwaver- years, it is not too difficult to find - ar ther reduction in device cost, what’s ing trend over the past four decades. ticles proclaiming the “death” of the happening instead is it’s becoming This trend is, of course, enabled only mouse and keyboard, or finding new economically sensible to manufacture by the astonishing accomplishments product announcements promoting a wider variety of increasingly special by the engineering teams within the new methods of input and output as its purpose computers rather than expen- companies that manufacture comput- key selling feature. Why has there been sive general purpose machines. For the ing equipment. Nevertheless, it doesn’t a recent spike in enthusiasm for new price of a nice dinner, consumers can take a huge amount of clairvoyance or interface technology? In my opinion, buy a computer that only plays music, risk-taking to claim that the trend will it is because we’ve recently crossed the only takes pictures, only shows maps, extend a bit further. inflection point—an inflection point only plays games, only plays movies, or However, interface technology has driven by Moore’s Law and the limited only lets you read the news. It’s likely not enjoyed the seven orders of magni- growth of human attention. tude in improvement of performance that core processors have achieved Consumption-Production Figure 1: The rise of diversification was a since 1970. In fact, aside from a slightly Imbalance result of a computational surplus. improved mechanical construction Over the past hundred years, the cog- and visual polish, the input and out- nitive capacity for an individual to put devices connected to the average consume and produce information desktop computer today are virtually has stayed relatively constant or has Capacity of Human Attention identical to the ones used by Douglas increased only modestly. While tech- Engelbart in his 1968 presentation, nology has certainly made it much later referred to as “The Mother of All easier to saturate our input and out- ComputationalPer Dollar SupplyDiversity of Computing Demos.” While there have certainly put channels, the rate at which we can Devices under $1K been several improvements along the read, write, speak, listen, the density way, such as the graphical user inter- of pixels our visual system can resolve, Time 2010

XRDS • summer 2010 • Vol.16 • No.4 9 In Search of a Natural Gesture

that we’ll see a significant rise in the re- “Natural interaction board. However, it has become much lease of new form factors and targeted more popular to use the term “natural” niche computing compared to what we is achieved through when referring to multi-touch interac- have in the past. (See Figure 1.) clever designs tion, motion sensing, gesture input, How is this relevant to interface and speech recognition. technology? that constrain the These input techniques certainly of- problem in ways fer a higher potential for expressing an Task-Specific Devices idea to a computer with less distortion As the diversity and specialization of that are transparent and rigid structure typically required devices increases, so does the diversity to the user.” by a mouse and keyboard. However, of interface technology. The best in- gesture and speech interfaces, in par- terfaces are typically task-specific. For ticular, have resonated well with the example, the ideal interface for choos- popular imagination. The allure of ing a radio station while driving your these methods of input is that they pro- car is not the same as the best inter- mand in exploring novel ways of inter- vide a glimpse into an easy-to-imagine face for checking your email while sit- acting with technology that permits vision of one day being able to commu- ting at your desk. As the distribution of alternative form factors, increases our nicate with a computer as easily and computing shifts away from usage sce- capacity to express an idea, or improves fluidly as we communicate with anoth- narios where a mouse and keyboard our ability to absorb information. Com- er human being using these skills we are acceptable, so does the adoption puting will be defined by how we inter- practice every day. of alternative input methods, whether act with the information rather than Now, it’s debatable whether com- touch, motion control, location, ges- by the chipsets on the motherboard, or municating with a computer in the ture, voice, or some other physiological the operating system it runs. The quest same manner that we communicate source. to create new devices dedicated to solv- with other humans is truly the most If you look at the components with- ing each of our own specialized un- desirable interface for all tasks. To in a modern laptop or mobile phone, satisfied desires is largely lead by the get a reasonably accurate picture of you’ll notice that there’s actually very search for better interface technology what a voice-and-gesture-only system little “computer” in a computer today that can better understand what we might be like, imagine if the only input (Figure 2). The largest internal compo- want, when we want it, where we want control to your computer were a video nents of a modern laptop are already it, in the way we want it. chat to a high school student sitting in those dedicated to human input and a distant room, and all you could do output. As the physical space required In Search of Nature is describe what you wanted. After a for computation continues to fall or is A phrase that has slowly received in- few minutes of saying, “Click on that. even replaced with a high-speed net- creasing traction, at least in the com- Move that ... no, not that one. The other work connection, the defining feature mercial exploration of alternative window. I mean the browser window. of the device and its suitable applica- input technologies, is “natural user Yeah. Make that bigger, I mean maxi- tions is the interface technology. interface” (NUI). While there’s no mize it,”you will probably say, “where As a result, there is a very high de- widespread consensus about the exact is my mouse and keyboard?” definition, NUI generally refers to an An often unspecified detail of that interface that is highly intuitive and vision is that the computer should be Figure 2: The human interface hard- effectively becomes invisible to the the embodiment of an exceptionally ware now dominates the form factor of user when performing a task. It is an competent and omniscient human many modern computing devices. interface that can easily and efficiently being with a reasonable personal- transmit an idea from a user’s mind ity that makes no recognition errors. into an action on the computer with But, for the sake of this article, I will little additional effort. concede there are components of that Input & Output Don Norman described the phi- vision that are conceptually desirable Computation losophy well when he said, “The real enhancements to existing interface & Storage Power & Cooling problem with the interface is that it is technologies and discuss some of its an interface. Interfaces get in the way. advantages and disadvantages. In par- Input & Output I don’t want to focus my energies on an ticular, this article will discuss the ges- Computation & interface. I want to focus on the job.” ture component in greater detail. Storage Unfortunately, the term NUI has also been coarsely applied to refer to Body Moving anything that is not a typical keyboard Body movements used to convey in- and mouse. It’s important to acknowl- formation from one person to another edge that the philosophy behind a nat- have been shown to be tightly coupled Power ural user interface is not conceptually with simultaneous speech or co-verbal & Cooling incompatible with a mouse and key- commands. According to researcher

10 XRDS • summer 2010 • Vol.16 • No.4 David McNeill’s 1992 study, 90 per- cent of these types of communicative Figure 3: Richard Bolt’s Put-That-There system combined speech and gesture input. gestures are found to be associated with spoken language. Additionally, gestures often identify underlying rea- soning processes that the speaker did not or could not articulate providing a complementary data source for inter- Bahamas North Atlantic preting a set of utterances. Hayama Thus, gesture and speech go hand- in-hand in daily human-to-human Dominican Republic communication, and it would be ap- propriate for any interactive system that attempts to provide a similar level Puerto Rico of fluidity to be designed with that in Honduras mind. Such systems that combine more than one mode of input are often called multimodal interfaces. A well known example demonstrating the power of combining speech and gesture is the Put-That-There system created by Rich- ard A. Bolt in 1980 shown in Figure 3. This system allowed the operator to sit comfortably in a chair, point his arm at a distant location on a large display wall and issue verbal commands such as “move that” and then pointing at a mation such as pointing at the object In 1989, Alex G. Hauptmann at- different location, continue the com- of conversation, or indicating the di- tempted to study the degree of consis- mand “… there.” rection for an action to be taken. Arbi- tency and variability in unprompted The gesture provided object focus trary gestures are learned motions typ- gestures when users attempted to per- and task parameters, and the speech ically used in specific communication form a three-dimensional spatial ma- component provided the task selection settings, such as the hand signals used nipulation task. The users were asked and event execution. These modalities in airplane guidance, baseball pitches, to try to perform a translation, rota- complement each other’s strengths, or infantry coordination. tion, or scaling operation on a virtual combining the physical specificity of In the context of a gestural interface wireframe cube rendered on a comput- pointing with the random access tem- prototype, arbitrary gestures are high- er display. Upon completion, a human poral nature of speech. ly popular choices in research systems operator observing the hand gesture because they can be easily designed would attempt to simulate the result- Gesture Challenges to be distinctive for the sake of recog- ing output. While there were coarse While a number of prototype systems nition and segmentation. But, these similarities in the type of gesture per- that used gesture alone have been dem- gesture sets tend to require significant formed for each of the three tasks, in- onstrated to be reasonably functional, user training, and they map to a rigid dividuals varied significantly in the many of these systems typically relied set of commands. number of fingers used, the position on a unique set of hand or body poses In general, performing complex and orientation of their hands, the that must be learned by the user, which communicative or manipulation tasks number of hands used, and the align- trigger a small number of pre-assigned using free-air gestures alone without ment of the hand movements. actions. tactile feedback or co-verbal com- Hauptmann made no attempt to In 1986, Jean-Luc Nespoulous iden- mands is actually quite unnatural. make the computing system recognize tified three classes of communicative However, there may be opportunities to and track these gestures as real interac- gestures that have come into common take advantage of the expressive power tive controls, which weakens the valid- use: mimetic, deictic, and arbitrary. of deictic and mimetic gestures to aug- ity of certain conclusions as interactive Mimetic gestures are motions that are ment or supplement interaction tasks feedback would certainly impact user intended to be representative of an because users will have a tendency to behavior. However, the findings do in- object’s shape or behavior. For exam- produce these gestures without addi- dicate that a fully functional system ple, indicating the shape of a person’s tional prompting or training. Unfortu- would have to accommodate a high de- beard, the size of a box, or the act of nately, these gestures are not typically gree of variability between users. tumbling. Deictic gestures are used to easy to segment and are subject to high Bolt attempted a partial implemen- provide context or explanatory infor- variability between individuals. tation in 1992, but this system only pro-

XRDS • summer 2010 • Vol.16 • No.4 11 In Search of a Natural Gesture

vided constrained object rotations and “Computing will be For guidance toward solutions to single-handed object placement in a this problem, it’s helpful to revisit the limited application scenario using two defined by how we philosophies behind “direct manipula- six-degree-of-freedom-tracked gloves interact with the tion” that made the graphical user in- with articulated fingers and heavily terface successful, as described by Ben relied on co-verbal input for action se- information rather Shneiderman in 1983. One of the ten- lection and control. Furthermore, the than by the chipsets ants of the design was that displaying variations Hauptmann observed oc- immediate visual feedback for input is curred in the constrained scenario on the motherboard, essential. This is the communicative where people were seated within 1 or the OS.” common ground between yourself and meter of a computer display and were the device that indicates it has an un- prompted to perform a simple spatial derstanding of what you are doing, and operation on a single object. As the that you understand what it is doing assumptions are pulled back on this in response to your actions. The num- problem, the opportunity for variation jump? What about walking? Is a one ber of degrees of freedom that can be goes up exponentially, such as allowing legged-kick a jump? What about indi- captured in the feedback should be as multiple objects on the screen simulta- viduals who jump at different heights? high as reasonably possible, giving us- neously, using non-spatial actions such ers the information they need to quick- as changing the object color, varying Guiding Gestures ly adjust their input to accomplish the the seated posture relative to the screen In this respect, freeform gesture recog- desired output. or standing at different distances, al- nition shares many of the difficulties Interactive objects should have a vi- lowing multiple users to attempt simul- of unstructured speech recognition. sual representation with understand- taneous control, and even choosing to Many spoken words have very simi- able metaphors. The visual appearance perform other peripheral tasks within lar acoustic properties. People speak of an interactive object should provide the tracking volume without activating with different accents and dialects. some affordance as to the actions or the system. Variations in body shape, There are multiple ways of expressing gestures to which it can respond. The size, and cultural background only ex- the same thought. Recognizing iso- interface should clearly provide rapidly acerbate the difficultly in interpreting lated words without additional context accessible, complimentary, and revers- a given gesture, or finding a common information is generally unreliable. ible commands. gesture for a given desired action. Recognizing speech in the presence The complexity of a gesture recog- of other noise can significantly reduce Natural is in the Design nition system is roughly proportional accuracy. Identifying when the user Regardless of the technology being to the complexity of the input vocabu- wants to engage and disengage with used, a good interface experience is lary. For example, if all that is desired the system can be challenging. Without one that is able to capture the intent is either motion or non-motion, there the speech equivalent of push-to-talk, of a user’s behavior with as little dis- are a variety of sensors such as acceler- prompted input, or escape keywords, tortion as possible. While gesture and ometers that can provide a simple data gesture interaction suffers from the speech technologies offer greater po- stream that is relatively easy to examine “Midas touch” problem of excessive ac- tential for us to express our thoughts to obtain this signal. In the case of an cidental or false activations. and ideas without thinking about the accelerometer, other data such as the Alternatively, naively applying rigid constraints of the interface, accurately magnitude and direction of motion, or structure and increasing recognition reconstructing those ideas within the the static orientation relative to gravity requirements would negate any po- computer does not come from tech- are moderately easy to extract. Howev- tential benefit from user intuition and nology for free. Natural interaction is er, as the desired expressiveness of the simply replace it with frustration from achieved through clever designs that input system goes up, so must the com- excessive false negatives. Understand- constrain the problem in ways that plexity of the gesture system. ing the strengths and weakness of a are transparent to the user but fall In effect, the device must have an particular input method is fundamen- within the capabilities of technology. A understanding of the world that is tal to understanding what combination good user experience is achieved only not only capable of distinguishing the of tools will make for a successful user through the hard work of individuals target input set, but all other similar experience. The design should provide with domain expertise and empathy gestures, in order to recognize that just enough guidance using other tech- for those who are not like themselves. they are not part of the input set. Oth- niques to prevent the user from falling erwise, the number of false positives into the poor performing areas of ges- Biography may be unacceptable. If the goal is to ture and speech recognition. If done Johnny Chung Lee is a researcher in Microsoft’s Applied Sciences Group exploring interface novel software recognize a “jump,” simply looking correctly, a relatively small amount of and hardware technology. In 2008, he graduated from for vertical movement would be insuf- recognition work can provide a delight- Carnegie Mellon University with a PhD in human-computer interaction and was named into MIT Technology Review’s ficient if a “squat” should not be con- ful experience giving the illusion that TR35. He is known for his video demonstrations of the technology has merely understood alternative applications for the Nintendo Wii remote that sidered a “jump.” Should sitting down have received more than 10 million views. and then standing up be considered a their intention.

12 XRDS • summer 2010 • Vol.16 • No.4 Crossroads ad:Layout 1 4/21/10 10:16 AM Page 1

Ever wonder...... what working in industry is like? ...about grad school? ...how you’ll manage a career and a life?

Find out! ACM has partnered with MentorNet, the award-winning nonprofit e-mentoring network in engineering, science and mathematics. MentorNet’s award-winning One-on-One Mentoring Programs pair ACM student members with mentors from industry, government, higher education, and other sectors.

• Communicate by email about career goals, course work, and many other topics. • Spend just 20 minutes a week and receive advice and support from somebody working in your field; build your pre-professional network. • Take part in a lively online community of professionals and students all over the world.

It’s FREE to all ACM members! Sign up today at: www.mentornet.net Find out more at: www.acm.org/mentornet

MentorNet’s sponsors include 3M Foundation, Alcoa Foundation, Agilent Technologies, Amylin Pharmaceuticals, Bechtel Group Foundation, Cisco Systems, Hewlett-Packard Company, IBM Corporation, Intel Foundation, Lockheed Martin Space Systems, National Science Foundation, Naval Research Laboratory, NVIDIA, Sandia National Laboratories, Schlumberger, S.D. Bechtel, Jr. Foundation, Texas Instruments, and The Henry Luce Foundation. Pen-Based Computing Pens may seem old-fashioned, but some researchers think they are the future of interaction. Can they teach this old dog some new tricks? By Gordon Kurtenbach DOI: 10.1145/1764848.1764854

hen I entered graduate school in 1986, I remember reading about the idea of using a pen as an input device to a computer. Little did I know that the idea had been around for long time, from the very early days of modern computing. Visionaries like Vannevar Bush in his famous 1945 article “As We May Think” and WIvan Sutherland’s SketchPad system from the early 1960s saw the potential of adapting the flexibility of writing and drawing on paper to computers.

The heart of this vision was that the sion, which was only a slice of the rich has been found to be valuable, along pen would remove the requirement for variety of ways a pen can be used in with where it is going. typing skills in order to operate a com- human-computer interaction. This ar- puter. Instead of typing, a user would ticle is about those other things: the Practicalities simply write or draw, and the com- ways in which pen input to a computer There are some very practical issues puter would recognize and act upon that have dramatically affected the this input. The rationale was that by adoption of pen-based systems in the supporting this “natural” expression, marketplace. Earlier work on comput- computing would be accessible to ev- “The original er input techniques, coming from a eryone, usable in broad range of tasks heritage of data entry, largely abstract- from grandmothers entering recipes, vision of pen-based ed away some of the practical differ- to mathematicians solving problems computers was that ences to present a more programmatic with the aid of a computer. or “data centric” view of computer in- Like many visions of the future, this they would bring put devices. one was inspiring but not wholly accu- the benefits of Early work on interactive computer rate. Certainly some of the key technol- graphics by Foley and Wallace clas- ogies to enable pen-based computing physical paper and sified mouse and pen input as pretty have come into fruition and have been pen to computer much the same thing: both provide an adopted widely. However, the dream x and y location and are capable of sig- of ubiquitous handwriting and draw- interaction... naling an event (a pen press or mouse ing recognition has not materialized. allowing people button press). However, later, research- One can argue that this type of tech- ers (such as Buxton) documented many nology has yet to mature but will in the to interact more subtle but important differences that future. ‘naturally’ with the affect the suitability of an input device What’s fascinating about pen-based for a specific task. For example, the computing is how it is being used in computer instead of mouse has some very practical proper- alternative ways from the original vi- typing.” ties that make it a successful and ubiq-

14 XRDS • summer 2010 • Vol.16 • No.4 uitous input device for desktop com- puters. It is a very efficient pointing Figure 1: (a) Pen input is essential to the art of sketching. Pen-based tools like device and allows the cursor location SketchBook Pro used with a Wacom tablet make powerful free-form sketching and to remain unchanged when buttons painting tools that capture the manual skills of an artist. are clicked. Similarly, the pen has a set of very a practical properties that define the contexts in which it will be effective. For example, one annoying aspect of pens is that they can be misplaced or lost, a problem that is exacerbated in the mobile device context. But it can be overcome by tethering the pen to the computer, or alternatively, the com- puter industry has recognized in many situations pointing by touch, without a pen, is sufficient. The reverse has been used to an ad- vantage too. For example, electronic white boards like Smart Board use multiple pens as an easy way to switch between ink colors when drawing. Another practical but subtle and (b) The same activity is still compelling in small formats (like the iPhone), even vexing issue with pens is that they re- drawing with the finger. (c) The quality of drawing that can be performed with quire picking up. What happens when SketchBook Mobile for the iPhone is very high. users want to switch hands or switch which input device they’re holding? b Some users become adept at keeping the pen in their hand while typing or using the mouse, but that is a generally inefficient and inelegant solution. The key observation is that there is a rich set of issues and preferences surrounding any particular computer input situation and, in many of these cases, even if perfect handwriting and drawing recognition were available, the pen would still not be a preferred choice. Pen input is not an effective input technique if you cannot find the pen, or if one can simply type faster than write or draw. c The Art of Sketching When is the pen a good choice? One task where the pen has a fundamental advantage is drawing. But even then, one has to be very careful to identify the precise drawing task. The original vision of the user draw- ing diagrams and having the computer recognize and replace the rough draw- ings with formal structured graphics is not what the pen does best. Historical- ly, the Achilles’ heel has been getting the computer to recognize properly what the user has drawn. However, if the goal is to create formal structured drawings, then why not create formal

XRDS • summer 2010 • Vol.16 • No.4 15 Pen-Based Computing

structures directly? Unfortunately for Figure 2: The user can interact with marking menus in two ways. (a) Selection can the pen, mouse-based computer inter- be made by popping up menus; or (b) once the user has memorized a path through faces for drawing structured graphics the menus, selection be made much faster with a quick mark representing the path. are effective and arguably have become a more “natural” way to create them. Meat Nowadays reverting from mouse to pen Bread would displace industry practices. However, if you need hand-drawn Staples FruitHouse & Veg graphics, like the kind you make when Misc drawing with a pen on paper, pen- based input shines. While this may Junk appear to be trivial example, its value is constantly misunderstood and un- derestimated. The appeal of sketching GroceriesHouse never fails to amaze me. I was a member of the original development team that Auto House created the first version of Autodesk Clothing SketchBook Pro, a paint application designed specifically for sketching and capturing hand-drawn ideas with all the sensitivity of drawing with pencils, Figure 3: Marking menus use the spatial mapping between a vocabulary of zigzag pens, and markers on paper (see Figure marks and a hierarchy of radial menus. This permits easy to draw marks to be 1). Recently, I was approached with the associated to arbitrary menu commands. idea of making a version of SketchBook for the Apple iPhone. While I thought it was a cool technical challenge, I could not imagine a really good reason why someone would want to use it. Fast-forward several months and the more fully featured paint program SketchBook Mobile went on sale on the iTunes App Store. Its popularity was a pleasant surprise and it became one of the most downloaded applications on the store. What was even more amaz- ing was what people were drawing with it. Figure 1b and 1c show an example. Obviously, the artists creating these works were enamored with sketching even on this small format, or rather, they were interested in sketching be- cause it was this format – the iPhone.

Pen-Based Input≠Ease of Use While the pen allows high-quality Menu hierarchy Mark set sketching, it also sets the user’s expec- tation for symbol recognition. Give pens to users, and they expect to be able Figure 4: The Unistrokes character entry technique reduces letter input to single to input handwriting and symbols and stroke marks that are easy to draw and easy to recognize. have the computer recognize them. I learned this lesson years ago demon- strating a little program that allowed the user to create circles, squares, and triangles by drawing them with a pen. I was demonstrating the program as part of our laboratory’s “open house” to the general public. People came by my booth and I explained how you could use a pen to input commands to

16 XRDS • summer 2010 • Vol.16 • No.4 the computer and showed examples draw the “C” with the pen. However, draw. Like icons and buttons, marking drawing circles, squares, and triangles complications arise when many sym- menus interactively reveal the avail- where the computer replaced the hand- bolic marks are needed, and there is no able functionality, and the computer drawn objects with “perfect” ones. I prior existing vocabulary of marks. If recognition of the marks is simple and then asked people to try it themselves. more than a dozen “command marks” reliable. Essentially, marking menus Surprisingly, most people didn’t try are needed it becomes difficult to de- combine the act of revealing functions drawing circles, squares, or triangles sign meaningful ones. In this case, it to the user and drawing a mark. but tried writing their names. People’s would be more effective to use existing A simple marking menu works expectations were that anything could graphical user interface techniques of as follows: As with a regular pop-up be recognized and this overrode any of icons, button, or menus for these com- menu, a user presses the pen down on my instructions beforehand. People ex- mands. screen and a menu pops up. This menu pect a system with very general symbol differs from the linear menus we com- recognition the moment a pen reaches Marking Menus monly see in that the menu items are their hands. A technique called marking menus displayed radially in a circle surround- The subtle lesson here is that pen was developed to address this prob- ing the tip of pen. A user can select a input or symbolic input is not inher- lem. Marking menus combine a vocab- menu item by moving in the direction ently “easy to use” because it does not ulary of marks with a pop-up graphi- of the desired item and lifting the pen reveal to the user the capabilities of the cal menu to allow a user to learn and (Figure 2). system. This is a critical insight and it use marks that are easy and fast to Marking menus are designed so is this property of “self-revelation” that makes modern graphic user interfaces “easy to use”—specifically, by display- Figure 5: InkSeine combines the advantages of the free-flow of a pen and paper ing graphics like text, icons, pictures, notebook with direct manipulation of digital media objects. and menus, the computer “reveals” to users what they can and can’t do, where, and when. We can think of graphical interaction widgets like but- tons, and menus as “self-revealing.” The method for finding out what func- tions are available and invoking those functions are combined into the same entity, namely, the button or menu. Symbolic markings made with a pen are not self-revealing. A user can draw any shape or symbol and there is nothing intrinsic in that interaction that shows the user what marks the system recognizes or what they do. One uman factors in computing systems (2007). simple method of “revealing marks” is to display a “cheat sheet,” a list that shows the correspondence between a mark and the command it invokes.

For example, a cheat sheet might show HI conference on H that drawing ”C” copies and drawing GC “V” pastes. Much research on pen-based user interfaces involves systems in which symbolic marks are used. Early work by C.G. Wolf used the symbolic lan- guage from paper and pencil proof- readers editing marks to design a sys- tem to edit text using a pen to mark up a document. Similarly, a system called MathPaper supports inputting math- his image originally appeared in Proc. of the SI ematical formulas using the pen. This approach holds the promise of com- mands being easy to remember and inckley. T en H perform. It’s easy to remember “C” K © is for copy and it is also very quick to

XRDS • summer 2010 • Vol.16 • No.4 17 Pen-Based Computing

that a user does not have to do any- items and can select them quickly by commercial CAD applications. thing special to switch between select- making a quick mark in that direction. Other similar clever ways of exploit- ing from the menu and using a mark. If Novice use is a rehearsal of expert per- ing pen input beyond recognizing tra- the user presses the pen down and hes- formance. Research has shown that ditional handwriting and symbols have itates (waiting for the system to show selection with a mark can be up to been explored. The text entry systems what’s available) the menu is displayed ten times faster than popping up the Graffitti and Unistrokes resulted from and selection can be performed in the menu, making this technique very use- research that analyzed what types of usual way by pointing to menu items. ful for frequently selected menu items. symbols are easy for a user to draw and However, if the user presses the pen Marking menus takes a meaning- easy for the computer to recognize, in down but does not hesitate and begins less vocabulary of zigzag marks and hopes of supporting handwriting input to move right away, a mark is drawn. makes correspondence to hierarchical that is easier, faster, and more reliable This way, a user can gradually move menu items (see Figure 3). The result is than traditional handwriting. Figure 4 from selecting a command via the a technique for providing an easy-to- shows how Unistrokes redesigned the menu to selecting by drawing a mark. learn, easy-to-draw, shorthand symbol alphabet to support this. Novices to the system can pop up the set to access arbitrary computer func- The concept behind marking menus menu to recall the location of a par- tions. Much research has been con- has also been applied to handwriting ticular menu items. With practice, the ducted on marking menus, and they input. Shumin Zhai and other research- user memorizes the location of menu have been successfully deployed in ers at the IBM Almaden Research Center

Figure 6: The application ShapeShop allows 3D shapes to be created quickly by inputting strokes and converting them to 3D graphics. Here the outline of the dog’s ear is drawn and will subsequently be translated into a “blob” that corresponds to the shape of the input stroke and then connected to the dog’s head.

18 XRDS • summer 2010 • Vol.16 • No.4 developed the SHARK system, a graphi- cal keyboard on which the user can in- Figure 7: A small camera in the tip of an Anoto pen allows the pen to sense and put words by dragging from key to key record what has being written and where. with the pen. The path being dragged essentially creates a symbol that repre- sents a particular word. Experiments on SHARK have showed that the user The Anoto pattern. can learn this method of input and be- Communication come proficient with it, and with prac- unit tice, the rate of input can match touch- Dot diameter 0.15mm. typing rates. As with marking menus, the displacement of Battery SHARK is a compelling example of how the dots from the grid makes the paper researchers are endeavoring to exploit “programmable”. a human’s skill with the pen and abil- Memory ity to learn, to create human computer interactions that go beyond emulating 3 mm Ink cartridge and traditional paper and pen. Ordinary paper printed with force sensor the Anoto pattern. Camera Pen, The Great Note-Taker Pen input research has also focused on the pen’s ability to be used to fluidly switch between text input, drawing and pointing. Inspired by how people combine both drawing and handwrit- ing in paper notebooks, these types of systems attempt to recreate and ampli- The full pattern is fy this experience in the digital world. approx. 60 million km2 Ken Hinckley’s InkSeine application exceeding the area of Asia and Europe is a prime example. With InkSeine a combined. user can quickly throw together notes where ink, clippings, links to docu- ments and web pages, and queries persist in-place. Figure 5 shows an ex- ample. Unlike a mouse and keyboard system, where a user must switch be- tween keyboard and mouse for text entry and pointing, InkSeine supports all these operations in a free form way, reminiscent of paper notebooks. (Ink- Seine is available as a free download from http://research.microsoft.com/ en-us/redmond/projects/inkseine/.) objects in the real world—and much progress on this general problem re- Pen Input 3D mains. However, researchers have “Researchers and Pen input can also be used to create 3D made progress with systems where graphics. The goal of this work is for a pen input is interpreted stroke by designers have user to be able to draw a perspective stroke into three-dimensional objects. made the key view of an object or scene and have the Research systems such as ILoveSketch computer automatically recognize the and ShapeShop (shown in Figure 6) al- observation that the 3D shape and recreate it accurately. low drawing on planes in 3D space or pen interactions A simple example is drawing a box in directly on to 3D surfaces. perspective and the computer auto- are distinctly matically recognizes it as cube struc- Beyond different from touch ture and creates the corresponding 3D Recently multi-touch systems, where geometry of a cube. The user can then the computer screen can sense mul- interactions and rotate the object to see it from differ- tiple finger touches simultaneously, can be exploited ent viewpoints. became a very hot topic for commer- In general, this is a computer vision cial usage. Some multi-touch systems in different and problem—recognizing shapes and are capable of sensing touch and pen interesting ways.”

XRDS • summer 2010 • Vol.16 • No.4 19 Pen-Based Computing

ible markings as the paper allowing Figure 8: The PenLight research prototype: A digital pen senses its location on a the pen to identify the object and the building blueprint and a micro projector mounted on the pen displays an overlay of location of markings. Drawing a door- information accurately positioned on the blueprint. way on the side of physical model of the building causes the doorway to be added to the virtual model. Pushing these ideas further, re- search has explored augmenting these “super pens” with output devices, like a LED screen that displays a line of text and an audio speaker. The PenLight system pushes this even further and uses a micro projector mounted on the pen that allows it to act like an “infor- mation flashlight.” With the PenLight, the camera in the pen not only knows the location of the pen relative to the document it is over, but with the micro projector it can overlay relevant infor- mation. Figure 8 shows the Pen Light system being used over a blueprint for a building.

The Pen to Come The original vision of pen-based com- puters was that they would bring the benefits of physical paper and pen to computer interaction by utilizing input. Work by Balakrishnan has ital technologies to be combined. handwriting input and free-form draw- shown systems where the dominant This concept has been adapted from ing, allowing people to interact more hand holds the pen and non-domi- paper to physical 3D objects. Guimb- “naturally” with the computer instead nant hand controls the frame of refer- iere and Song developed a computer- of typing. However, because reliable ence (e.g., rotation of the image being aided design system where a user could handwriting recognition wasn’t avail- drawn on) are effective and desirable make editing marks on 3D physical ob- able and people needed typeface text ways of interacting. Work by Hinckley jects using an Anoto pen. The objects, for a majority of tasks, the keyboard has shown the benefits of a simple de- printed with a 3D rapid prototyping and mouse became more popular—so sign in these multi-touch and pen sys- printer, have the same type of invis- much so that the keyboard and mouse tems where “the pen writes,” “touch now seem to be the “natural” means of manipulates,” and “the two combine interacting with a computer. for special functions.” But the pen remains essential for Ultimately, how to use the pen in “Some super pens, some tasks, like sketching and free combination with multi-touch will be form idea input, and in these applica- largely determined by needs of appli- like Anoto’s, can tions it has found success. Further- cations. However, researchers and de- store handwriting, more, researchers explored ways of us- signers have made the key observation ing pen input beyond emulating pen that the pen interactions are distinctly markings, and pen and paper, such as marking menus different from touch interactions and movements, and and PenLight, and this research has can be exploited in different and inter- resulted in useful, successful and in- esting ways. identify on which teresting technologies. Using this type Research has looked at ways of aug- document they were of thinking, many interesting and in- menting the pen with special hardware spiring explorations for pen input re- functions to create “super pens.” The made and precisely main ahead. Anoto pen is a digital pen with a tiny where the markings camera embedded in its tip (Figure 7). Biography The pen can store handwriting, mark- are. This allows the Dr. Gordon Kurtenbach is the director of research at Autodesk Inc., where he oversees the Autodesk Research ings, and pen movements, and identify world of paper and group whose focus is on human-computer interaction, on which document they were made graphics and simulation, environment and ergonomics, and high-performance computing. He has published and precisely where the markings are. digital technologies numerous research papers and holds more than 40 This allows the world of paper and dig- to be combined.” patents in the field of human-computer interaction.

20 XRDS • summer 2010 • Vol.16 • No.4 Interactive Surfaces and Tangibles Tap. Slide. Swipe. Shake. Tangible user interfaces have some scientists toying around with stuff you can really put your hands on. By Sergi Jordà, Carles F. Julià, and Daniel Gallardo DOI: 10.1145/1764848.1764855

n the last decade, human-computer interaction research has witnessed a change in focus from conventional ways to control and communicate with computers (keyboard, joystick, mouse, knobs, levers, buttons, etc.) to more natural and unconventional devices, such as gloves, speech recognition tools, eye trackers, cameras, and tangible Iuser interfaces. With the advent of the Internet and the ubiquity of personal computers in the 1990s, the graphical user interface (GUI) emerged as the pervasive interface that both users and designers had to deal with. At the same time, albeit lesser known at that time, computing started to progressively move beyond the desktop into new physical and social contexts as a result of both technological advances and a desire to surpass the WIMP— window, icon, menu, pointing device—limitations.

Nowadays, when “multi-touch” We will conclude by discussing some plication areas where TI is showing and “the two-finger pinch-zoom” are of the key benefits of this type of in- more promising results. part of a user’s daily life, the area of teraction, surveying some of the ap- tangible interaction seems to have Tangible Interaction finally entered the mainstream. De- In a way, tangible interaction can be spite this, many do not realize that seen as an extension and deepening tangible interaction conveys much of the concept of “direct manipula- more than sharing photo collections “While multi-touch tion,” a term that was first introduced or navigating Google Maps using our mobile phones by Ben Shneiderman in 1983 within fingers. the context of office applications In this article we give an overview should increase and the desktop metaphor [32]. Since of tangible interaction and tangible the richness of then, it has also become closely as- user interfaces, their conceptual sociated with GUIs and WIMP-based origins, and some of their seminal interaction of these interaction. While WIMP-based GUIs implementations. We’ll also describe interfaces, at the always incorporate direct manipula- some of the more popular current tion to some degree, the term does not tendencies such as multi-touch and moment we only find just imply the use of windows or even interactive tabletops and surfaces. basic gestures.” graphical output. The idea behind

XRDS • summer 2010 • Vol.16 • No.4 21 Interactive Surfaces and Tangibles

direct manipulation was to allow us- ronments, literally allowing users to some prototypes such as a multi-touch ers to “directly manipulate” objects grasp data with their hands, thus fus- tablet [22]. In 1995, Buxton together presented to them, using actions that ing the representation and control of with then PhD students Fitzmaurice would loosely correspond to the physi- digital data and operations with phys- and Ishii, demonstrated how the sens- cal world, assuming that real-world ical artifacts. (See also in this issue a ing of the identity, location and rota- metaphors for both objects and ac- profile of Ishii on page 49.) tion of multiple physical devices on tions would make it easier for them to a digital desktop display, could be learn and use an interface. Pioneers: 1983-1997 used for controlling graphics using Tangible user interfaces combine The use of the nowadays-ubiquitous both hands. This work introduced the control and representation in a single pinch gesture so cherished by Apple notion of graspable interfaces [11], physical device [34]. With GUIs, us- iPhone users dates back at least to which were to become two years later ers interact with digital information 1983 [35]. At that time, the media art- “tangible interfaces.” From the same by selecting graphic representations ist and engineer Myron Krueger, pas- research team is also the Active Desk (icons, windows, etc.) with pointing sionate about unencumbered rich (see Figure 2), one of the first table- devices, whereas tangible interaction gestural interaction (no mice, no top systems that combined a sensing emphasizes tangibility and materiali- gloves) was already working with vi- camera and a projector [4]. ty, physical embodiment of data, bodi- sion capture systems that were able to Perhaps conceptually closer to the ly interaction, and the embedding of track the users’ full bodies, as well as notion of grabbing and manipulat- systems in real spaces and contexts. their hands and multiple fingers, al- ing information is the work of the ar- Hiroshi Ishii at the MIT Media Lab lowing them to interact using a rich chitect and professor of design John coined the term tangible user inter- set of gestures [20, 21]. Whereas in his Hamilton Frazer, who as early as in face in 1997 [17], although several re- work Video Place (1983) users interact- the late 1970s started working on what lated research and implementations ed with a big screen, predating Sony’s he called “Intelligent Physical Model- predate this concept. Ishii picked the EyeToy (2002) by 20 years (see Figure ing Systems”, developing a set of intel- abacus as the source of inspiration 1), Video Desk (1983), based on a hori- ligent cubes that would know the po- and the ultimate tangible interaction zontal desktop configuration, showed sition of its surrounding neighbors. metaphor because, unlike pocket cal- the use of pinch and other two-finger The purpose of these first prototypes culators or computers, in the abacus, and two-hand gestures for redefining was to build a sketching system that input and output components coin- objects’shapes, scaling them or trans- would let people easily and intuitively cide and arithmetical operations are lating them. work and prototype in three dimen- accomplished by the direct manipu- One year later, the same year the sions, thus permitting users of future lation of the results. first Macintosh computer was re- buildings, without any special design Following this captivating idea, leased, Bill Buxton started research- knowledge, to be directly involved in Ishii envisioned TUIs as interfaces ing multi-touch and bimanual input their conception process [12]. Thirty meant to augment the real physical [5], first at the University of Toronto years later, 2D or 3D “intelligent build- world by coupling digital information and later at Xerox Park and at the ing blocks,” such as David Merrill’s 2D to everyday physical objects and envi- Alias|Wavefront company, developing Siftables [24] (see Figure 3), still con- stitute an emerging and promising trend inside TI! Figure 1: The model-view-controller in tangible interaction (inspired by Hiroshi Ishii). Consolidation: 1997-2005 The TUI “consolidation phase” can be considered to start with the publica- tion of Ishii and Ulmer’s seminal pa- per “Tangible Bits: Towards Seamless Interfaces between People, Bits and e.g. visual feedback Atoms” [17], and the subsequent work sound carried out by his team at the MIT nongrapable representation control Media Lab. A first student of Buxton, Ishii helped conceptualize the philos- tangible representation ophy behind tangible interaction and, physical together with his own students at the intangible representation Media Lab, he started to conceive and digital implement new developments that demonstrated the potential of TUIs. Their Sensetable [28] used a top- model digital information down projected system and an electro- magnetic tracking system for detect- ing the positions and the orientations

22 XRDS • summer 2010 • Vol.16 • No.4 Figure 2: Bill Buxton interacts with the Active Desk.

of multiple wireless objects on the Figure 3: David Merrill’s Siftables are known as “intelligent building blocks.” table surface. It became the base for most of the tabletop-related projects developed at the Media Lab in the fol- lowing years, including the Audiopad, one of the first musical tabletops. Around the same time, Mitsubishi Electric Research Laboratories start- ed to explore multi-user and multi- touch tabletop interfaces with Dia- mondTouch [7], a rectangular table focused on supporting meetings and discussions, and the first and still only one capable of distinguishing between users. The Reactable is a collaborative musical tabletop conceived and devel- oped since 2003 at the Pompeu Fabra University in Barcelona (see Figure 4). It provides both multi-touch and tan- gible objects interaction by means of reacTIVision, an open-source, cross- platform computer vision framework for the tracking of fiducial markers and combined multi-touch finger tracking [2]. Specially developed for

Photo by Amit Zoran; small portraits by Orit Zuckerman, MIT Media Lab the Reactable project, reacTIVision

XRDS • summer 2010 • Vol.16 • No.4 23 Interactive Surfaces and Tangibles

Figure 4: The Reactable.

is nowadays widely spread among sion tracking technology, which analyz- YouTube finally pushed multi-touch the tabletop developer community. es the user’s shadows to detect contact from the academic world fully into the The Reactable was first presented in a with the interface, was also innovative. larger technology space. concert in 2005 [19], but two years lat- With changes and additions, most of er it accomplished unparalleled mass the work carried out by Wilson and his Tabletops’ Affordances popularity (considering it is an aca- team at Microsoft Research would lead and Typologies demic work), when millions of people to the presentation of Microsoft’s com- Several of the early aforementioned watched the demo on YouTube. Not mercial tabletop, the Surface, in 2007. prototypes were in fact tables that of- long after, Björk used it in her 2007 Above everything else, the most ten allowed their users to interact in world tour. influential work on the multi-touch two complementary ways: touching PlayAnywhere [36] from Microsoft field, arguably leading to its hype and the table’s surface directly, and manip- Research seems to be the first attempt consolidation, was Jeff Han’s “low-cost ulating specially configured real physi- at building a compact and portable multi-touch sensing through frustrat- cal objects on its surface. multi-touch surface (see Figure 5). By ed total internal reflection” 15 [ ]. He Typically, this type of interface al- handling a projector on the side instead demonstrated a new, precise, and inex- lows more than one input event to of above the interface, PlayAnywhere pensive technology for rear-projected enter the system at the same time. In- also partially solved a common prob- multi-touch interfaces, along with gor- stead of restricting input to an ordered lem with most of the contemporary geous demo applications that captivat- sequence of events (click, click, double interfaces—occlusion. Its computer vi- ed the common user. Han’s demos on click, etc.), any action is possible at any

24 XRDS • summer 2010 • Vol.16 • No.4 time and position, by one or several si- “Nowadays, when many individual use cases, if tangible multaneous users. objects where not to be supported, Multi-touch interaction is arguably ‘multi-touch’ and tilted interfaces (like traditional archi- the most commercially successful ca- ‘the two-finger tect’s tables) seem to be more engaging pability of horizontal surfaces. And the and feel more natural [26]. “pinch-zoom” technique is only one ex- pinch-zoom’ are part An early example of a tangible ta- ample of hundreds of possibilities. of a user’s daily life, bletop interface, which may allow us The other implicit capacity of table- to unveil and better understand its shaped interfaces is the ability to liter- the area of tangible potential, is Urban Planning (URP), a ally support physical items on them. interaction seems to system developed as a town planning Users can interact with objects of vol- aid in the late 1990s at the MIT Media ume, shape, and weight, and when have finally entered Lab [3]. URP is a town-planning simu- the tracking system is able to identify the mainstream.” lator in which various users can ana- these objects and track their position lyze in real time the pros and cons of and orientation, the potential band- different urban layouts by arranging width and richness of the interaction models of buildings on the surface of goes thus far beyond the simple idea of an interactive table, which represents multi-touch. Interacting with the fin- fill the seminal ideal of tangible inter- the plan of a town or a district. The gers still belongs to the idea of point- action, of “adding digital information surface provides important informa- ing devices, while interacting with to everyday physical objects,” allowing tion, such as building shadows at dif- physical objects can take us much digital entities to coexist as fully digi- ferent times of the day. farther. Such objects can represent tal non-physical form and as shared URP is executed on an augmented abstract concepts or real entities. They digital-physical form. table with a projector and a camera can relate to other objects on the sur- It should be stressed, though, that pointed at the surface from above. face. They can be moved and turned not all existing tabletops technolo- This system permits the detection of around on the table surface, and all gies allow dual interaction, and others, changes in the position of the physical these spatial changes can affect their such as SMART Table 2003 and Dia- objects on the table and also projects internal properties and their relation- mondTouch, support only multi-touch. visual information concerning the sur- ships with neighboring objects. To be able to place objects on a surface, face. Other elements can be included, Additionally, with the combination the tabletop must be horizontal—not such as a clock to control the time of of the tracking of control objects on the tilted. In that sense, some recent stud- day in the system. URP detects any table with projection techniques that ies analyze the benefits and disadvan- changes made in real time and proj- do convert the table into a flat screen- tages of tilted versus horizontal surfac- ects shadows according to the time of ing surface, these systems can also ful- es (for example [25]), suggesting that in day. All these properties could obvious- ly be achieved with the usual mouse- controlled software on a conventional Figure 5: PlayAnyWhere from Microsoft Research. screen, but the interest of this simple prototype does not lie in its capacity to calculate and simulate shadows, but rather in the way in which informa- tion can be collectively manipulated, directly and intuitively. This example, although quite simple, already unveils some of the most important benefits of tabletop interaction: collaboration, naturalness, and directness.

Multi-user Collaboration The social affordances associated with tables directly encourage concepts such as “social interaction and collab- oration”[16] or “ludic interaction” [14]. Many researchers do in fact believe that the principal value of tangible in- terfaces may lie in their potential for facilitating kinds of collaborative ac- tivities that are not possible or poorly supported by single user technologies [23]. A research community has been

XRDS • summer 2010 • Vol.16 • No.4 25 Interactive Surfaces and Tangibles

Figure 6: The Reactable, a collaborative musical tabletop, has a circular surface for interaction.

Figure 7: Turtan, a Logo tangible programming project, is circular like Reactable.

26 XRDS • summer 2010 • Vol.16 • No.4 growing around these technologies Some tabletop implementations set of tangible resources for children and the concept of “shareable interfac- have even strengthened this collabora- in which tangible artifacts are better es,” a generic term that refers to tech- tive aspect with idiosyncratic design understood as resources for shared nologies that are specifically designed decisions. Such is the case of the col- activity rather than as representa- to support groups of physically co-lo- laborative tabletop electronic music tions of shared information. Fer- cated and co-present to work together instrument Reactable [18] (see Figure naeus, Tholander, and Jonsson [10] on and around the same content. 6), the Logo tangible programming identify a ”practice turn” in tangible Until recently, most research on project Turtan [13] (see Figure 7) or the interaction and HCI in general, that computer-supported cooperative work Personal Digital Historian System [31], is moving from a data-centric view (a term coined by Irene Greif and Paul all of which are based on circular ta- of interaction to one that focuses on M. Cashman in 1984), has more often bles and use radial symmetry, for pro- representational forms as resources concentrated on remote collaboration. moting collaboration and eliminating for action. Instead of relying on the But if we restrict ourselves to collocated head position, leading voices, or privi- transmission and sharing of data, the collaboration, the type of devices used leged points of view and control. action-centric perspective is looking (for example screens versus tables) for solutions that emphasize user con- seem to make a manifest difference. Sharing Control trol, creativity, and social action with Rodgers and Rodden have shown Sharing data between users, for exam- interactive tools. for example that screen-based systems ple in the form photo collections (for No matter how influential this para- inevitably lead to asymmetries con- example [6, 31]), probably constitutes digm shift may be felt in a near future, cerning the access and the creation of nowadays, together with map naviga- the truth is that it is hard to think on information [29]. While these systems tion, the most popular demo for table- shared control when models and inspi- make possible for all participants to top prototypes (for example Microsoft rational sources comes from WIMP- view the external representations be- Surface). Communication in general is based, single-user interactive com- ing displayed (for example, through definitely about sharing data 30[ ], but puter applications. Much of the efforts using whiteboards and flipcharts), it is it is not about sharing documents or taken until today in the field of CSCW more difficult for all group members files—it is about sharing real-time, on- have been in that direction, trying to to take part in creating or manipulat- the-fly-generated data. convert single-user applications into ing them. In particular, one person This idea of sharing control versus multi-user collaborative applications. can often dominate the interactions by sharing data is indeed becoming more But sequential interaction has proved monopolizing the keyboard, mouse, or frequent on tabletops. One clear exam- to be too inflexible for collaborative pen when creating and editing a docu- ple is Reactable [18], a multi-user musi- work requiring concurrent and free in- ment on a shared interactive white- cal instrument that is better described teractions [1, 8, 27, 33]. board. Once a person is established in as a contraption for sharing real-time Sharing control is problematic and a particular role (for example note-tak- control over computational actions, cumbersome in WIMP-based interac- er, mouse controller) she or he tends rather than for sharing data among its tion. While synchronicity problems to remain in it. Moreover, those not in users. and inconsistencies caused by simul- control of the input device, can find it This idea of sharing control versus taneous accesses can be ”solved“ and more difficult to get their suggestions the sharing of data is indeed strongly managed, this interaction model does and ideas across. linked to music performance, but clearly not constitute the best inspira- Rodgers et al. [29] have done some tangible applications with a similar tional source for the new types of col- user studies around interactive tables, philosophy are also becoming more laboration we envision tangibles can for learning the new opportunities for frequent in non performance-related convey. collaborative decision-making that domains. The Patcher [9] presents a shared interactive tabletops can pro- Mass Market vide. They conclude that collaborative If we were to look at the present com- decision-making can indeed be pro- mercial success of the technologies moted by providing group members “It is hard to and interaction models we have de- with equal access and direct interac- scribed, it seems that the only one tion with digital information, dis- think about currently reaching the mass market played on an interactive table surface. shared controls is multi-touch. The new generation of They observe that these interfaces also mobile phones advertises the ability of foment discussion, and that the shar- when models and recognizing multi-touch gestures, and ing of ideas and invitations to others to inspirational sources the same happens to the now changing take a turn, to respond, confirm, or to market of personal computers: the new participate, all tended to happen at the comes from WIMP- versions of operating systems already same time the participants were inter- based, single-user add some support for multi-touch in- acting with the table, supporting their put on the screen. opinions with gestures and the table interactive computer While this should increase the rich- responses. applications.” ness of interaction of these interfaces,

XRDS • summer 2010 • Vol.16 • No.4 27 Interactive Surfaces and Tangibles

independence in replicated application sharing 21. Krueger, M. W., Gionfriddo, T., Hinrichsen, K. 1985. at the moment we only find basic ges- systems. ACM Transactions on Computer-Human VIDEOPLACE—An artificial reality. In CHI ‘85: tures such as pinch-and-zoom and Interaction (TOCHI), 6, No. 2. 95-132. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 35-40. similar ones that are limited to two fin- 2. Bencina, R., Kaltenbrunner, M. and Jordà, S. 2005. Improved Topological Fiducial Tracking in the 22. Lee, S.K., Buxton, W., Smith, K. C. 1985. A multi-touch gers. It’s a starting point, but we’re still reacTIVision System. In Proceedings of the IEEE three dimensional touch-sensitive tablet. SIGCHI85 International Workshop on Projector-Camera ACM. 21-25. missing the main benefits of multi- Systems. 23. Marshall, P., Rogers, Y. and Hornecker, E. 2007. Are touch technologies: full hand, biman- 3. Ben-Joseph, E. and Ishii, H. and Underkoffler, J. and Tangible Interfaces Really Any Better Than Other Kinds ual and multi-user interaction. Piper, B. and Yeung, L. 2001. Urban simulation and the of Interfaces? Workshop on Tangible User Interfaces luminous planning table: Bridging the gap between the in Context and Theory, ACM CHI 2007. Sadly, these restrictions come from digital and the tangible. Journal of Planning Education 24. Merrill, D., Kalanithi, J., and Maes, P. 2007. Siftables: and Research. intrinsic limitations of cell phones and towards sensor network user interfaces. In computer screens which are specifical- 4. Buxton, W. 1997. Living in Augmented Reality: Proceedings of the 1st International Conference on Ubiquitous Media and Reactive Environments. In Video Tangible and Embedded Interaction (Baton Rouge, ly designed for single user interaction. Mediated Communication, K. Finn, A. Sellen & S. Wilber Louisiana, February 15-17, 2007). New York, NY. 75- Additionally, in the case of phones, (eds.). Hillsdale, N.J.: Erlbaum, 363-384. 78. their small size limits full hand or bi- 5. Buxton, W., Myers, B. 1986. A study in two-handed 25. Morris, M.R., Brush, A.J., Meyers, B.R. 2008. A field input. In SIGCHI86 ACM. 321-326. study of knowledge workers’ use of interactive manual interaction. On the personal horizontal displays. In 3rd IEEE International 6. Crabtree, A., Rodden, T., and Mariani, J. 2004. Workshop on Horizontal Interactive Human Computer computers side, developers are strug- Collaborating around collections: informing the Systems, 2008. TABLETOP 2008. 105-112. gling for adapting multi-touch to the continued development of photoware. In Proceedings of the 2004 ACM Conference on Computer Supported 26. Muller-Tomfelde, C. and Wessels, A. and Schremmer, rigid structure of WIMP, which pre- Cooperative Work (Chicago, Illinois, November 6-10, C. 2008. Tilted tabletops: In between horizontal vents the apparition of most of the fea- 2004). CSCW ‘04. ACM, New York, NY. 396-405. and vertical workspaces. In 3rd IEEE International Workshop on Horizontal Interactive Human Computer 7. Dietz, P., Leigh, D. 2001. DiamondTouch: a multi-user Systems, 2008. TABLETOP 2008, 49-56. tures of multi-touch. touch technology. UIST ‘01: In Proceedings of the 14th Tabletop interfaces favor multi-di- Annual ACM Symposium on User Interface Software 27. Olson, J., Olson, G., Strorrosten, M., and Carter, M. and Technology. 219-226. 1992. How a group-editor changes the character of a mensional and continuous real-time design meeting as well as its outcome. In Proceedings 8. Stefik, M., Bobrow, D., Foster, G., Lanning, S., and of the ACM Conference on Computer-Supported interaction, exploration and multi-user Tatar, D. 1987. WYSIWIS revised: Early experiences Cooperative Work. 91-98. collaboration. They have the potential with multiuser interfaces. ACM Transactions Office Information Systems, 5, No. 2. 147-167. 28. Patten, J., Ishii, H., Hines, J., Pangaro, G. 2001. to maximize bidirectional bandwidth Sensetable: A wireless object tracking platform for 9. Fernaeus, Y. and Tholander, J. 2006. Finding Design tangible user interfaces. In CHI ‘01: Proceedings of the while also contributing to delicate and Qualities in a Tangible programming space. In SIGCHI Conference on Human Factors in Computing Proceedings CHI 2006, Montreal. intimate interaction, and their seam- Systems. 253-260. 10. Fernaeus, Y., Tholander, J., and Jonsson, M. 2008. less integration of visual feedback and 29. Rogers, Y. and Rodden, T. 2004. Configuring spaces Beyond representations: Towards an action-centric and surfaces to support collaborative interactions. In physical control allows for more natu- perspective on tangible interaction. International O’Hara, K., Perry, M., Churchill, E. and Russell, D. (eds.) Journal of Arts and Technology. ral and direct interaction. They are Public and Situated Displays. Kluwer Publishers. 45- promising interfaces, but while some 11. Fitzmaurice, G. W., Ishii, H., Buxton, W. 1995. Bricks: 79. laying the foundations for graspable user interfaces. 30. Shannon, C.E. 1948. A Mathematical Theory of vendors are starting to push tabletop In CHI ‘95: Proceedings of the SIGCHI Conference on Communication. Bell System Technical Journal, 27, Human Factors in Computing Systems. 442-449. products on the market, their penetra- 379-423, 623-656, July, October. 1948. 12. Frazer, J. 1995. An evolutionary architecture. tion is still minimal. However, we are 31. Shen, C., Lesh, N. and Vernier, F. 2003. Personal Architectural Association. Digital Historian: Story sharing around the table. confident that tabletops will be soon 13. Gallardo, D., Julia, C.F. and Jordà, S. 2008. TurTan: Interactions, 10, No. 2, 15-22. A tangible programming language for creative ubiquitous, not necessarily in the 32. Shneiderman, Ben. 1983. Direct Manipulation: A step exploration. In 3rd IEEE International Workshop on beyond programming languages. IEEE Computer, 16, desktop, but on the meeting rooms, Horizontal Interactive Human Computer Systems, 57-69. the stages, and wherever complex so- 2008. TABLETOP 2008. 89-92. 33. Sun, C., Xia, S., Sun, D., Chen, D., Shen, H. and Cai, 14. Gaver, W.W., Bowers, J., Boucher, A., Gellerson, H., cial interaction, exploration, and col- W. 2006. Transparent Adaptation of Single-User Pennington, S., Schmidt, A., Steed, A., Villars, N. and Applications for Multi-User Real-Time Collaboration. laboration take place. Walker, B. 2004. The drift table: designing for ludic In ACM Transactions on Computer-Human Interaction, engagement. In CHI ‘04 Extended Abstracts on Human 13, No. 4. 531-582. Factors in Computing Systems, ACM Press. 885-900. Biographies 34. Ullmer, B. and Ishii, H. 2001. Emerging Frameworks 15. Han, J. Y. 2005. Low-cost multi-touch sensing Sergi Jordà holds a BS in fundamental physics and a PhD for Tangible User Interfaces. In Human-Computer through frustrated total internal reflection. In in computer science and digital communication. He is a Interaction in the New Millennium, ed. John M. Carroll, Proceedings of the 18th Annual ACM Symposium on researcher in the Music Technology Group of Universitat Addison-Wesley. 579-601. User Interface Software and Technology (Seattle, WA, Pompeu Fabra in Barcelona, and a lecturer at the same October 23-26, 2005). UIST ‘05. ACM, New York, NY. 35. Westerman, W. 1999. Hand tracking, finger university, where he teaches computer music, HCI, and 115-118. identification, and chordic manipulation on a multi- interactive media arts. He is one of the inventors of the touch surface. University of Delaware PHD thesis. Reactable. 16. Hornecker, E. and Buur, J. 2006. Getting a grip on tangible interaction: A framework on physical space 36. Wilson, A. D. 2005. PlayAnywhere: a compact Carles F. Julià is a PhD student in the Music Technology and social interaction. In Proceedings CHI 2006. 437- interactive tabletop projection-vision system. In UIST Group at Universitat Pompeu Fabra. He teaches 446. ‘05: Proceedings of the 18th Annual ACM Symposium interactive systems to undergraduates and tangible and on User Interface Software and Technology. 83-92. tabletop interfaces in a professional master in Interactive 17. Ishii, H., Ullmer, B. 1997. Tangible bits: towards Musical Systems. He has a background in computing seamless interfaces between people, bits and atoms. science, cognitive science, and interactive media. In CHI ‘97: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 234-241. Daniel Gallardo is a PhD student in the Music Technology Group at Universitat Pompeu Fabra, working on a thesis 18. Jordà, S., Geiger, G., Alonso, M., Kaltenbrunner, M. on a new tabletop prototype founded by Microsoft 2007. The reacTable: exploring the synergy between Research Lab (Cambridge, UK). He teaches CS and live music performance and tabletop tangible telecommunications, and a course in physical computing interfaces. In TEI ‘07: Proceedings of the 1st in a professional master in Interactive musical systems International Conference on Tangible and Embedded design. His background is in computer science, and he has Interaction. 139-146. a masters in cognitive systems and interactive media. 19. Jordà, S., Kaltenbrunner, M., Geiger, G. and Bencina, R. 2005 The reacTable*. In Proceedings of the References International Computer Music Conference (ICM05). 1. Begole, J., Rosson, M. and Shaffer, C. 1999. Flexible 20. Krueger, Myron, W. 1991. Artificial Reality II. Reading, collaboration transparency: Supporting worker MA: Addison-Wesley.

28 XRDS • summer 2010 • Vol.16 • No.4 queue_CACM_ad.qxp:acmqueue 2/10/10 2:30 PM Page 1

BLOGS ARTICLES COMMENTARY CTO ROUNDTABLES CASE STUDIES MULTIMEDIA RSS

acmqueue has now moved completely online!

acmqueue is guided and written by distinguished and widely known industry experts. The newly expanded site also offers more content and unique features such as planetqueue blogs by queue authors who “unlock” important content from the ACM Digital Library and provide commentary; videos; downloadable audio; roundtable discussions; plus unique acmqueue case studies.

acmqueue provides a critical perspective on current and emerging technologies by bridging the worlds of journalism and peer review journals. Its distinguished Editorial Board of experts makes sure that acmqueue's high quality content dives deep into the technical challenges and critical questions software engineers should be thinking about.

Visit today! http://queue.acm.org/ Interfaces on the Go Enabling mobile micro-interactions with physiological computing. By Desney Tan, Dan Morris, and T. Scott Saponas DOI: 10.1145/1764848.1764856

e have continually evolved computing to not only be more efficient, but also more accessible, more of the time (and place), and to more people. We have progressed from batch computing with punch cards, to interactive command line systems, to mouse-based graphical user interfaces, and more recently to mobile computing. WEach of these paradigm shifts has drastically changed the way we use technology for work and life, often in unpredictable and profound ways.

With the latest move to mobile technique allows users to place devices tentional resources on their real-world computing, we now carry devices with on ordinary surfaces, like tables, and activity (see [8] for the full write up). significant computational power and then use them as ad hoc gestural finger At this point, the dual task becomes capabilities on our bodies. However, input canvases. This is achieved with a cognitively taxing as users are con- their small size typically leads to lim- microphone on the underside that al- stantly interrupted by having to move ited interaction space (diminutive lows the device to sense audio signals focus back and forth. In a separate screens, buttons, and jog wheels) and transmitted through the material, like line of work, Daniel Ashbrook of Geor- consequently diminishes their usabil- taps and scratches [4]. These types of gia Institute of Technology measured ity and functionality. This presents a solutions work really well in situations the overhead associated with mobile challenge and an opportunity for de- where the user is situated (in an office, interactions and found that just get- veloping interaction modalities that airport, hotel room), but is impractical ting a phone out of the pocket or hip will open the door for novel uses of when the user is on the go. holster takes about 4 seconds and computing. This mobile scenario is particularly initiating interaction with the device Researchers have been exploring challenging because of the stringent takes another second or so [1]. They small device interaction techniques physical and cognitive constraints of propose the concept of micro-interac- that leverage every available part of the interacting on-the-go. In fact, Antti tions—interactions that take less than device. For example, NanoTouch, de- Oulasvirta and colleagues showed that 4 seconds to initiate and complete, so veloped by Patrick Baudisch and Gerry users could attend to mobile interac- that the user can quickly return to the Chu at Microsoft Research, utilizes tion bursts in chunks of about 4 to 6 task at hand. An example of this type the backside of devices so that the fin- seconds before having to refocus at- of interaction is Whack Gestures [6], gers don’t interfere with the display on created by Carnegie Mellon and Intel the front [2] (see also in this issue “My Labs researchers, where quite simply, New PC is a Mobile Phone,” page 36). you do things like whack the phone In more conceptual work, Ni and Bau- “Micro-interactions in your pocket to silence an incoming disch explore the advent of “disappear- phone call. ing mobile devices” (see [7]). could significantly We believe that such micro-inter- Other researchers have proposed expand the set of actions could significantly expand the that devices should opportunistically set of tasks we could perform on-the- and temporally “steal” capabilities tasks we could go and fundamentally alter the way from the environment, making creative perform on-the-go we view mobile computing. We assert use of existing surfaces already around that while seemingly subtle, augment- us [9]. One example of this type of in- and fundamentally ing users with always-available micro- teraction is Scratch Input, developed alter the way we view interactions could have impact on the by Chris Harrison and Scott Hudson of same magnitude that mobile com- Carnegie Mellon’s HCI Institute. This mobile computing.” puting had on enabling a set of tasks

30 XRDS • summer 2010 • Vol.16 • No.4 Interfaces on the Go

that were never before possible. After Figure 1: To contract a muscle, the brain sends an electrical signal through the ner- all, who would have imagined mobile vous system to motor neurons, which then transmit electrical impulses to adjoin- phonse would make the previously ing muscle fibers, causing them to contract. Electromyography (EMG) senses this onerous task of arranging to meet a muscle activity by measuring the electrical potential between a ground electrode group of friends for a movie a breeze? and a sensor electrode. Who would have imagined when mo- bile data access became prevalent that we’d be able to price shop on-the-fly? Or resolve a bar debate on sports sta- tistics with a quick Wikipedia search? Imagine what we could enable with seamless and even greater access to in- formation and computing power. To realize this vision, we’ve been looking at ways to enable micro-inter- actions. Often, this involves develop- ing novel input modalities that take advantage of the unique properties of the human body. In this article, we de- scribe two such technologies: one that senses electrical muscle activity to in- fer finger gestures, and the other that monitors bio-acoustic transmissions through the body, allowing the skin to be turned into a finger-tap-sensitive interaction surface. We conclude with some of the challenges and lessons learned in our work using physiologi- cal sensing for interaction.

Muscle-Computer Interfaces Removing manipulation of physical transducers does not necessarily pre- clude leveraging the full bandwidth available with finger and hand ges- tures. To date, most efforts at enabling implement-free interaction have fo- cused on speech and computer vision, both of which have made significant strides in recent years, but remain prone to interference from environ- mental noise and require that the user make motions or sounds that can be

XRDS • summer 2010 • Vol.16 • No.4 31 Interfaces on the Go

sensed externally and cannot be easily “With the latest cent when carrying a bag. These results concealed from people around them. demonstrate the feasibility of detect- Advances in muscular sensing and move to mobile ing finger gestures in multiple scenari- processing technologies provide us computing, we os, and even when the hands are other- with the unprecedented opportunity wise occupied with other objects. to interface directly with human mus- now carry devices For more details about this work, cle activity in order to infer body ges- with significant see [10,11,12] tures. To contract a muscle, the brain sends an electrical signal through the computational power Bio-Acoustic Sensing nervous system to motor neurons, and capabilities on To further expand the range of sensing which then transmit electrical impuls- modalities for always-available input es to adjoining muscle fibers, causing our bodies.” systems, we developed Skinput (see them to contract and the body to move. Figure 3), a novel input technique that Electromyography (EMG) senses this allows the skin to be used as a finger muscle activity by measuring the elec- input surface. When a finger taps the trical potential between ground and a skin, several distinct forms of acoustic sensor electrode. energy are produced and transmitted In our work, we focus on a band of achieved for gestures performed on through the body. We chose to focus sensors placed on the upper forearm flat surfaces. In one experiment with on the arm, although the technique that senses finger gestures on surfaces 13 novice users, we attained an aver- could be applied elsewhere. This is an and in free space (see Figures 1 and age of 78 percent accuracy for sensing attractive area to “steal” for input as it 2). We have recently built a small, low- whether each of two fingers is curled, provides considerable surface area for powered wireless prototype EMG unit 84 percent for which of several pres- interaction, including a contiguous that uses dry electrodes and that can sure levels are being exerted on the and flat area for projection. be placed in an armband form factor, surface, 78 percent for which of the five Using our prototype, we’ve con- making it continuously wearable as an fingers have tapped the surface, and 95 ducted several experiments that dem- always-available input device. The sig- percent for which of the five have lifted onstrate high classification accuracies nals from this device are streamed to off the surface. even with a large number of tap loca- a nearby computer, where features are Similarly, in a separate test with 12 tions. This remains true even when the extracted and machine learning used different novice users, we attain 79 per- sensing armband was placed above the to model and classify gestures. Howev- cent classification accuracy for pinch- elbow (where taps are both separated er, this could also be done entirely on a ing the thumb to fingers in free space, in distance and by numerous joints). mobile device. 85 percent when squeezing different For example, for a setup in which we Reasonably high accuracies can be fingers on a coffee mug, and 88 - per cared to distinguish between taps on each of the five fingers, we attain an av- erage accuracy of 88 percent across our Figure 2: Our prototype features two arrays of sensing elements incorporated into 13 novice participants. If we spread the an armband form factor. Each element is a cantilevered piezo film tuned to respond five locations out across the whole arm, to a different, narrow, low-frequency band of the acoustic spectrum. the average accuracy goes up to 95 per- cent. The technique remains fairly ac- curate even when users are walking or jogging. Although classification is not perfect—nor will it likely ever be—we believe the accuracy of our proof-of- concept system clearly demonstrates that real-life interfaces could be devel- oped on top of the technique. While our bio-acoustic input ap- proach is not strictly tethered to a par- ticular output modality, we believe the sensor form factors we explored could be readily coupled with a small digital projector. There are two nice proper- ties of wearing such a projection device on the arm: 1) the arm is a relatively rigid structure—the projector, when attached appropriately, will naturally track with the arm; 2) since we have fine-grained control of the arm,- mak

32 XRDS • summer 2010 • Vol.16 • No.4 ing minute adjustments to align the Figure 3: Augmented with a pico-projector, our sensing armband allows interac- projected image with the arm is trivial tive elements to be rendered on the skin, potentially enabling a new class of mobile (e.g., projected horizontal stripes for computing experiences alignment with the wrist and elbow).

Challenges and Opportunities Using the human body as the interac- tion platform has several obvious ad- vantages. Foremost, it is great that we can assume a consistent, reliable, and always-available surface. We take our bodies everywhere we go (or rather it takes us). Furthermore, we are inti- mately familiar with our bodies, and proprioceptive senses allow us to inter- act even in harsh circumstances (like a moving bus). We can quickly and easily make finger gestures or tap on a part of our body, even when we cannot see it and are on the move. That said, using the signals gener- ated by or transmitted through the body as a means of intentional control comes with various new challenges and opportunities for innovation. From a technical perspective, build- ing models of these signals that work across multiple users and multiple ses- sions with minimal calibration is often

“Who would have imagined when mobile data access became prevalent that we’d be able to price shop on- the-fly? Or resolve a bar debate on sports statistics with a quick Wikipedia search? Imagine what we could enable with seamless and even greater access to information and computing power.”

XRDS • summer 2010 • Vol.16 • No.4 33 Interfaces on the Go

challenging. Most of our current work for his work on physiological computing and healthcare, ACRONYMS including a 2007 MIT TR35 Young Innovators award, SciFi is calibrated and trained each time the Channel’s Young Visionaries at TED 2009, and named to French version of Forbes’ Revolutionaries list in 2009. He will chair the CHI AZERTY user dons the device, and while these 2011 Conference, which will be held in Vancouver, BC. the keyboard layout individual models work surprisingly Dan Morris is a researcher in the Computational User known in English as well across different body types, we Experiences group in Microsoft Research. His research interests include computer support for musical QWERTY recognize that this overhead of train- composition, using physiological signals for input, and ing is not acceptable for real world improving within-visit information accessibility for hospital patients. Dan received his PhD in Computer BCI Brain-Computer use. Furthermore, regardless of uni- Science from Stanford University in 2006. Interface: tech that versality of the models, processing the T. Scott Saponas is a PhD candidate in the Computer reads your mind! often-noisy signals coming from these Science and Engineering department at the University of Washington. His research interests include Human- sensors is not trivial and will likely Computer Interaction (HCI), Ubiquitous Computing Bio-Acoustics Sounds never yield perfect results. This is true (UbiComp), and Physiological Computing. Scott received his B.S. in Computer Science from the Georgia Institute of produced by the body, because of the complexity of the noise Technology in 2004. such as touching one’s patterns as users move through differ- arm,which computers ent environments, perform different References can distinguish tasks, and as the physiological signals 1. Ashbrook, D. L., Clawson, J. R., Lyons, K., Starner, T. E., and Patel, N. Quickdraw: the impact of mobility changes throughout the course of their and on-body placement on device access time. In EEG electroencephalo- normal activities. Hence, interaction Proceedings of CHI 2008, 219-222. graphy: a BCI that techniques must be carefully designed 2. Baudisch, P. and Chu, G. Back-of-device interaction allows creating very small touch devices. In measures electrical to tolerate or even take advantage of Proceedings of CHI 2009, 1923-1932. signals from sensors imperfect interaction input. 3. Benko, H., Saponas, T. S., Morris, D., and Tan, D. 2009. on your scalp On the interaction design front, Enhancing input on and above the interactive surface with muscle sensing. In Proceedings of Interactive there are many problems that must be Tabletops and Surfaces. GUI Graphical user addressed. For example, the system 4. Harrison, C., and Hudson, S.E. Scratch Input: Creating interface must provide enough affordances that large, inexpensive, unpowered and mobile finger input surfaces. In Proceedings of UIST 2008, 205-208. the user can learn the new system. This 5. Harrison, C., Tan, D., and Morris, D. Skinput: HCI Human-computer is not specific to physiological sensing, Appropriating the body as an input surface. To appear interaction though the level of indirect interpreta- in Proceedings of CHI 2010. 6. Hudson, S. E., Harrison, C., Harrson, B. L., LaMarca, tion of signals can sometimes make A. 2010. Whack gestures: Inexact and inattentive QWERTU Eastern European end-user debugging difficult, especial- interaction with mobile devices. In Proceedings of the 4th International Conference on Tangible, Embedded version of QWERTY ly when the system does not act as it is and Embodied Interaction (Cambridge, MA, January expected to. The interface must also be 25 - 27, 2010). TEI ‘10. ACM, New York, NY. QWERTZ Central European designed to handle the “midas touch” 7. Ni, T. and Baudisch, P. 2009. Disappearing mobile devices. In Proceedings of UIST 2009, 101-110. version of QWERTY problem, in which interaction is un- 8. Oulasvirta, A., Tamminen, S., Roto, V., and Kuorelahti, intentionally triggered when the user J. Interaction in 4-second bursts: The fragmented SSVEP Steady state visually performs everyday tasks like turning a nature of attentional resources in mobile HCI. In Proceedings of CHI 2005, 919-928. evoked potentials: doorknob. We have purposely designed 9. Pierce, J.S., and Mahaney, H.E. 2004. Opportunistic predictable brain our gesture sets in order to minimize annexing for handheld devices: Opportunities and responses to visuals this, but we imagine there are more challenges. Human-Computer Interface Consortium. that an EEG can graceful solutions. 10. Saponas, T. S., Tan, D. S., Morris, D., and Balakrishnan, R. Demonstrating the feasibility of using forearm detect In fact, with many interaction mo- electromyography for muscle-computer interfaces. In dalities, our first instinct is often to em- Proceedings of CHI 2008, 515-524. Pens 11. Saponas, T. S., Tan, D. S., Morris, D., Balakrishnan, Super Pens ulate existing modalities (e.g., mouse R., Turner, J., and Landay, J. A. Enabling always- augmented with and keyboard) and use it to control ex- available input with muscle-computer interfaces. In special hardware, isting interfaces. However, the special Proceedings of UIST 2009, 167-176. 12. Saponas, T. S., Tan, D. S., Morris, D., Balakrishnan, R., such as cameras affordances found in the mobile sce- Turner, J., and Landay, J. A. Making muscle-computer nario bring with it enough deviations interfaces more practical. To appear in Proceedings of CHI 2010. TUI Tangible user from our traditional assumptions that interface we must be diligent in designing for it. We should also emphasize the impor- WIMP Windows, icons, tance of designing these systems so menus, pointers—the that they operate seamlessly with oth- typical way we er modalities and devices that the user interact with a GUI carries with them.

WYSIMOLWYG What you Biographies see is more or less Desney Tan is a senior researcher at Microsoft Research, where he manages the Computational User Experiences what you get group in Redmond, Washington and the Human-Computer Interaction group in Beijing, China. He has won awards

34 XRDS • summer 2010 • Vol.16 • No.4 Announcing ACM’s Newly Improved Career & Job Center!

Are you looking for your next IT job? Do you need Career Advice?

Visit ACM’s newest career resources at http://www.acm.org/careercenter

◆ ◆ ◆ ◆ ◆

The ACM Career & Job Center offers ACM members a host of career-enhancing benefits:

➜ A highly targeted focus on job opportunities in the computing industry

➜ Access to hundreds of corporate job postings

➜ Resume posting keeping you connected to the employment market while letting you maintain full control over your confidential information

➜ An advanced Job Alert system that notifies you of new opportunities matching your criteria

➜ Career coaching and guidance from trained experts dedicated to your success

➜ A content library of the best career articles complied from hundreds of sources, and much more!

The ACM Career & Job Center is the perfect place to begin searching for your next employment opportunity! Visit today at http://www.acm.org/careercenter My New PC is a Mobile Phone Techniques and devices are being developed to better suit what we think of as the new smallness. By Patrick Baudisch and Christian Holz DOI: 10.1145/1764848.1764857

he most popular computational device in the world is neither a desktop computer nor a netbook nor a hundred-dollar laptop—it is the mobile phone. More than 4 billion mobile phones are in use worldwide today. Unlike any other computational device on the market, mobile phones have a very large user base, which Tincludes people across the world, from developed countries to developing nations, and among both urban and rural populations. It is equally popular between people of different ages, both young and adult. The mobile phone’s popularity creates a vast range of new use cases and scenarios that need to be designed for.

On the one hand, mobile devices also analyze and manipulate it. This revolves around the very basics of the allow PC users to undertake a broader varies from editing simple text docu- interaction: input and output. The and broader range of activities on the ments to complex processing of data. eventual goal is to create interaction road, disconnected from the wired That these applications are still models that will evade the constrain- world. Most modern devices allow in- missing on today’s mobile devices is ing human factors discussed above. teractive web browsing, as well as the the result of the limited size of these To overcome the limitations of dis- viewing and editing of documents, devices and the result of human fac- playing output to the users on tiny spreadsheets, images, and so on. tors. Because human eyesight is lim- screens with limited screen size and On the other hand, mobile devices ited, a screen of a certain size can resolution, much research has taken are the most likely devices to keep communicate only a certain amount place. For example, researchers have people connected to the digital world. of information. Because fingers have created systems that provide display With widespread availability and low a certain size, a mobile keyboard or space on demand using projection, production costs, mobile phones are touch screen can offer only so many such as the Sixth Sense system. To keep on their way to becoming the mass controls or buttons. the device truly mobile, this projection computation platform of the future, a To eventually enable these complex mechanism requires a flat surface at all task that neither desktop computers applications, a lot of current research times and at all places. Other research- nor netbooks have succeeded in doing ers have instead built on zooming, pan- so far. ning, and scaling techniques. Ka Ping Yee’s Peephole Display allows users to The Challenge “Precise input on navigate a virtual world by moving a The role of mobile devices as desktop device in the physical world. The un- replacements and as terminals to the small devices opens derlying concept allows users to lever- digital world requires new categories of up a large space of age landmarks in the physical world mobile applications, ones that will al- to return to the associated locations low users to not only view the data, but new device designs.” in the virtual world. Summary Thumb-

36 XRDS • summer 2010 • Vol.16 • No.4 nails are miniature views of a web page Figure 1: Gesture input allows for input on the tiniest of mobile devices (a) on the that keep font size readable by cropping fi nger, providing tactile feedback, (b) in the earlobe, providing auditory feedback, text rather than scaling it. Off-screen or (c) on the wrist, providing visual feedback. The user is entering a “2” by “scan- visualization techniques, such as Halo ning” two fi ngers (see Ni and Baudish’s Disappearing Mobile Devices for more [5]). and Wedge virtually extend the screen by leveraging off-screen space. a In this article, however, we focus on the other aspect of the challenge: input- related issues.

geSTUre inPUT Gesture input bypasses the lack of in- put space by using the device as a whole. Users either move the device, which is tracked by an accelerometer present in the device (for example, Rekimoto’s Tiltable Interfaces), or the users move their hands in front of the device, as in the Gesture Pendant by Thad Starner and colleagues. By performing ges- tures right on the surface of the device, gesture input can be brought to the tiniest form factors (Figure 1). Scratch b Input by Harrison and Hudson [3] is basically a gesture interface—it allows users to enter commands by scratching on arbitrary surfaces, which is sensed using a microphone.

PoinT, ToUCh, anD faT fingerS On the fl ip side, gesture interaction is disjointed from the output space. Many complex desktop applications allow us- ers to manipulate data, but only after having selected them, either through a pointing device, like a mouse, or through a keyboard. If we want to bring such applications to mobile devices, we need to be able to select these objects c on the screen. Miniature joysticks, still common on mobile phones, let the user select objects but with very limited pointing c accuracy and abilities. Touch screens on the other hand offer much better performance. In addition to the ease of use, they are well-suited for mobile de- vices since they can integrate the input medium and the output screen into the a same physical space, thus allowing for physical compactness of the device. The opposite is true, however, be- cause of the so-called fat fi nger prob- lem. The softness of the user’s skin causes the touch position to be sensed anywhere within the contact area be- tween the user’s fi ngertip and the de- vice. Not only that, the relatively larger fi nger compared to that of the screen

XRDS • summer 2010 • Vol.16 • No.4 37 b c My New PC is a Mobile Phone

Figure 2: (a) Small targets are occluded by a user’s finger. (b) Shift reveals occluded screen content in a callout displayed above the finger. This allows users to fine tune with take-off selection. (c) By adjusting the relative callout location, Shift handles targets anywhere on the screen. a b c

causes the finger to occlude the target. We addressed this and other short- pointer. The Shift technique also does This prevents the target from providing comings with the Shift technique [6], not work well on very small devices— visual feedback, preventing users from shown in Figures 2(a) and 2(b). While the smaller the device, the larger the compensating for the randomness. Offset Cursor requires users to aim be- finger in proportion and the harder it As a result of the fat finger prob- low the target, Shift lets them aim at is to find space to place the callout. lem, today’s touch screen devices are the target itself—a benefit in itself, as Researchers started exploring other therefore not smaller than their joy- it reestablishes the direct touch affor- options to use the user’s finger as an in- stick-based predecessors, but actually dance of touch screens. It then reveals put device. Sidesight by Butler et al. [2] larger. the occluded screen content in a call- allows users to move their finger next Several researchers have proposed out displayed above the finger. This al- to the device. As a wrist-worn device it techniques and devices that resolve the lows Shift to handle targets anywhere effectively creates a virtual touch pad fat finger problem by physically sepa- on the screen by adjusting the relative on the user’s arm. Sidesight thereby of- rating the input and the output space. callout location (Figure 2 (c)). fers the functionality of an offset cur- The first technique based on this idea However, the Shift Cursor technique sor that extends beyond the edge of the was the Offset Cursor designed by Pot- has its limitations as well. The callout screen. ter and Shneiderman and published mechanism limits the dragging of the Abracadabra by Harrison and Hud- in 1988. In their design, when a user son follows a similar approach. It mag- touches the screen, a pointer appears nifies the input space, allowing users to half an inch above the contact point. point in the space around the device. The user would move the pointer over “The role of mobile On the flip side, all these techniques the target and select it by lifting the affect the registration of input and out- finger off. Offset Cursor resolved the devices as desktop put space, thereby limiting the users’ occlusion issue and was the first tech- replacements ability to simply “touch a target.” nique to allow for accurate input on touch screen devices that had previ- requires new Back-of-Device Interaction ously been considered inherently inac- categories of mobile To reestablish this registration, we curate. proposed back-of-device interaction. However, Offset Cursor has a num- applications, ones Figure 3 shows our second prototype ber of limitations. For example, it that will allow users called Nanotouch [1]. does not allow selecting the contents The main idea was to maintain the at the very bottom of the screen. This to not only view the metaphor of direct touch but by touch- becomes a particularly big issue when data, but also analyze ing the back of the device so that fin- applied to the tiny screens of today’s gers never occlude the target. To allow mobile devices. and manipulate it.” users to accurately point and touch

38 XRDS • summer 2010 • Vol.16 • No.4 the target, Nanotouch always shows otouch with a simulated screen with a some representation of the finger on “Mobile phones diagonal size of only 8mm. the front-side screen. To help the us- are on their way Precise input on small devices opens ers learn the new input paradigm, we up a large space of new device designs, show an image of an actual finger to to becoming the including the ones shown in Figure 4. first-time users, as shown in Figure 3. mass computation All device concepts follow the same ba- For any real application, however, we sic design: the front-side is for the dis- remove the finger and replace it with a platform of the play and the backside is for the touch small dot—basically a pointer image— future.” input. The edges hold a set of buttons which minimizes occlusion and allows with specific functions. The notion for precise targeting. of “back of the device” leaves us with The key idea behind the design of some flexibility, such as in the case of any front-side touch or a back-of-device ing a mirror. When shaving or applying the watch, where the back of wristband touch must be that the human interac- makeup, users perceive the “virtual serves as the back of the device. tion map to the same region from the person” in the mirror as facing them, Still, the translation from front-side user’s perspective. Making the device yet interact backwards. interaction to back-of-device leaves appear transparent in the back-of-de- Interaction with Nanotouch also space for interpretation. In an infor- vice design reinforced this correspon- turned out to be highly precise and in mal survey, we asked people to write dence nicely. a user study, participants were able to characters on the back of the device. The new input design worked sur- acquire a 2.8mm target with 98 per- About 80 percent of participants wrote prisingly well in our experiments. One cent accuracy. More importantly, back- left-to-right, which is consistent with of the reasons could be that the users of-device interaction works practically the front-side experience of eyesight, are already familiar with this notion independent of the device size. In our shoulder motion, and elbow motion. from activities that they perform us- user study, participants operated Nan- The remaining 20 percent, however,

Figure 3: Users operate Nanotouch using pointing input on the back of the device.

XRDS • summer 2010 • Vol.16 • No.4 39 My New PC is a Mobile Phone

Figure 4: Four of the back-of-device designs we envision: (a) a clip-on device with 2.4-inch screen, (b) a watch with 1.2-inch screen, (c) a ring with a screen diagonal of less than half an inch, and (d) a pendant. a b d

c

wrote right-to-left, which is consistent sociated with the fat fi nger problem close to as inaccurate as it is commonly with the familiar motion of the wrist. turned out to be the result of differ- assumed to be. Instead, the inaccuracy Back-of-device touch input can ences between users and variations we observe with today’s touch devices enable pointing input on very small in fi nger postures. During the experi- appears to be the result of overlaying screens. However, this entire series of ment, we forced users to main-tain a many different contact point distri- techniques and devices goes back to constant fi nger posture, such as keep- butions, each of which is actually very the fat fi nger problem, i.e., the under- ing a 45-degree tilt between the fi nger small. standing that the touch interface is and the pad. We then varied the angle These observations suggest that the inaccurate. Given that so much work of the fi nger. As a result, the contact inaccuracy of touch devices can be re- has been done on this model, we felt point distributions moved as shown in solved if a device can identify users and it was time to go back and verify our Figure 5(a). Each of the fi ve white ovals determine the angle between the fi nger underlying assumptions. Surprisingly, in the fi gure is the result of a different and the pad. We created a series of de- we found that the fat fi nger problem is fi nger angle. We found similar shifts in vices that ex-ploit this insight in order largely a myth. the offsets across users, but the size of to achieve very high touch accuracy. the distributions remained small. DiSProving ‘faT finger’ anD This is a surprising observation. aCCUraTe ToUCh for reDeeMing fronT-SiDe ToUCh The smallness of each of the white MoBile DeviCeS We conducted some experiments to ovals suggests that touch is not even Figure 6 shows a setup that imple- verify the existence of the fat fi nger ments two of these prototype devices. problem. Subjects repeatedly acquired The cameras surrounding the fi nger crosshairs with their index fi nger. For belong to an Optitrack optical tracking every trial we logged the resulting con- system that determine fi nger angles by tact point, as re-ported by a capacitive observing tiny markers glued to the us- touch pad. We expected the contact “We need to let go of er’s fi ngernail. The resulting setup al- points to form a single large distribu- the notion that the lows users to acquire targets of 4.3mm tion. di-ameter with 95 percent accuracy, a Surprisingly, this was not the case. mobile devices are 300 percent improvement over tradi- Contact point distributions turned out auxiliary devices that tional touch screens. to be much smaller than ex-pected, However, this setup is hardly mo- about only a third of the expected size. we use while on the bile. We therefore implemented a Instead, the error generally as- road.” second method called RidgePad [4],

40 XRDS • summer 2010 • Vol.16 • No.4 also shown in Figure 6. This method is based on the fi ngerprint scanner in Figure 5: (a) A touch input study found that contact points formed much more the center of the photo. Un-like a tra- compact distributions than expected. (b) The RidgePad device exploits the eff ect ditional touchpad, the device obtains that a fi ngerprint identifi es not only a user, but also the angle between fi nger and not only the outline of the contact area device. between fi nger and device, but also the fi ngerprint within this area. By com- paring the fi ngerprint’s ridge pattern against samples in the database, the a device fi rst determines the user and looks up his or her personal calibra- b tion data. The device now determines where the observed part of the fi nger- print is located on the user’s fi nger, which allows RidgePad to reconstruct the fi nger’s posture during the touch. By taking this angle into the account, RidgePad is 1.8 times more accurate than traditional touch pads.

MoBile PhoneS aS PCS Mobile devices are on the verge of be- coming the computational platform of the world. In order to suc-ceed, a Figure 6: This experimental setup tracks fi nger angles using an optical tracker. It wide range of challenges needs to be also implements the RidgePad prototype, which extracts user ID and fi nger angles tackled. We have discussed only on from the user’s fi ngerprint. one particular facet: bringing accu- rate pointing and manipulation to tiny touch screens. This forms the basis for direct manipulation and thus has the potential to opens up mobile devices as a platform for more complex and more interactive applications. But we have only scratched the sur- face. In order to tackle the new chal- lenges, we need to make a major con- ceptual shift. We need to let go of the notion that the mobile devices are auxiliary devices that we use while on the road. Instead, we need to adopt a model in which the mobile devices are the main computational devices, if not the only computational device.

Biographies Patrick Baudisch is a professor in Computer Science at Hasso Plattner Institute in Berlin/Potsdam and chair of the Human-Computer Interaction Lab. His research focuses on the miniaturization of mobile devices and touch input. Previously, he worked as a research scientist in the Adaptive Systems and Interaction Research Group Acknowledgements ”touch” interaction around small devices. Proc. UIST ‘08. 201-204. at Microsoft Research and at Xerox PARC and served as an The authors thank everyone in their lab group at Hasso affi liate professor in computer science at the University Plattner Institute and former colleagues at Microsoft 3. Harrison, C. and Hudson, S. (2008). Scratch input: of Washington. He holds a PhD in Computer Science from Research, in particular Ken Hinckley and Ed Cutrell. They creating large, inexpensive unpowered and mobile Darmstadt University of Technology, Germany. also thank Dan Vogel who worked on Shift, and Gerry fi nger input surfaces. Proc. UIST ‘08. 205-208. Chu who worked on NanoTouch. Finally, they thank the Christian Holz is a PhD student in Human-Computer 4. Holz, C. and Baudisch, P. 2010. The Generalized collaborators on back-of-device interaction, in particular, Interaction at Hasso Plattner Institute in Potsdam, Perceived Input Point Model and How to Double Touch Daniel Wigdor and Cliff Forlines. Germany. Previously, he worked as a research scholar Accuracy by Extracting Fingerprints. To appear in at Columbia University. His research focuses on Proceedings of CHI’10. 10 pages. understanding and modeling touch input on very small mobile devices. References 5. Ni, T. and Baudisch, P. Disappearing Mobile Devices. In 1. Baudisch, P. and Chu, G. Back-of-device interaction Proc. UIST’09. 101-110. allows creating very small touch devices. In Proc. 6. Vogel, D. and Baudisch, P. (2007). Shift: A Technique CHI’09. 1923-1932. for Operating Pen-Based Interfaces Using Touch. 2. Butler, A., Izadi, S., and Hodges, S. SideSight: multi- Proc. CHI‘07. 657-666.

XRDS • summer 2010 • Vol.16 • No.4 41 From Brains to Bytes Brain-computer interfaces have the potential to change the way we use devices, and there are at least four methods for implementation. By Evan Peck, Krysta Chauncey, Audrey Girouard, Rebecca Gulotta, Francine Lalooses, Erin Treacy Solovey, Doug Weaver, Robert Jacob DOI: 10.1145/1764848.17664858

cience fiction has long been fascinated by brain-computer interfaces (BCI)­—the use of sensors to identify brain states. From Andre Maurois’ 1938 story “The Thought- Reading Machine,” in which a professor stumbles on a machine that reads people’s thoughts, to the recent blockbuster Avatar, where humans control surrogate bodies Swith their minds, the public is captivated by the interaction between the human brain and the computers created by those brains.

Although most people are likely to ing the type and number of commands to understand the user’s needs before conjure images of Neo’s frightening the computer can recognize. An appli- the user can articulate them. “head port” from The Matrix before they cation that uses audio increases the But this is all far in the future. On dream of a university student wearing bandwidth from the computer to the the wide continuum between analyz- an elastic cap studded with electrodes, user, by adding to the type of informa- ing electroencephalo-graphs to Avatar the media has closely followed less tion the computer can output. Seen in mind-machines, where are we now? sinister, if also less all-powerful, BCI this context, brain-computer interfac- And perhaps more importantly, where research. In the past year, University of es present an opportunity to expand are we going? Wisconsin-Madison’s Brain-Twitter in- the user-to-computer bandwidth in a In this article, we will discuss sev- terface received Time Magazine’s honor unique and powerful way. Instead of eral directions for research into brain- as the no. 9 invention of the year. Fur- identifying explicit actions, we can de- computer interaction, and the relative thermore, as brain-imaging technolo- tect intent. Instead of evaluating action merits using these brain measure- gy has become more portable and less artifacts, we can recognize purpose. ments to give the user direct or pas- expensive, the human-computer inter- Even more interesting, we may be able sive control of computer interfaces. We action (HCI) community has begun to will also introduce projects across the bring science fiction closer to reality. world that offer a glimpse into the fu- ture of BCIs. Mind Matters “As brain-imaging Imagine the following scenario: In the larger field of human-computer interaction, we are often concerned technology has It’s 9:30 p.m., and you’re driving in with the bandwidth of interaction be- become more heavy traffic through Boston, unsure tween a user and the computer. How of where you’re going. Your phone is can we give the computer more infor- portable and less alerting you of text messages that your mation, and more relevant informa- expensive, the HCI sister is sending every 30 seconds. Your tion? How can the computer give us GPS is commanding you to turn in half more information without overloading community has a mile, then a tenth of a mile, but still, our sensory systems? begun to bring you cannot tell which of the six exits off Using a mouse in addition to a key- the rotary to take. To make things worse, board increases the bandwidth from science fiction closer the radio commercials are blaring, the user to the computer by augment- to reality.” but you are focused on the road, and

42 XRDS • summer 2010 • Vol.16 • No.4 Art by Paul Wearing

XRDS • summer 2010 • Vol.16 • No.4 43 From Brains to Bytes

uncomfortable taking your hand off the wheel to reach for the volume. Figure 1: Unlike some BCI systems that require inserting devices directly into the brain, EEG electrode caps are worn externally. Unlike the head-ports in the fantas- tical world of The Matrix, this scenario is close to our current reality. Also un- like a fantastical movie world, this is the world in which any brain-computer interface will have to function. But how could a brain-computer interface deal with this situation without adding to the already full spectrum of sensory input? Researchers are taking steps to- ward answering this question and de- signing BCIs to meet our needs. Several approaches to BCIs are being pursued using current brain- sensing technology. These systems detect biological changes occurring naturally during the operator’s activ- ity. Interface designers can use this information to deduce the operator’s state and translate it into commands that adjust the computer accordingly. Changes to the interface can be the re- sult of a direct—voluntary—input, or a passive measure. technologies in the car (much like how To survey current direct control re- we learn to touch type). Without taking search, we further divide the topic into Direct Control Interfaces your eyes off the road, you decide to two brain-imaging techniques: invasive Brain interfaces that allow direct con- silence your phone, and perform mental and non-invasive. Although brain sens- trol often replace a user’s normal mo- arithmetic, which the brain-computer ing is not limited to direct control inter- tor function (generally used to move interface recognizes as the command faces, we are not aware of any passive mouse cursors or type on a keyboard), for muting the phone. You imagine systems that use invasive techniques. and are currently the dominant strain swinging your right arm up and down, of research in brain-computer interac- and the device also recognizes this as a Invasive tion. Direct control involves a struc- command, turning the radio down. You Invasive BCIs involve implanting mi- tured mental activity that results in an imagine yourself moving your left pinky, croelectrodes into the grey matter of explicit command to the computer. To and again, the device recognizes your the brain during neurosurgery in an move your mouse to the right, you might brain state, redrawing the GPS map in effort to capture brain activity more imagine moving your hand to the right. more detail. Through the entire process, accurately. While significant, suc- These direct-control interfaces rely your eyes never leave the road, your cessful work is being performed on on the fact that the brain activity oc- hands never leave the steering wheel, and invasive techniques at research fa- curring when you move your hand to you never say a word. cilities such at Brown University and the right is very similar to the activity University Wisconsin­-Madison, there that occurs when you imagine moving If this sort of control seems con- are many risks, difficulties and limi- your hand to the right. This consisten- trived, it is because most direct-con- tations involved. Aside from the diffi- cy can be used to pair mental “move- trol interfaces are currently geared culty of inserting sensors directly into ments” with commands: when partici- toward disabled users, such as para- the brain, there is the risk of scar tissue pants imagine waving their arms up lyzed patients, or people with severe formation, infection to the brain, and and down, for example, the volume on physical disabilities. In disabled us- the unknown long-term stability of their phone might mute, or the zoom ers, familiar, physical mental com- implanted micro-electrodes. Depend- level on their screen might change. mands can be repurposed to perform ing on the task and the participant, ac- Using this mechanism, we can other tasks, as these motor skills are complishing a task using brain states imagine a world for our car scenario not available. However, if we look fur- can require long training, and some in which direct control interfaces are ther into the future, users may not users may never reach the desired level commonly available to everyone. need to perform mental gymnastics, of reliability. instead using the thought processes Despite this, any improvement in You have trained yourself to produce that occur in the brain anyway to con- communication is worthwhile and re- specific brain activity to control different trol external devices. warding for paralyzed users, who may

44 XRDS • summer 2010 • Vol.16 • No.4 not have voluntary control over any “One recent example which include visually evoked poten- muscles at all. Invasive BCIs can pro- tials (SSVEP), P300-based BCI, motor vide remarkable new opportunities of a BCI that uses imagery, event-related synchronization for disabled users; although currently EEG is a wheelchair and slow cortical potentials. A great ex- such systems are limited to fairly ru- ample of work in this direction can be dimentary communication such as that can be seen by reading Wolpaw et al.’s 2002 choosing pictures, letters, or cursor controlled through article in Clinical Neurophysiology [3]. directions, some potential advantages The P300-based BCI allows selection for future applications include repair- brain activity.” by taking advantage of brain activity, ing damaged sight and providing new called a P300, that occurs when your functionality to those with paralysis. intended target is highlighted. This has been used successfully to create a Non-invasive spelling application. Non-invasive direct control technolo- One recent example of a BCI that gies, unlike their invasive counterparts, uses EEG is a wheelchair that can be use external systems, such as electro- of electrodes used and the techniques controlled through brain activity, cre- encephalography (EEG) or functional to process the data captured. ated by Brice Rebsamen et al. The re- near-infrared spectroscopy (fNIRS), to One advantage of this system is that searchers created a list of paths to lo- measure brain activity. See Figure 1 the data recorded has a high temporal cations in a small apartment and then for an example. Non-invasive imaging resolution. The system can detect brief presented those target locations to us- technologies are commonly used in a changes in brain activity, in the milli- ers. To select a target, the users were number of fields, including medicine second realm. Additionally, it is possi- instructed to focus on that target when and psychology, and provide valuable ble to buy EEG systems that are small, it was presented to them. After several insight into brain activity without re- lightweight, and portable. However, a minutes of training with a participant, quiring surgery or implantation. As number of limitations affect the util- the system could detect the desired lo- a result, they are an attractive option ity of the data collected from EEG sys- cation with almost perfect accuracy. for researchers who want portable and tems. For example, the EEG system is Clearly, there are limitations to this flexible systems for measuring and in- sensitive to muscle contraction, limit- system, such as predetermining a list terpreting this data. ing the user’s physical movements dur- of routes and targets, which may not be EEG measures brain activity using ing cognitive activity measurements. possible in large or complex environ- electrodes placed on the scalp that de- Additionally, these systems have low ments. However, this work is an exam- tect electrical potential caused by neu- spatial resolution. It can be difficult or ple of the possibilities for EEG systems rons firing, providing researchers with impossible to determine which precise to be developed and incorporated into information about activation in numer- region of the brain is being activated. more sophisticated technologies. ous regions of the brain. The complexity There are many successful direct Functional near-infrared spectros- of these systems varies with the number control paradigms using EEG signal, copy (fNIRS) is a vastly different tech- nology than EEG in that it measures blood flow changes instead of elec- Figure 2: An fNIRS sensor with five light sources and one detector. trical activity. It uses optical wires to emit near-infrared light, which is then refracted from the tissue of the head, including the brain (see Figure 2 for an example of the device). Sensors in the system detect changes in oxygenated and deoxygenated blood in that region [1]. This technology is marked by a high degree of spatial resolution but is less temporally sensitive to changes in brain activity than EEG. The brain data recorded by fNIRS is less susceptible to movement artifacts and can be used in combination with computer usage. fNIRS has another no- table advantage: it has a shorter setup time than EEG, which makes it a more practical option for use in research, government work, and commercial ap- plications. Additionally, the part of the fNIRS system placed on the scalp or

XRDS • summer 2010 • Vol.16 • No.4 45 From Brains to Bytes

forehead is typically small and there- “Brain-computer ­invasive measuring techniques such fore less bothersome to users than oth- as EEG and fNIRS. fNIRS, as previ- er brain measurement technologies. interfaces present ously mentioned, requires little in the Ruldolph L. Mappus IV and his an opportunity way of set up and imposes relatively colleagues at the Georgia Institute of few physical or behavioral restraints Technology have used fNIRS to devel- to expand the on the user. Because many passive op a technology for drawing letters us- user-to-computer BCIs aim to observe brain signals that ing data collected from brain activity. can be used in a relatively ordinary The subjects were instructed to trace bandwidth... Instead computer task environment, these the letters “C” and “J” by perform- of identifying qualities make fNIRS particularly at- ing different mental tasks to control tractive as a measuring tool. the drawing direction. The research- explicit actions, we How would a future of passive BCIs ers noted some difficulties with their can detect intent. impact our previous car example? design, including the slow fNIRS re- Imagine now that you’re wearing the sponse time for changes in mental ac- Instead of evaluating headband shown in Figure 3: tivity. However, the researchers indi- action artifacts, cated that, in the future, they believe You drive, think, and behave normally, that this work can be expanded to pro- we can recognize as you would before the BCI was vide users with a broad range of tools purpose.” introduced. Brain-sensing devices to draw using fNIRS technologies. determine that you are mentally overloaded, juggling the phone, GPS, Passive BCIs radio, and driving simultaneously. As The previous examples have helped il- a result, the system gently simplifies lustrate direct control BCIs, which use the map on your GPS. You may not brain activity as the primary input de- have a clear understanding of the vice, but which often require consider- University has turned to passive BCIs, neighborhood, but you won’t miss this able user training to generate specific interfaces that detect brain activity turn. The system subtly turns down the brain states. However, these technolo- that occurs naturally during task per- interior dashboard lights to prevent gies have a reduced relevance in the formance [2]. Passive BCIs focus on glare on the windshield. Finally, since ordinary computer environment. A the brain as a complementary source the sender of the text messages had healthy student has little need to hook of information, an additional input previously been designated as low- herself up to an EEG in order to move used in conjunction with convention- priority, the system silences the phone, a cursor. It’s easier, faster, and less er- al computer inputs such as the mouse waiting for a better time to let you know ror-prone to simply use a mouse. With or keyboard. about the messages. this in mind, our research at Tufts Generally, passive BCIs use non- The principal advantage of passive BCIs is that they operate using the Figure 3: fNIRS sensors placed on the forehead can be non-intrusive when secured brain activity that occurs naturally by a simple headband. during task performance, and so they do not add to the user’s task load (in this example, the driver does not have to think about waving her arms or any other extraneous mental process). Cur- rently, we are working towards a simi- lar class of BCIs to create adaptive en- vironments. In a recent experiment designed to lead to adaptive interfaces, partici- pants played Pac-Man at two different levels of difficulty, while their brain activity was measured using fNIRS. Statistical analysis of the fNIRS mea- surements allowed the participants to be classified both by their playing state (playing versus rest) and difficulty level (easy versus hard) with high accuracy. Using this data, the interface could be subtly adapted to the task at hand. If the user is resting, play quiet, soothing 46 XRDS • summer 2010 • Vol.16 • No.4 Figure 4: Stockbrokers might use BCIs to project appropriate market visualization based on how much distraction they can handle at the moment. The image on the left is what a broker might see when engaged in low amounts of work, while the more simplified version might be more appropriate when the mental workload is greater.

music. If the user is playing the game, cannot afford to be distracted by com- knowledge of cognitive neuroscience to the develop- ment of brain-computer interfaces, particularly those speed up the pace and volume of music plex stock information, the display can which use electrophysiological or optical brain data as an to make the experience more intense simply lower the level of detail. In this auxiliary source of information to improve the user’s task performance or experience. and enjoyable. Other research at Tufts way, the broker will still recognize ma- Audrey Girouard is a PhD candidate in Computer Science has examined fNIRS use in an ordi- jor changes in the data without getting at Tufts University, specializing in HCI. She is currently nary computer environment, as well bogged down in the details. fascinated by passive brain computer interfaces, studying the use of brain imagery to enhance HCI for all. She is the as determined differences in semantic We can imagine a class of interfaces laureate of the PostGratudate Scholarship from NSERC. and syntactic workload. that work to benefit the general popu- Rebecca Gulotta is a senior at Tufts University and a research assistant in the Tufts Human-Computer Passive BCIs also lead to a completely lation, systems that dynamically filter Interaction Lab. She studies Human Factors Engineering different paradigm for interaction with streams of information (Twitter, RSS, and Computer Science. the user. It is no longer acceptable to use email) to accommodate the workload Francine Lalooses is a computer science doctoral candidate at Tufts University, where she is studying bold, explicit reactions to brain states of the user, systems that dynamically human computer interaction and specifically brain- that we make in direct control inter- adjust teaching methods to best suit a computer interaction. She currently works at The MITRE faces. Instead, we find ourselves drawn child’s learning style. Corporation while attending Tufts. Erin Treacy Solovey is a computer science doctoral toward gentle, more implicit changes to candidate at Tufts University, where she is interested in the interface. Fading, overlaying infor- Looking Ahead next-generation interaction techniques. She is currently investigating the use of brain sensors to detect signals mation, or changing an application’s BCIs currently in development have the that users naturally give off in order to augment standard screen real estate are options that, if potential to open worlds of communi- input devices. done slowly enough, may affect the us- cation and mobility to disabled users; Doug Weaver is a master’s student at Tufts University. Currently, he is participating in research developing er’s focus in a positive way. further up the pipeline, BCIs have the adaptive brain-computer interfaces. While this type of research is still potential to adjust our constantly-busy Robert Jacob is a professor of computer science at Tufts University, where his research interests are new in its infancy, there are countless ex- environments to provide us with the interaction media and techniques and user interface amples of possible adaptive interfaces best possible chance of completing software. He was also a visiting professor at the Universite Paris-Sud and at the MIT Media Laboratory, in the Tangible that could be used with fNIRS mea- our tasks. Whether this is successful Media Group. He is a member of the editorial board of surements. For example, at any given navigation to a new place, remotely Human-Computer Interaction and the ACM Transactions on Computer-Human Interaction. He was elected to the moment during the day, a stock broker commanding a robot, or buying all the ACM CHI Academy in 2007. could be overwhelmed with work or ex- items on a grocery list, BCIs hold the periencing a lull in activity; either way, promise of performing tasks or chang- References he must always be aware of the market. ing environ-ments in the real world 1. Chance, B., Anday, E., Nioka, S., Zhou, S., Hong, L., A display that gently changes visualiza- with no physical manipulation at all. Worden, K., Li, C., Murray, T., Ovetsky, Y., and Thomas, R. 1998. A novel method for fast imaging of brain tions of the market according to the function, non-invasively, with light. Optics Express, user’s workload could be invaluable. If Biographies 411-423. the stockbroker is not currently exert- Evan Peck is a PhD candidate in the Human-Computer 2. Cutrell, E., and Tan, D. S. 2008. BCI for passive input in Interaction lab at Tufts University. His research focuses on HCI. Paper presented at the ACM CHI’08 Conference ing a high mental workload, the dis- creating adaptive brain-computer interfaces to improve on Human Factors in Computing Systems Workshop play can show his stock visualization user experience with commonplace computer tasks. on Brain-Computer Interfaces for HCI and Games. as highly detailed, giving the stockbro- Krysta Chauncey is currently a National Research 3. Wolpaw, J.R., Birbaumer, N., Mcfarland, D.J., Council Research Associate at the U.S. Army Natick Pfurtscheller, G., and Vaughan, T.M. Brain-computer ker as much information as possible. Soldier Research, Development & Engineering Center in interfaces for communication and control. Clinical collaboration with the Brain-Computer Interface Project Neurophysiology, 113 (6). 2002, 767-791. See Figure 4. If the stockbroker is work- at the Tufts Human-Computer Interface Laboratory. ing exceptionally hard at his email and Her primary interest is in applying the methods and

XRDS • summer 2010 • Vol.16 • No.4 47 Student_Books_and_Courses_4C_full-page_LMNTK:Layout 1 4/26/10 3:25 PM Page 1

Online Books ACM’s & Courses Programs! Helping Members Meet Today’s Career Challenges

NEW! 3,200 Online Courses in Multiple Languages Plus 1,000 Virtual Labs from Element K!

ACM’s new Online Course Collection includes over 3,200 online courses in multiple languages, 1,000 virtual labs, e-reference tools, and offline capability. Program highlights: The ACM E-Learning Catalog - round-the-clock access to 3,200 online courses on a wide range of computing and business topics, in multiple languages. Exclusive vLab® Virtual Labs - 1,000 unique vLab® exercises place users on systems using real hardware and software allowing them to gain important job-related experience. Reference Tools - an e-Reference Library extends technical knowledge outside of the classroom, plus online Executive Summaries and quick reference cards to answer on-the-job questions instantly. Offline Player - members can access assessments and self-study courses offline, anywhere and anytime, without a live Internet connection. A downloadable Quick Reference Guide and a 15-minute site orientation course for new users are also available to help members get started. The ACM Online Course Program is open to all ACM Student Members.

500 Online Books from Books24x7 All ACM Student Members have full access to 500 online books from Books24x7®, in ACM’s rotating collection of complete unabridged books on the hottest computing topics. This virtual library puts information at your fingertips. Search, bookmark, or read cover-to-cover. Your bookshelf allows for quick retrieval and bookmarks let you easily return to specific places in a book. Topics include:

� .Net � Java � Software Architecture � 3D Programming � Linux � Software Engineering � Algorithms � Mobile Systems � Software Testing � C++ � MySQL � Web Development � Certification � PHP � XML � Database Design and � Project Management � Plus, Engineering Titles Implementation � Python Business Titles � Flex � Ruby � Information Security

pd.acm.org www.acm.org/join Student_Books_and_Courses_4C_full-page_LMNTK:Layout 1 4/26/10 3:25 PM Page 1

Online Books & Courses Programs! ACM’s PROFILE Helping Members Meet Today’s Career Challenges “By complementing physical objects with NEW! 3,200 Online Courses in Multiple Languages Plus digital information, 1,000 Virtual Labs from Element K! we can improve and ACM’s new Online Course Collection includes over 3,200 online augment the way we courses in multiple languages, 1,000 virtual labs, e-reference perform tasks.” tools, and offline capability. Program highlights:

The ACM E-Learning Catalog - round-the-clock access to 3,200 online courses on a wide range of initial “shock” when he first saw the computing and business topics, in multiple languages. Xerox Alto (hailed as the first computer with a GUI) back in 1973 was what Exclusive vLab® Virtual Labs - 1,000 unique vLab® exercises place users on systems using real Photo of ClearBoard by NTT ICC; Inset photo by Webb Chappell. prompted his interests in HCI. Years hardware and software allowing them to gain important job-related experience. later, Ishii is now a leader in tangible user interface research and development. In Reference Tools - an e-Reference Library extends technical knowledge outside of the classroom, plus 2006, Ishii was elected by ACM SIGCHI online Executive Summaries and quick reference cards to answer on-the-job questions instantly. into the prestigious CHI Academy for his significant contributions to the field. Offline Player - members can access assessments and self-study courses offline, anywhere and anytime, Certainly, Ishii’s success did not come without a live Internet connection. without some initial roadblocks. One of the great challenges he has faced is A downloadable Quick Reference Guide and a 15-minute site orientation course for new users are also discovering “compelling applications” available to help members get started. to convince people of their vision in well-established HCI conferences, which The ACM Online Course Program is open to all ACM Student Members. Hiroshi Ishii Tangible Bits have traditionally been more focused on By David Chiu user-centered designs. Another ongoing DOI: 10.1145/1764848.1764859 challenge involves the fact that tangible user interfaces often require proprietary iroshi Ishii sees the a research group toward developing and non-standard hardware platforms, 500 Online Books from Books24x7 world differently. The two critical projects: TeamWorkStation but Ishii says he is optimistic about their Massachusetts Institute of and ClearBoard. TeamWorkStation was acceptance in the future. All ACM Student Members have full access to 500 online books from Books24x7®, in ACM’s rotating Technology professor of media designed in 1990 to provide real-time The growing number of researchers, artsH and sciences, widely regarded as the sharing of drawing space between designers, and artists who are collection of complete unabridged books on the hottest computing topics. This virtual library puts pioneer of tangible user interfaces (TUI), geographically disparate collaborators. contributing to the field of tangible information at your fingertips. Search, bookmark, or read cover-to-cover. Your bookshelf allows for is changing the way we interact with our It was enabled through a translucent user interfaces share his optimism. In quick retrieval and bookmarks let you easily return to specific places in a book. Topics include: surroundings by integrating computing video overlay of the collaborators’ fact, Ishii refers to the success of the and physical objects. Specifically, within workspaces. ClearBoard, developed in International Conference in Embedded � .Net � Java � Software Architecture his Tangible Media Group at MIT, Ishii and 1992, allowed for vis-à-vis interaction and Embodied Interaction series (TEI), � 3D Programming � Linux � Software Engineering his students are looking for ways to tie between two collaborators, and for the most recently held in January 2010 at physical objects to digital information in first time supported gaze awareness (so MIT Media Labs, as an encouraging sign � Algorithms � Mobile Systems � Software Testing a vision they call Tangible Bits. that the partner’s focus of attention was for the community. � C++ � MySQL � Web Development Their vision, which departs from the communicated) over a large, clear screen With these challenges currently � Certification � PHP � XML pervasive “painted bits” within current for drawing. These seminal efforts have being addressed and novel high-level graphical user interfaces, is led by the since been succeeded by a cornucopia of concepts coming to fruition, Ishii is � Database Design and � Project Management � Plus, Engineering Titles observation that humans have developed interface projects under Ishii’s lead. prepared to invoke the next big idea. He Implementation � Python Business Titles a lifetime of intuition manipulating Ishii, who received his PhD in believes that in the next five to ten years, � Flex � Ruby objects in the physical world. By computer engineering in 1992 from we can expect to see an integration of � Information Security complementing physical objects with Hokkaido University in Japan, recalls manipulatory and ambulatory interfaces digital information, we can improve and the circumstances that led him to as well as “a departure from a table augment the way we perform tasks. his current work. “My father was a [interface] to an entire room, building, Before joining MIT Media Labs in programmer of the IBM 360 mainframe and city.” As tangible user interfaces 1995, Ishii worked for NTT Human when I was a kid, [which] is why I chose continue to emerge and mature, we can Interface Labs in Japan, where he led computer science.” He added that his surely expect Ishii to lead this movement. pd.acm.org www.acm.org/join XRDS • summer 2010 • Vol.16 • No.4 49 Hello World

hello world Real-Time Detection with Webcam by Dmitry Batenkov

penCV is an open-source, interpreter knows the location for the Resources cross-platform library for OpenCV Python bindings. In Linux, it Object identification links real-time computer vision. should be set automatically. In Windows, Viola-Jones algorithm Originally developed set the environment variable http://www.face-rec.org/algorithms/ byO Intel, the library will use Intel’s set pythonpath = \ Boosting-Ensemble/16981346.pdf Integrated Performance Primitives, Python2.6\Lib\site-package. if it is found on the system. It is very Now, connect your webcam and run Haar training tutorial well-documented with a full reference the program: python facedetect.py http://note.sonots.com/SciSoftware/ manual, many examples and tutorials, To exit, press Esc. Have fun! haartraining.html and a book (which is also a good introduction to computer vision). Improvements Haar cascades repository Interfaces for C, C++, and Python are Once an object is detected, we can http://alereimondo.no-ip.org/ also available in OpenCV. start tracking it. OpenCV has an OpenCV/34 Example applications of the OpenCV implementation for CamShift tracking library include human-computer algorithm. (See the example on http:// HCI projects using opencv interaction; object identification, XRDS.acm.org.) HandVu segmentation and recognition; face Add detection of the eyes, mouth, Gesture recognition recognition; gesture recognition; and so on. (OpenCV ships with www.movesinstitute.org/~kolsch/ motion tracking, ego motion, motion corresponding classifiers.) You can HandVu/HandVu.html understanding; structure from motion recognize emotions! See the video: www. (SFM); stereo and multi-camera youtube.com/watch?v=V7UdYzCMKvw. EHCI Head tracking calibration and depth computation; and If you replace the face classifier with http://code.google.com/p/ehci mobile robotics. hands classifier, and add tracking, you In this tutorial, we will learn how can now recognize gestures! PyEyes Eyes tracking to do real-time face detection using —Dmitry Batenkov http://eclecti.cc/olpc/pyeyes-xeyes-in- a webcam. We will utilize a machine- python-with-face-tracking learning object detection algorithm known as the Viola-Jones detector. It’s CCV/touchlib Multi-touch library a fast classification mechanism using http://nuigroup.com Haar-like wavelet features. OpenCV ships with a very good “classifier file” other HCI/cv toolkits for faces, but one can also train the TUIO Common API for tangible classifier to recognize any kind of multitouch surfaces objects. www.tuio.org/?software (list of implementations) Instructions First, download the latest OpenCV “In this tutorial, Trackmate Do-it-yourself tangible release for your platform from we will learn how tracking system http://opencv.willowgarage.com and http://trackmate.media.mit.edu install it. to do real-time face Next, copy the attached program to detection using a Sphinx Speech recognition toolkit a file named facedetect.py. You can also www.speech.cs.cmu.edu/sphinx/ download it from http://XRDS.acm.org. webcam. We will tutorial.html In the downloaded source utilize a machine- archive, locate the classifier file VXL versatile computer vision data/haarcascades/haarcascade_ learning object libraries frontalface_alt_tree.xml and replace the detection algorithm http://vxl.sourceforge.net placeholder in the code with this original location. known as the Viola- Integrating Vision Toolkit Make sure that the Python Jones detector.” http://ivt.sourceforge.net

50 XRDS • SUMMER 2010 • Vol.16 • No.4 import sys # Draw the rectangle on the image import cv cv.Rectangle(img,pt1,pt2,cv.CV _ RGB(255,0,0),3,8,0) cv.ShowImage( “result”, img ) storage=cv.CreateMemStorage(0) image _ scale=1.3 if __name__==‘__main__’: haar _ scale=1.2 # Load the Haar cascade min _ neighbors=1 cascade _ name=“./haarcascade _ frontalface haar _ flags=0 alt _ tree.xml”

def detect_and_draw(img): cascade=cv.Load(cascade _ name) # allocate temporary images

gray=cv.CreateImage((img.width,img.height),8,1) # Start capturing.Can change index if more than one small _ img=cv.CreateImage((c v.R o u n d(i m g.w idth/ camera present

image _ s c a l e), capture=cv.CaptureFromCAM(0) cv.Round(img.height/image _ scale)), 8, 1 ) # Create the output window # convert color input image to grayscale cv.NamedWindow(“result”,1) cv.CvtColor( img, gray, cv.CV _ BGR2GRAY )

frame _ copy=None # scale input image for faster processing

cv.Resize( gray, small _ img, cv.CV _ INTER _ NN ) while True: frame=cv.QueryFrame( capture ) cv.EqualizeHist( small _ img, small _ img ) # make a copy of the captured frame # start detection if not frame _ copy: if( cascade ): frame _ copy=cv.CreateImage((frame. faces=cv.HaarDetectObjects( small _ img, cascade, storage, width,fram e.h e i g h t), haar _ scale, min _ neighbors, haar _ flags ) cv.IPL _ DEPTH _ 8U, frame.nChannels ) if faces: cv.Copy( frame, frame _ copy )

for (x,y,w,h),n in faces: detect _ and _ draw(frame _ copy) # the input to cvHaarDetectObjects was resized, so scale the # bounding box of each face and convert it to two CvPoints c=cv.WaitKey(7) pt1=(int(x*image _ scale),int(y*image _ scale)) if c==27: # Escape pressed pt2=(int((x+w)*image _ scale), break int((y + h )*i m a g e _ s c a l e))

XRDS • SUMMER 2010 • Vol.16 • No.4 51 end

Interns and employees alike gather and swap ideas in the atrium of Microsoft Research’s building 99.

Research interns visualized in new ways using tools they had developed. Proj- ects such as Bing Collaborative Search, Kodu, and Microsoft Portrait all have roots in this group. The Windows Mobile building next door provides a huge selection of fresh food cooked on demand. During lunch, it’s easy to get wrapped up chatting about projects, but it also often moti- vates lucrative collaboration between colleagues, which is encouraged by the company. Researchers typically spend half a work day per week on something aside from their primary projects. All interns are assigned a mentor in a field related to their project to guide them and act as their first point of con- tact. Additionally, Microsoft Research LABZ hosts weekly talks from other Microsoft divisions about current projects, gener- al topics, or even to get feedback on re- Microsoft Research leased products, opening up even more venues for networking and learning. Redmond, Washington If prototypes need to be construct- ed, the fully equipped on-site hardware lab, kitted out with a laser cutter, a CNC t 9:00 a.m., the Microsoft shut- and many of the same responsibilities, milling machine, an optics bench, and tle bus pulls up to an incon- as regular employees. more, is a perfect place to test out ideas spicuous office building in a Internships are one of the best ways in practice. It’s common to find groups leafy suburb of Seattle. Known to get a foot in the door with main- huddling over half-built prototypes late Aas building 99, this structure provides stream industry research. Internships at night, frantically preparing for dem- the homebase for Microsoft Research. give students a taste of the research os the next day. Inside, under the glass roofed atrium, community in a specific field, but also Amazingly, many of the cool demos groups of researchers are hard at work let them start networking with experi- seen at events like TechFest are actually discussing their projects over coffee, enced and likeminded people. built by a couple of dedicated guys in available free from Starbucks ma- building 99 using everyday materials. chines on every floor. The Intern Experience Although these projects rarely become The atrium serves as a hub for col- In the social computing lab on the full products, some do—such as Sur- laboration and discussion, with resi- ground floor, it’s hard not to get drawn face, Project Natal (for Xbox 360), and dents gathering throughout the day to into discussions on the benefits of Twit- Azure. Many projects contribute toward meet and socialize. The 300 people at ter with interns who are working on future releases in core technologies, Microsoft Research are a mix of per- ways to visualize and analyze social net- such as .NET and Windows, as well as manent researchers, visiting collabo- working data. In June 2009, when Mi- collaborations with other institutions. rators, and interns. Although interns chael Jackson died, news of his death hit Walking around building 99, it’s are only there for three months at a Twitter and caused fascinating social apparent that people enjoy working time, they get all the same privileges, grouping patterns, which the Microsoft here, and not just for the free coffee.

52 XRDS • summer 2010 • Vol.16 • No.4 20.9% Apple 33.1% 33.1% 6.4% Nokia 29.5% 1965 10.2% HTC 10.2% 564 29.5% Samsung 6.4% Douglas Engelbart of Stanford Others 20.9% Average number of mouse clicks per Research Laboratory (now SRI) Worldwide Touch Screen day by an average employee develops the computer mouse Smartphone Market Source: Remedy Interactive

Research is driven by the need to con- BACK tribute to the scientific community, by both improving existing ideas and sys- tems, and introducing new ones. Apple’s Mouse If interns have any energy left after a full day in the lab, they can join some of 25 Years Later the local trips that Microsoft organizes to see the sights of Seattle and Wash- Few input devices revolutionized interaction as much as the mouse. ington state. Some excursions have in- Much of the world was first exposed to the mouse when Apple cluded the nearby Olympic Mountains, released the Macintosh 128k in 1984 and the boxy mouse that came downtown Seattle, the Boeing factory, with it. In 2009, Apple integrated multi-touch gestures with the and local arts festivals. Magic Mouse. Users now have a range of motions tied directly to their fingertips. Such drastic improvements raise just one question: Will How to Get Involved the mouse ever disappear from our desks? —James Stanier Microsoft Research accepts applica- tions for interns on a rolling basis via an online submission form (http:// research.microsoft.com/en-us/jobs/ intern/apply.aspx), though it is restrict- ed to masters-level and PhD students only. Internships, which are paid, usu- ally last 12 weeks and are offered at any of Microsoft’s eight labs worldwide, though the majority are in Redmond. The review process can take up to three months. Selected candidates are 1984 2009 then invited to interview. If accepted, Apple Macintosh Mouse Apple Magic Mouse the company provides some support in finding accommodation, making trav- Released Released el arrangements, and securing visas. January 24, 1984 October 20, 2009 —Tom Bartindale Price Price Other top industry labs Included with Macintosh 128k $69, or incl. with desktop computer Intel Labs has three labs in three lo- cations: Berkeley, California; Seattle, Interactivity Interactivity Washington; and Philadelphia, Penn- One physical button Gestural multi-touch interface with sylvania. www.intel-research.net 360° scroll and two-finger swipe

IBM Research has eight labs spread Tracking Tracking all over the world, offering numerous Mechanical ball Infrared laser diode opportunities for students outside North America. www.research.ibm.com Slogan Slogan “If you can point, you can use a “Suddenly, everything clicks. And INRIA is a French national research Macintosh.” swipes. And scrolls.” institution, focused on three areas: computer science, control theory, and Criticism Criticism applied mathematics.www.inria.fr “The Macintosh uses an “We struggled through a difficult experimental pointing device called learning curve due to its uniformly PARC, the Palo Alto Research Center is a ‘mouse.’ There is no evidence that narrow profile that sits too low for business-focused and to some extent people want to use these things.” comfort.”—CNET client-driven. www.parc.com —John C. Dvorak

XRDS • summer 2010 • Vol.16 • No.4 53 end

EVENT DATES June 20-24 Cost: $250-$600 Conferences & www.hawaii.edu/UMAP2010 Journals Afrigraph 2010 Computer Graphics International 2010 Rickety Bridge Wine Estate Singapore, Singapore Franschhoek, South Africa June 8-11,2010 June 21-23 Cost: SGD 960, SGD 580 (student) Cost: 5200 ZAR (international students) http://cgi2010.miralab.unige.ch www.afrigraph.org/conf2010

IDC 2010- The 9th International SG ‘10: 10th International Symposium on featured event Conference on Interaction Design and SMART GRAPHICS Children Banff Centre MobileHCI International Conference Pompeu Fabra Universit Banff, Canada on Human-Computer Interaction with Barcelona, Spain June 24-26, 2010 Mobile Devices and Services June 9-12 Cost: unavailable at press time Lisboa, Portugal Cost: €410, €370, €130 (student) www.smartgraphics.org September 7-10,2010 www.iua.upf.edu/idc2010 http://mobilehci2010.di.fc.ul.pt Create10 - Innovative Interactions /index.html EuroVis 2010 - Eurographics/IEEE Edinburgh Napier University Technology is shrinking and becoming Symposium on Visualization Edinburgh, U.K. more and more mobile, and the Bordeaux 1 campus June 30-July 2 opportunities to work with small, Bordeaux, France Cost: £290, £80 (Students) mobile devices are constantly growing. June 9-11 www.create-conference.org MobileHCI provides a unique forum for Cost: €240 (student) academics and practitioners to come http://eurovis.org EuroHaptics 2010 together and discuss innovations, Vrije Universiteit challenges, solutions, and ideas for EuroITV 2010 - 8th European Conference Amsterdam, The Netherlands the many aspects of human-computer on Interactive TV & Video July 8-10 interaction in mobile computing. The Tampere University of Applied Sciences Cost: unavailable at press time relaxed environment of this conference and the hotel Holiday Club Tampere www.eurohaptics2010.org encourages open discussion among Tampere, Finland attendees and provides an opportunity June 9-11 eNTERFACE 10 - 6th International to interact with a wide variety of people Cost: unavailable at press time Summer Workshop on Multimodal with common interests and many www.euroitv2010.org Interfaces different perspectives. UvA Building at Science Park Amsterdam MobileHCI 2010 will be held in ACM Hypertext 2010 Amsterdam, The Netherlands Lisbon, Portugal, and promises an Emmanuel College, University of Toronto July 12 - August 6 engaging program covering design, Toronto, Canada Cost: free evaluation, and application for June 13-16 http://enterface10.science.uva.nl mobile and wearable computing. Cost: $450, $350 (ACM/SIGWEB), The conference will include full and $195 (student) SIGGRAPH 2010 short research papers; workshops to www.ht2010.org Los Angeles Convention Center foster discussion based on selected Los Angeles, U.S. themes; tutorials; interactive panels; ACM SIGCHI Symposium on Engineering July 25-29 practically-motivated industrial Interactive Computing Systems Cost: $900, $850(ACM SIGGRAPH case studies; a design competition Ernst-Reuter-Haus member), $350 (student member) to encourage innovative problem Berlin, Germany www.siggraph.org/s2010 solving; a future innovations track to June 21-23 encourage fresh concepts; a doctoral Cost: €450 (student) DIS 2010: Designing Interactive Systems consortium to encourage student http://research.edm.uhasselt.be Aarhus School of Architecture feedback and interaction; posters that /~ep-muti2010 Aarhus, Denmark provide the opportunity to informally August 16-20 discuss current work; and hands-on UMAP 2010 - 18th International Cost: unavailable as of press time demonstrations. Conference on User Modeling, Adaptation www.dis2010.org Keynote speakers include Patrick and Personalization Baudisch, Scott Jenson, and Josh Ulm. Hilton Waikoloa Village HCI 2010 - Play is a serious business —Jason Wiese Waikoloa, Hawaii, U.S. University of Abertay Dundee,

54 XRDS • summer 2010 • Vol.16 • No.4 Dundee, Scotland AISES Google Scholarship POINTERS September 6-10 Deadline: June 15 Cost: unavailable as of press time Eligibility: American Indian, Alaska Native Where to Find HCI Events http://hci2010.abertay.ac.uk and Native Hawaiian AISES members ACM Special Interest Group on Computer pursuing degrees in computer science or Human Interaction’s calendar 23rd ACM Symposium on User Interface computer engineering, enrolled full-time www.sigchi.org/conferences/ Software and Technology as undergraduate or graduate students calendarofevents.html New York Hall of Science (or in their second year of a two-year New York, New York, U.S. college with a plan to move to a four-year ACM Conferences Deadline to submit posters, doctoral institution) with 3.5/4.0GPA. www.acm.org/conferences symposium: June Benefits: $10,000 scholarship Conference date: October 3-6, 2010 www.aises.org Interaction-Design.org’s calendar Cost: unavailable as of press time www.interaction-design.org/calendar www.uist2010.org Hertz Graduate Fellowship Award Deadline: October 2010 HCI Groups CHIMIT 2010: Computer Human Eligibility: Graduate students working Special Interest Group on Computer Interaction for the Management of toward a PhD degree in the applied Human Interaction Information Technology physical, biological, and engineering www.sigchi.org San Jose, California, U.S. sciences. Must study at one of the Deadline to submit: July 3 Foundations tenable schools (see www. HCI Resources Conference date: November 7-8 hertzfoundation.org/dx/fellowships/ HCI Resources & Bibliography Cost: $75-$100 (student) schools.aspx) or be willing to petition for www.hcibib.org www.chimit10.org your school to be accepted. Benefits: Merit-based scholarships vary but Scientometric Analysis of the CHI TEI’11: Fifth International Conference can consist of cost-of-education support Proceedings on Tangible, Embedded, and Embodied and personal support, up to $31,000. www.bartneck.de/publications/2009/ Interaction www.hertzfoundation.org scientometricAnalysisOfTheCHI Funchal, Portugal Deadline to submit: August 1 Contests & Events ACM interactions magazine Conference date: January 22-26, 2011 http://interactions.acm.org Cost: unavailable as of press time ACM Student Research Competition at www.tei-conf.org/11 ASSETS 2010 graduate Programs Held in conjunction with ACM SIGACCESS HCI International 2011 ASSETS 2010 Carnegie Mellon University, Human- Hilton Orlando Bonnet Creek Orlando, FL Computer Interaction Institute Orlando, Florida, U.S. Deadline to submit: July 2 www.hcii.cmu.edu Deadline to submit papers: October 15 Event date: October 25-27 Conference date: July 9-14, 2011 www.sigaccess.org/assets10/competition.html Georgia Institute of Technology, Cost: unavailable as of press time Human-Computer Interaction www.hcii2011.org/index.php International Olympiad in Informatics www.cc.gatech.edu/hci University of Waterloo Scholarships, Fellowships & Waterloo, Ontario, Canada Massachusetts Institute of Technology, Grants Deadline to submit: Competitions take MIT Media Laboratory place August 16 and 18 during week-long www.media.mit.edu Korean American Scholarship Fund event Deadlines: May 31 (schools in the western, Event dates: August 14-20 Stanford University, HCI Group mid-western, mid-eastern, and southern www.ioi2010.org http://hci.stanford.edu regions); June 15, 2010 (eastern schools); June 26 (northeastern schools) SMART Multitouch Application Contest University of California—Irvine, Eligibility: Full-time undergraduate Design a software application for the Human Computer Interaction and graduate students, as well as high SMART Table www.ics.uci.edu/faculty/area/area_hci.php school juniors, of Korean-American Deadline to submit: July 1 background, studying in the U.S. may Winners Announced: September 1 University of Maryland, apply. Scholarships are awarded based on www2.smarttech.com/ Human-Computer Interaction Lab financial need, academic achievement, www.cs.umd.edu/hcil school activities, and community services. Benefits: Award amounts and distribution University of Michigan, Human-Computer timelines vary. Interaction Specialization www.kasf.org www.si.umich.edu/msi/hci.htm

XRDS • summer 2010 • Vol.16 • No.4 55 end

BEMUSEMENT Can Oleg Beat Erdos? A puzzle to get started, from our friends at CMU Oleg and (the ghost of) Erdös play the fol- who then chooses a non-negative inte- —Tom Bohman, Oleg Pikhurko, Alan Frieze,

lowing game. Oleg chooses a non-nega- ger bi, and then Oleg defines ai+1 = |ai − and Danny Sleator at The Puzzle Toad.

tive integer a1 with at most 1,000 digits. bi| or ai+1 = ai + bi. Erdös wins if a20 is a In Round i the following happens: power of 10, otherwise Oleg wins. Find the solution at: http://www.

Oleg tells the number ai to Erdös, Who is the winner, Oleg or Erdös? cs.cmu.edu/puzzle/Solution22.pdf Debugging

Borders xkcd.com

submit a puzzle Can you do better? Bemusements would like your puzzles and mathematical games (but not Sudoku). Contact [email protected] to submit yours!

56 XRDS • Summer 2010 • Vol.16 • No.4 Crossroads App FY'11:Crossroads Apprev 4/21/10 11:09 AM Page 1

acm STUDENT MEMBERSHIP APPLICATION Join ACM online: www.acm.org/joinacm

CODE: CRSRDS Name Please print clearly INSTRUCTIONS

Address Carefully complete this application and return with payment by mail or fax to ACM. You must

City State/Province Postal code/Zip be a full-time student to qualify for student rates.

Country E-mail address CONTACT ACM

Area code & Daytime phone Mobile Phone Member number, if applicable phone: 800-342-6626 (US & Canada) MEMBERSHIP BENEFITS AND OPTIONS +1-212-626-0500 (Global) • Print & electronic subscription to XRDS: The ACM • Free e-mentoring service from MentorNet Magazine for Students hours: 8:30am–4:30pm • ACM online newsletter MemberNet (twice monthly) US Eastern Time • Electronic subscription to Communications of • Student Quick Takes, ACM student e-newsletter fax: +1-212-944-1318 the ACM magazine (quarterly) email: [email protected] • Free software and courseware through the ACM • ACM's Online Guide to Computing Literature Student Academic Initiative mail: Association for Computing • Free "acm.org" email forwarding address plus Machinery, Inc. • 3,200 online courses in multiple languages, filtering through Postini 1,000 virtual labs and 500 online books General Post Office • Option to subscribe to the full ACM Digital Library P.O. Box 30777 • ACM's CareerNews (twice monthly) • Discounts on ACM publications and conferences, New York, NY 10087-0777 • ACM e-news digest TechNews (tri-weekly) valuable products and services, and more For immediate processing, FAX this PLEASE CHOOSE ONE: application to +1-212-944-1318. ❏ Student Membership: $19 (USD) ❏ Student Membership PLUS Digital Library: $42 (USD) ❏ Student Membership PLUS Print CACM Magazine: $42 (USD) PAYMENT INFORMATION ❏ Student Membership w/Digital Library PLUS Print CACM Magazine: $62 (USD) Payment must accompany application PUBLICATIONS Please check one Member dues ($19, $42, or $62) $ Check the appropriate box and calculate Issues Member Member Rate To have Communications of the ACM amount due. per year Code Rate PLUS Air* sent to you via Expedited Air Service, • ACM Inroads 4 247 $13 ❐ $65 ❐ • acmqueue website www.acmqueue.com N/A N/A N/A N/A add $50 here (for residents outside of • Computers in Entertainment (online only) 4 247 $40 ❐ N/A North America only). $ Computing Reviews 12 104 $52 ❐ $85 ❐ • Computing Surveys 4 103 $33 ❐ $59 ❐ Publications $ • XRDS: Crossroads, The ACM Magazine for Students 4 XRoads N/A N/A (included with student membership) Total amount due $ • interactions (included in SIGCHI membership) 6 123 $19 ❐ $48 ❐ Int’l Journal of Network Management (Wiley) 6 136 $90 ❐ $116 ❐ Check or money order (make payable to ACM, • Int’l Journal on Very Large Databases 4 148 $81 ❐ $107 ❐ • Journal of Experimental Algorithmics (online only) 12 129 $29 ❐ N/A Inc. in U.S. dollars or equivalent in foreign currency) • Journal of Personal and Ubiquitous Computing 6 144 $63 ❐ $89 ❐ • Journal of the ACM 6 102 $52 ❐ $104 ❐ ❏Visa/Mastercard❏ American Express • Journal on Computing and Cultural Heritage 4 173 $46 ❐ $63 ❐ • Journal on Data and Information Quality 4 171 $46 ❐ $65 ❐ ❐ ❐ • Journal on Emerging Technologies in Computing Systems 4 154 $39 $56 Card number Exp. date • Linux Journal (SSC) 12 137 $29 ❐ $58 ❐ • Mobile Networks & Applications 4 130 $70 ❐ $95 ❐ • Wireless Networks 4 125 $70 ❐ $95 ❐ Signature Transactions on: • Accessible Computing 4 174 $46 ❐ $64 ❐ Member dues, subscriptions, and optional contributions • Algorithms 4 151 $49 ❐ $66 ❐ are tax deductible under certain circumstances. Please • Applied Perception 4 145 $40 ❐ $57 ❐ consult with your tax advisor. • Architecture & Code Optimization 4 146 $40 ❐ $57 ❐ • Asian Language Information Processing 4 138 $36 ❐ $53 ❐ • Autonomous and Adaptive Systems 4 158 $38 ❐ $55 ❐ EDUCATION • Computational Biology and Bio Informatics 4 149 $18 ❐ $63 ❐ • Computer-Human Interaction 4 119 $40 ❐ $59 ❐ • Computational Logic 4 135 $41 ❐ $60 ❐ ❐ ❐ • Computation Theory 176 $46 $72 Name of School • Computer Systems 4 114 $44 ❐ $63 ❐ • Computing Education (online only) 277 $38 ❐ N/A Please check one: ❐ High School (Pre-college, Secondary • Database Systems 4 109 $43 ❐ $62 ❐ School) College: ❐ Freshman/1st yr. ❐ Sophomore/2nd yr. • Design Automation of Electronic Systems 4 128 $40 ❐ $59 ❐ ❐ Junior/3rd yr. ❐ Senior/4th yr. Graduate Student: ❐ • Embedded Computing Systems 4 142 $41 ❐ $58 ❐ ❐ ❐ • Graphics 4 112 $48 ❐ $67 ❐ Masters Program Doctorate Program Postdoctoral • Information & System Security 4 134 $41 ❐ $58 ❐ Program ❐ Non-Traditional Student • Information Systems 4 113 $44 ❐ $63 ❐ • Internet Technology 4 140 $39 ❐ $56 ❐ • Knowledge Discovery From Data 4 170 $47 ❐ $64 ❐ Major Expected mo./yr. of grad. • Mathematical Software 4 108 $44 ❐ $63 ❐ • Modeling and Computer Simulation 4 116 $48 ❐ $67 ❐ ❐ ❐ ❐ ❐ • Multimedia Computing, Communications, and Applications 4 156 $39 ❐ $56 ❐ Age Range: 17 & under 18-21 22-25 26-30 • Networking 6 118 $28 ❐ $76 ❐ ❐ 31-35 ❐ 36-40 ❐ 41-45 ❐ 46-50 ❐ 51-55 ❐ 56-59 ❐ 60+ • Programming Languages & Systems 6 110 $56 ❐ $82 ❐ • Reconfigurable Technology & Systems 4 172 $46 ❐ $64 ❐ Do you belong to an ACM Student Chapter? ❐ Yes ❐ No • Sensor Networks 4 155 $39 ❐ $56 ❐ • Software Engineering and Methodology 4 115 $40 ❐ $59 ❐ I attest that the information given is correct and that I will • Speech and Language Processing (online only) 4 153 $39 ❐ N/A abide by the ACM Code of Ethics. I understand that my • Storage 4 157 $39 ❐ $56 ❐ membership is non transferable. • Web 4 159 $38 ❐ $55 ❐ marked • are available in the ACM Digital Library *Check here to have publications delivered via Expedited Air Service. For residents outside North America only. PUBLICATION SUBTOTAL: Signature cscw2011-full-page-v04-paths.pdf 1 4/30/10 12:25 PM

C

M

Y

CM

MY

CY

CMY

K