Open Source Eyetap: Empowering Every Maker with Phenomenal Augmented Reality and Wearable Computing

ISWC '17, SEPTEMBER 11–15, 2017, MAUI, HAWAII, USA

Open Source EyeTap: Empowering Every Maker with Phenomenal Augmented Reality and Wearable Computing

Sarang Nerkar Max Hao Lu Abstract University of Toronto University of Toronto Augmented reality and wearable computing development Toronto, ON M5S 3G4, Canada Toronto, ON M5S 3G4, Canada has skyrocketed in the consumer product domain in the [email protected] [email protected] past few decades, while the open source domain remained rather neglected. We believe that this is due to the existing development platforms being expensive, closed-source and Sen Yang Alex Papanicolaou due to the lack of a strong open source augmented reality University of Toronto University of Toronto community. Toronto, ON M5S 3G4, Canada Toronto, ON M5S 3G4, Canada sen@eyetap.org [email protected] We present a hardware and software open source wearable augmented reality platform which enables users to make augmented reality glasses that cost less than $250 and re- Cindy Jinhee Park Steve Mann alize new applications of augmented reality which can then University of Toronto Stanford University be added to the platform for others to use and contribute to. Toronto, ON M5S 3G4, Canada Stanford, CA 94305, USA [email protected] [email protected] Author Keywords Mediated Reality; Augmented Reality; Wearable Comput- ing; EyeTap; Open Source, Open Innovation; Maker; Devel- oper; Tinkerer; Open Source Community

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed Introduction for profit or commercial advantage and that copies bear this notice and the full citation Augmented reality technology enhances the natural percep- on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, tion of humans by enabling them to sense and interact with to post on servers or to redistribute to lists, requires prior specific permission and/or a things that they normally wouldn‘t be able to, thus augment- fee. Request permissions from [email protected]. ing to the natural capabilities of humans[17]. In his child- ISWC '17, September 11–15, 2017, Maui, HI, USA © 2017 Association for Computing Machinery. hood in the 1970s, Mann invented phenomenological aug- ACM ISBN 978-1-4503-5188-1/17/09...$15.00 mented reality[10], wearable computing, and quantimetric https://doi.org/10.1145/3123021.3123075

203 ISWC '17, SEPTEMBER 11–15, 2017, MAUI, HAWAII, USA

self-sensing (later called “quantified self”) which he brought open source framework that is not only open source in the to MIT in the early 1990s to found the MIT wearable com- software end but also at the hardware end, while also being puting project (http://wearcam.org/nn). The miniaturization affordable [13][7]. We are not aware of any existing solu- of the overall system enabled users to wear this system tions that enable the developer community to develop and outside the laboratory environment and use it on a daily ba- contribute to a platform. In this paper, we present a hard- sis[18]. Mann later used this work to co-found InteraXon[4] ware and software open source wearable augmented reality and Metavision (Meta Company)[6]. platform which enables the users to make augmented reality glasses that cost less than $250 and realize new ap- Recent implementations of augmented reality are focused plications of augmented reality which can then be added to mostly on overlaying images onto reality. However, the ac- the platform for others to use and contribute to. tual essence of augmented reality isn’t limited to just that, it can also be performed in forms of various seeing or hear- Motivation ing aids [12]. One example is using real time high dynamic Wearable computing and augmented reality have grown ex- range video to enhance the overall dynamic range of the ponentially in the past few decades, in the consumer mar- human eye [16][1][14]. Thus enabling humans to see ob- ket domain. Although, there are various augmented reality jects that are a lot brighter than what their eyes can handle. development glasses available in the market today, these Mann developed one of the first augmented reality seeing products enable developers to develop games and apps for aids, called EyeTap [11]. Since then various augmented the consumer market only. We believe that the real potential Figure 1: Top: One of world‘s first reality headsets have come into place such as Google of augmented reality hasn’t been realized. Current devel- wearable augmented reality Glass, Microsoft Hololens and Meta glasses. There has opment has happened for people living in cities that can af- computer systems developed by been exponential development in the consumer applications ford to spend $1000 or more on augmented reality glasses Mann. Middle: The EyeTap of augmented reality glasses since Mann’s EyeTap [20] . to replace their desktop computers, and this development system. Bottom: Mann‘s recent However, there does not seem to be significant growth in EyeTap, completed in 1998 (19 is mostly closed-source. The applications of augmented the open source augmented reality domain. There are a years ago), which he wears on a reality in rural areas, underprivileged areas and minority daily basis. lot of open source augmented reality software frameworks communities have not been realized yet. We believe that that enable hackers, tinkerers and developers to develop this is because the available technology is expensive and augmented reality applications. But the way these frame- doesn‘t focus on the needs of the billions of people in the works are designed only enables development of con- world. The technology, right now, focuses on the few thou- sumer applications of augmented reality such as games sands who can buy these products. One way of realizing and apps., while prohibiting enthusiasts from having the these applications is by enabling the billions of people to freedom to modify or design hardware systems tailored to find the solutions to their problems with augmented reality their unique applications. Furthermore, these frameworks and wearable computing on their own [5][2][19][3]. This can require the developers to buy the consumer augmented re- be done by making the technology open source and provid- Figure 2: EyeTap principle as in [9] ality glass developer kits that cost over $1000, a cost that ing a platform where people can use and contribute to the most early-stage developers cannot afford. To embody the technology, similar to what Linux did with operating systems true essence of augmented reality our society needs an [15]. This thought of having an open source platform for

204 DESIGN EXHIBITION

augmented reality and wearable computing has been the motivation for this project.

Modularity: A New Approach in Wearable Eyeglasses We approached various maker communities to realize the requirements for augmented reality development. While conducting this survey we recognized certain attributes that make a device suitable for augmented reality development. We realized that in order for the maker community to realize applications in augmented reality that solve problems in their localities, the device needs to be modular and open source, not only in software but also in hardware. Open source and modularity together eliminate the problem of being limited by the given technology, because the makers can just add the missing technology pieces for the development of their application. This enables open innovation, an approach that has proven to be successful in enabling new specialized application for various professions[3][2][19]. We Figure 5: Modularity enables multiple functionalities in the Open have already seen makers develop different modules with Source EyeTap our system for various applications such as eye tracking, clear vision at a distance and thermal vision(See figure 5). We envision that our system will enable a lot more makers Design Approach to solve problems and enhance the capabilities of people in The existing EyeTaps were made from various materials in- Figure 3: Open Source EyeTap their localities with augmented reality. cluding off-the-shelf safety helmet, CNC milled stainless design as part of Mobile World steel, and injection moulded plastic as seen in figure 1. Congress special fashion show in We tested different wearable eyeglasses and head up dis- We performed an evaluation of the existing manufactur- Barcelona, 2017. plays to see how they compare with our system based on ing methods and witnessed the benefits and drawbacks of the attributes that we realized are suitable for augmented using these methods. We then compared them with our reality development(See figure 6). We found out that cost proposed manufacturing method: 3D printing. Although 3D is a major reason why modularity is not a favorable choice printing is still at the early stages of development and adop- for commercial products. The customization makes it less tion, it shows a promising future in the ergonomic related, Figure 4: Base frame design with effective for mass production. Our system does not face highly customized applications like smart wearables. With separated frame and nose piece this problem because advanced tabletop manufacturing an emphasis on design for everyday use, this Open Source techniques such as 3D printing allow for production of cus- EyeTap design was also used in a fashion show at Mobile tomized personal apparel such as eyeglasses. World Congress 2017, as shown in figure 3.

205 ISWC '17, SEPTEMBER 11–15, 2017, MAUI, HAWAII, USA

"# $%! #$ & & &

Figure 6: Comparison of different wearable eyeglasses and head up displays based on attributes suitable for augmented reality development

We have designed spaces to mount external mechanical modules to meet the functional needs of various applica- Design Description tions. In the current design, there is a reserved spot at the The open source EyeTap is modular in nature, these mod- centre of the main frame for these external modules. Users ules fall into three major architectures: mechanical, hard- can choose to implement various modules such as cam- ware, and software. eras, IMUs (Inertial Measurement Units), and other sen- sors. Mechanical Architecture Mechanical architecture is composed of four main compo- Hardware Architecture nents: the base frame, EyeTap optics, computing appara- We propose the use of the Raspberry-Pi system as the tus, and external modules. The base frame is where the main processing component for our design.The choice other components are mounted, as shown in figure 4. It aligns well with our goal of providing cheap, accessible, consists of a round piece of plastic with a separate nose open-source solutions to individual developers and hobby- piece. The reasons for this detachable design are: 1. It re- ists. The compact size and low power consumption of these duce the complexity of the 3D printing job. 2. It allows the units also suits well for a wearable device system. Lastly, nose piece to have various sizes to fit comfortably on differ- Figure 7: Evolution of open the open-source, Linux based operating system, Raspbian, ent facial structures. source EyeTap. Top: Initial design is compatible on any version of the Raspberry Pi, and can be written into a microUSB card to enable plug-and-play using Raspberry Pi 3, which The EyeTap optics consist of the components that are de- requires a head strap to hold in experience for everyone. scribed in [8], as shown in figure 2. It has: A two sided see- place. Middle: Intermediate design through mirror: this is a cheap alternative to expensive with ear hook and a rail for easy Software Architecture assembly and micro-adjustment of silver-plated optical beam splitters. Thinner see-through As the Raspbian operating system is entirely open-source, components. Bottom: Current mirrors are recommended to avoid double image effects. A any developer is welcome to develop, add, and interface design using Raspberry Pi Zero to camera and an off-the-shelf micro-display with a resolution the proposed system with customized components to de- minimize the physical size. of 720*540 pixels. velop individualized applications. By default, we suggest to provide the developers with stable releases of a wrapper

206 DESIGN EXHIBITION

application, that acts as a user interface and can be used be solved with augmented reality and develop solutions for to execute the installed applications. The application is pro- them using our platform. This will help us to get an initial grammed to launch automatically on boot-up, and runs in application base for our platform while also serving as an fullscreen. The user may navigate through these applica- example for its´ use. This will involve detailed documenta- tions using knobs, buttons or any connected interface, and tion of the development process from our base platform to may use another button to select them. The operating sys- a complete application, added to our application database. tem manages the launching and termination of the internal We will also conduct more user studies and start pilot pro- applications. grams in schools and universities where students will use our platform to develop solutions to problems in their locality Prototypes [5][2][19][3]. These pilot programs will be done in various One particular benefit of 3D printing is that it allows de- locations such as villages in India, China, Silicon Valley and signers to go through short iteration. We started with an public schools in Canada. This will enable us to get a good integrated design that had most of the components in an spectrum of people to use our platform and show the real enclosure. While this design had most components hidden, potential of augmented reality. it is was very complicated to print, because it required high precision printers to print it. Thus, making it not accessible Software Distribution to most at-home makers. Also, this integrated design left Upon system release, we propose a forum to be hosted to little space for other modules to be mounted while having provide instructions on releasing applications for develop- poor weight balance near the forehead. In the following iter- ers, to the server. The site will also distribute applications ations, we emphasised on design for printability, human fac- and interface releases for download. Lastly, we propose the tors, and modularity. In our most recent iteration, the weight site to be used to allow topic discussions and community of the device is reduced by 30% with a more open concept support. design principle. The evolution of this open source EyeTap design is shown in figure 7. The current Open Source Eye- Conclusion Tap design has been reproduced by multiple makers as a A hardware and software open source wearable augmented user study, and as a result, over 75% of them were able reality platform has been presented. We hope to witness to manufacture and assemble this device by themselves. various applications of augmented reality being added to This initiative has gained interest not only from experienced this platform as time passes. We have thus created tech- makers, but also junior makers and professionals. Some nology that will enable the developer, hacker and tinkerer pictures of people wearing their implementation of the Eye- community to get their hands on an affordable platform that Tap are shown in figure 8. helps them use and contribute the the various applications of augmented reality [5][2][19][3]. Future Work We are using this design exhibition as a platform for releas- References Figure 8: Makers wearing their ing our open source augmented reality platform. In the next [1] Mir Adnan Ali et al. “Comparametric HDR (High Dy- implementations of Open Source step, we will continue to realize various problems that can namic Range) imaging for digital eye glass, wearable EyeTaps.

207 ISWC '17, SEPTEMBER 11–15, 2017, MAUI, HAWAII, USA

cameras, and sousveillance”. In: Technology and So- [12] Steve Mann et al. “Blind navigation with a wearable ciety (ISTAS), 2013 IEEE International Symposium range camera and vibrotactile helmet”. In: Proceed- on. IEEE. 2013, pp. 107–114. ings of the 19th ACM international conference on [2] Andrea Bonaccorsi and Cristina Rossi. “Why open Multimedia. ACM. 2011, pp. 1325–1328. source software can succeed”. In: Research policy [13] Steve Mann et al. “FreeGlass for developers,"haccessibility", 32.7 (2003), pp. 1243–1258. and Digital Eye Glass+ Lifeglogging research in a [3] Henry William Chesbrough. Open innovation: The (sur/sous) veillance society”. In: Information Society new imperative for creating and proﬁting from tech- (i-Society), 2013 International Conference on. IEEE. nology. Harvard Business Press, 2006. 2013, pp. 48–53. [4] Jon Evans. “Here comes the wetware”. In: "TechCrunch, [14] Steve Mann et al. “Realtime HDR (high dynamic https://techcrunch.com/2010/12/04/wetware/" (Dec 4, range) video for eyetap wearable computers, fPGA- 2010), pp. 27–45. based seeing aids, and glasseyes”. In: in Proc. IEEE [5] Rod Furlan. “Build your own google glass [Resources_Hands CCECE 2012. Citeseer. 2012. On]”. In: IEEE Spectrum 50.1 (2013), pp. 20–21. [15] Glyn Moody. Rebel code: Linux and the open source [6] Meron Gribetz and Steve Mann. Head Mounted Dis- revolution. Basic Books, 2002. play Connected with Computational Device and Depth- [16] Sarang Nerkar et al. “Extrapolative Lightspace Method Sensor with Software Applications and User Inter- for HDR Video Exposure Selection”. In: Multimedia face. January 3, 2013. (ISM), 2016 IEEE International Symposium on. IEEE. [7] Steve Mann. “Humanistic computing:" wearcomp" as 2016, pp. 397–398. a new framework and application for intelligent sig- [17] Pete Scourboutakos et al. “Phenomenologically Aug- nal processing”. In: Proceedings of the IEEE 86.11 mented Reality With New Wearable LED Sequential (1998), pp. 2123–2151. Wave Imprinting Machines”. In: Proceedings of the [8] Steve Mann. Intelligent image processing. John Wiley Tenth International Conference on Tangible, Embed- & Sons, Inc., 2001. ded, and Embodied Interaction. ACM. 2017, pp. 751– [9] Steve Mann. “My "augmediated" life: What I’ve learned 755. from 35 years of wearing computerized eyewear”. In: [18] Thad Starner et al. “Augmented reality through wear- IEEE Spectrum (2013). able computing”. In: Presence: Teleoperators and [10] Steve Mann. “Phenomenal Augmented Reality: Ad- Virtual Environments 6.4 (1997), pp. 386–398. vancing technology for the future of humanity”. In: [19] Eric Von Hippel. “Innovation by user communities: IEEE Consumer Electronics (October 2015), cover + Learning from open-source software”. In: MIT Sloan 92-97. management review 42.4 (2001), p. 82. [11] Steve Mann, James Fung, and Eric Moncrieff. “Eye- [20] Feng Zhou, Henry Been-Lirn Duh, and Mark Billinghurst. tap technology for wireless electronic news gather- “Trends in augmented reality tracking, interaction and ing”. In: ACM SIGMOBILE Mobile Computing and display: A review of ten years of ISMAR”. In: Pro- Communications Review 3.4 (1999), pp. 19–26. ceedings of the 7th IEEE/ACM International Sympo- sium on Mixed and Augmented Reality. IEEE Com- puter Society. 2008, pp. 193–202.

208