Eyetap Devices for Augmented, Deliberately Diminished, Or Otherwise Altered Visual Perception of Rigid Planar Patches of Real-Wo
Total Page:16
File Type:pdf, Size:1020Kb
Steve Mann EyeTap Devicesfor Augmented, [email protected] Deliberately Diminished,or James Fung Otherwise AlteredVisual [email protected] University ofToronto Perceptionof Rigid Planar 10King’ s College Road Patches ofReal-World Scenes Toronto,Canada Abstract Diminished reality is as important as augmented reality, and bothare possible with adevice called the RealityMediator. Over thepast twodecades, we have designed, built, worn,and tested many different embodiments ofthis device in thecontext of wearable computing. Incorporated intothe Reality Mediator is an “EyeTap”system, which is adevice thatquanti es and resynthesizes light thatwould otherwise pass through one or bothlenses ofthe eye(s) ofa wearer. Thefunctional principles of EyeTap devices are discussed, in detail. TheEyeTap diverts intoa spatial measure- ment system at least aportionof light thatwould otherwise pass through thecen- ter ofprojection ofat least one lens ofan eye ofa wearer. TheReality Mediator has at least one mode ofoperation in which itreconstructs these rays oflight, un- der thecontrol of a wearable computer system. Thecomputer system thenuses new results in algebraic projective geometry and comparametric equations toper- form head tracking, as well as totrack motionof rigid planar patches present in the scene. We describe howour tracking algorithm allows an EyeTap toalter thelight from aparticular portionof the scene togive rise toa computer-controlled, selec- tively mediated reality. Animportant difference between mediated reality and aug- mented reality includes theability tonot just augment butalso deliberately diminish or otherwise alter thevisual perception ofreality. For example, diminished reality allows additional information tobe inserted withoutcausing theuser toexperience information overload. Our tracking algorithm also takes intoaccount the effects of automatic gain control,by performing motionestimation in bothspatial as well as tonal motioncoordinates. 1Introduction IvanSutherland, apioneer in the eld ofcomputergraphics, described a head-mounted display with half-silvered mirrors so that the wearer could see a virtual world superimposed on reality (Earnshaw, Gigante,& Jones,1993; Sutherland, 1968), giving rise to augmentedreality (AR). Others have adopted Sutherland’s concept ofa head-mounted display (HMD) but generally without the see-through capability. An articial environ- Presence, Vol. 11,No. 2, April2002, 158–175 mentin which the user cannot see through the display is generally referred as a © 2002by theMassachusetts Institute of Technology virtual reality (VR) environment. Oneof the reasons that Sutherland’s ap- 158 PRESENCE: VOLUME11, NUMBER 2 Mannand Fung 159 proach was not more ubiquitously adopted is that he did not merge the virtual object (a simple cube) with the real world in ameaningful way. Feiner ’s group was responsible for demonstrating the viability ofAR as a eld ofresearch, using sonar (Logitech 3-D trackers) to trackthe real world so that the real and virtual worlds could be registered (Feiner, MacIntyre,& Seligmann, 1993a,1993b). Other research groups (Fuchs, Bajura, &Ohbuchi;Caudell &Mizell, 1992) also contributed Figure 1. (a)The wearable computercan be used like clothing to to this development. Someresearch in ARalso arises encapsulate the user and function as aprotective shell, whether to fromwork in telepresence (Drascic &Milgram,1996). protectus from cold or physical attack(as traditionally facilitated by However, the concept ofthe Reality Mediator, which armor),or toprovide privacy (byconcealing personal information and arises fromthe eld ofhumanistic intelligence (HI) personal attributesfrom others). Interms of signal ow, this (Mann,1997a, 2001a, 2001b) differs fromaugmented encapsulation facilitates the possible mediation of incoming reality, which has its origins in the eld ofvirtual reality. information topermitsolitude and the possible mediation of outgoing HI is dened as intelligence that arises fromthe human information topermitprivacy. Itis not somuchthe absolute blocking being in the feedbackloop ofa computational process of these information channels thatis important;it is the factthat the in which the humanand computerare inextricably in- wearer can control towhat extent, and when, these channels are tertwined. Wearable computinghas emerged as the per- blocked, modied, attenuated, or amplied, in various degrees, that fecttool for embodying HI.When awearable computer makes wearable computingmuch more empowering to the user than functions as asuccessful embodiment ofHI,the com- other similar formsof portable computing.(b) An equivalent depiction of encapsulation (mediation) redrawn where the encapsulation is puter uses the human ’smind and body asone ofits pe- understood tocomprise aseparate protective shell. ripherals, just as the humanuses the computeras ape- ripheral. This reciprocal relationship, in which each uses the other in its feedbackloop, is at the heart ofHI. Within an HIframework,the wearable computeris 2EyeTap Devices worn constantly to assist the user in avariety ofday-to- day situations. Just as half-silvered mirrors are used to create aug- Animportant observation arising fromthis constant mented reality, EyeTap devices are used tomediate use is that,unlike handheld devices, laptop computers, one’sperception ofreality. EyeTap devices have three and PDAs, the wearable computercan encapsulate us maincomponents: (Mann,1998). Itcan function asaninformation lter and allow ustoblockout material we mightnot wish to c ameasurement system typically consisting ofa cam- experience (such as offensive advertising) orsimply re- era system,or sensor array with appropriate optics; place existing media with different media.Thus, the c adiverter system,for diverting eyeward bound light wearable computeracts to mediate one ’sexperience into the measurement system and therefore causing with the world. The mediating role ofEyeTap and the eye ofthe user ofthe device to behave, in effect, wearable computers can be better understood by exam- asif it were acamera;and ining the signal ow paths between the human,com- c anaremacfor reconstructing atleast some ofthe puter,and external world as illustrated in gure 1. diverted rays ofeyeward bound light. (Thus, the There exist well known email and Web browser lters aremacdoes the opposite ofwhat the cameradoes that replace orremove unwanted advertising, mediating and is, in manyways, acamerain reverse. The ety- one’suse ofthe media.Diminished reality extends this mology ofthe word aremac itself arises fromspell- mediation to the visual domain. ing the word camera backwards (Mann,1997c).) 160 PRESENCE: VOLUME11, NUMBER 2 Anumber ofsuch EyeTap devices, together with Inthe focus-tracking embodiments,the aremachas wearable computers,were designed, built, and worn by focus linked to the measurement system (for example, the authors for manyyears in awide variety ofsettings “camera”)focus,so that objects seen depicted on the and situations, both inside the lab as well as in ordinary aremacof the device appear to be at the samedistance day-to-day life (such as while shopping,riding abicycle, fromthe user ofthe device asthe real objects so de- going through airport customs,attending weddings, picted.In manualfocus systems, the user ofthe device is and so on). This broad base ofpractical real-life experi- given afocus control that simultaneously adjusts both ence helped usbetter understand the fundamentalissues the aremacfocus and the “camera” focus.In automatic ofmediated reality. focus embodiments,the camerafocus also controls the Although the apparatus is for altering our vision, in aremacfocus. Such a linked focus gives rise to amore mostof the practical embodiments that we built we pro- natural view nder experience. Itreduces eyestrain as vided atleast one modeof operation that can preserve well. Reduced eyestrain is important because these de- our vision unaltered. This one mode,which we call the vices are intended tobe worn continually. “identity mode, ” serves as abaseline that formsa point The operation ofthe depth trackingaremac is shown ofdeparture for when certain changes are desired. To in gure 2. achieve the identity moderequirement, the EyeTap Because the eye ’s own lens (L3)experiences what it mustsatisfy three criterion: would have experienced in the absence ofthe apparatus, the apparatus,in effect,taps in toand out ofthe eye, c focus: The subject matter viewed through the eyetap mustbe displayed at the appropriate depth offocus. causing the eye to becomeboth the cameraand the viewnder (display). Therefore, the device is called an c orthospatiality: The rays oflight created by the aremacmust be collinear with the rays oflight en- EyeTap device. tering the EyeTap,such that the scene viewed Often, lens L1 is avarifocal lens or otherwise has a through the EyeTap appears the sameas if viewed variable eld ofview (such as a “zoom” functionality). in the absence ofthe EyeTap. Inthis case,it is desired that the aremacalso have avari- c orthotonality: Inaddition to preserving the spatial able eld ofview. Inparticular, eld-of-view control light relationship oflight entering the eye, we de- mechanisms (whether mechanical,electronic, orhybrid) sire that the EyeTap device also preserves the tonal are linked in such away that the aremacimage magni - relationships oflight entering the eye. cation is reduced as the cameramagni cation is in- creased. Through this appropriate linkage,any increase in magnication by the camerais negated exactly by 2.1 Focus