US 2014O184496A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2014/0184496 A1 Gribetz et al. (43) Pub. Date: Jul. 3, 2014 (54) EXTRAMISSIVESPATIAL IMAGING Related U.S. Application Data DIGITAL EYE GLASS APPARATUSES, (60) Provisional application No. 61/748,468, filed on Jan. Y.SR RS.SQRE OR 3, 2013, provisional application No. 61/916,773, filed 9 9 CREATION, OR INTERACTION WITH on Dec. 16, 2013. OBJECTS, MATERIALS, OR OTHER Publication Classification ENTITIES (51) Int. Cl. (71) Applicant: Msta Company, Portola Valley, CA G06F 3/0 (2006.01) (US) (52) U.S. Cl. (72) Inventors: Meron Gribetz, New York, NY (US); Sc - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - G06F 3/07 E. W. Steve G. Mann, Toronto (CA) '''' “ (57) ABSTRACT (73) Assignee: Meta Company, Portola Valley, CA (US) A sensing and display apparatus, comprising: a first phenom enon interface configured to operatively interface with a first augmediated-reality space, and a second phenomenon inter (21) Appl. No.: 14/147,199 face configured to operatively interface with a second aug mediated-reality space, is implemented as an extramissive (22) Filed: Jan. 3, 2014 spatial imaging digital eyeglass. 18OW 181W - 180DEG L-4 a? a Lu-n 181DEG 180U 1 181U 18OMS 18OAM Patent Application Publication Jul. 3, 2014 Sheet 1 of 35 US 2014/O184496 A1 1300 1993 IEEE| 1995, 1903 h-962 1300 Patent Application Publication Jul. 3, 2014 Sheet 2 of 35 US 2014/O184496 A1 41 O FIG. 1AA Patent Application Publication Jul. 3, 2014 Sheet 3 of 35 US 2014/O184496 A1 1300 GO) () () () 9 1 O 12 1. () () 910 (1 1. 916 1300 Patent Application Publication Jul. 3, 2014 Sheet 4 of 35 US 2014/O184496 A1 1300 1300 Patent Application Publication Jul. 3, 2014 Sheet 5 of 35 US 2014/O184496 A1 s 3 5 e 3 Patent Application Publication Jul. 3, 2014 Sheet 6 of 35 US 2014/O184496 A1 1EE1 O 1EE20 -1EE25 VISUAL ACOUSTIC 1EE30 TRUSION TRUSION 1EE35 SENSOR SENSOR 1EE40 1EE45 O C C C C C 1EE5O &ge e sess 1EE7O OOOOOOONSSESs earé2 1EE8O 1EE90 EE9 NTRUS9N || TOUC EXTRUSION DETECTOR | | DETECTOR | | DETECTOR 1EE93 1EE94 GESTURE 1EE96 1EE97 SENSOR 1EE98 FIG. EE Patent Application Publication Jul. 3, 2014 Sheet 7 of 35 US 2014/O184496 A1 96|| ||ÑSOEOCH||8 Patent Application Publication Jul. 3, 2014 Sheet 8 of 35 US 2014/O184496 A1 Patent Application Publication Jul. 3, 2014 Sheet 9 of 35 US 2014/O184496 A1 A BODY PART 1 C 10 EXTREMITY IS N PLANET TERMINATE MANPLAY 1C40 C20 MANPLAY TM DISPLAYS POINT(S) ONSURFACE THAT 1C30 TRACK(S) PART POINT REMAINS IN PLANET Patent Application Publication US 2014/O184496 A1 09||ZOZOOZ07@07| Patent Application Publication Jul. 3, 2014 Sheet 11 of 35 US 2014/O184496 A1 G09Z Patent Application Publication Jul. 3, 2014 Sheet 12 of 35 US 2014/O184496 A1 ~?~p dS Patent Application Publication Jul. 3, 2014 Sheet 13 of 35 US 2014/O184496 A1 ... R %22 2 W V ' ' , , Sf HIll MW\ -- -s W N ill......! J iiiiii Patent Application Publication Jul. 3, 2014 Sheet 14 of 35 US 2014/O184496 A1 w O CO us 1. & INIII-IIIllW s it." Hill --YsNsi-iss ---------------- -3.---3.NSas 3. Patent Application Publication Jul. 3, 2014 Sheet 15 of 35 US 2014/O184496 A1 L11 L21 L31 ty 2.É 2F64 -1 s N-NN sers FIG. 2F Patent Application Publication Jul. 3, 2014 Sheet 16 of 35 US 2014/O184496 A1 0919, Patent Application Publication Jul. 3, 2014 Sheet 17 of 35 US 2014/O184496 A1 e S g ? r 3 S3 8 y c5 TNa 9 y O) y S. CN O Cro L?) CY) O) y CY) O r CD Cro CO O) Y- L O CO OO Y O LO OO CY) so SO Y p Y N 8 N - , Cro OO GN O) CY)31 Y y 3 o s Patent Application Publication Jul. 3, 2014 Sheet 18 of 35 US 2014/O184496 A1 28|| Patent Application Publication Jul. 3, 2014 Sheet 19 of 35 US 2014/O184496 A1 530 560 520 550 510 540 500 531 56 521 551 511 541 5O1 571 573 575 570 572 574 FIG. 5 Patent Application Publication Jul. 3, 2014 Sheet 20 of 35 US 2014/O184496 A1 500 510 FIG. 5A Patent Application Publication Jul. 3, 2014 Sheet 21 of 35 US 2014/O184496 A1 Patent Application Publication Jul. 3, 2014 Sheet 22 of 35 US 2014/O184496 A1 o s: s Patent Application Publication Jul. 3, 2014 Sheet 23 of 35 US 2014/O184496 A1 800 804 805 806 807 803 FIG. 8A Patent Application Publication Jul. 3, 2014 Sheet 24 of 35 US 2014/O184496 A1 N/- Patent Application Publication Jul. 3, 2014 Sheet 25 of 35 US 2014/O184496 A1 V 801 FIG. 8C Patent Application Publication Jul. 3, 2014 Sheet 26 of 35 US 2014/O184496 A1 O 802 ( is p 803 FIG. 9A Patent Application Publication Jul. 3, 2014 Sheet 27 of 35 US 2014/O184496 A1 8OO 802 t Patent Application Publication Jul. 3, 2014 Sheet 28 of 35 US 2014/O184496 A1 FIG. 10A Patent Application Publication Jul. 3, 2014 Sheet 29 of 35 US 2014/O184496 A1 802 t 804 FIG. OB Patent Application Publication Jul. 3, 2014 Sheet 30 of 35 US 2014/O184496 A1 800 808 FIG. 11A Patent Application Publication Jul. 3, 2014 Sheet 31 of 35 US 2014/O184496 A1 FIG. 11B Patent Application Publication Jul. 3, 2014 Sheet 32 of 35 US 2014/O184496 A1 FIG. 11C Patent Application Publication Jul. 3, 2014 Sheet 33 of 35 US 2014/O184496 A1 O 802 t Patent Application Publication Jul. 3, 2014 Sheet 34 of 35 US 2014/O184496 A1 FIG. 13 Patent Application Publication Jul. 3, 2014 Sheet 35 of 35 US 2014/O184496 A1 1300 FIG. 14 US 2014/0184496 A1 Jul. 3, 2014 EXTRAMISSIVE SPATIAL MAGING the second space; a processing apparatus operatively coupled DIGITAL EYE GLASS APPARATUSES, to the interface assembly, and the processing apparatus con METHODS AND SYSTEMS FOR VIRTUAL OR figured to process the sensor signals and the effector signals AUGMEDIATED VISION, MANIPULATION, conveyed by the interface assembly; and a memory assembly CREATION, OR INTERACTION WITH configured to tangibly embody a processing program, includ OBJECTS, MATERIALS, OR OTHER ing a sequence of programmed instructions configured to ENTITIES direct the processing apparatus to execute operations on the sensor signals and the effector signals. PRIORITY CLAIMS AND CROSS-REFERENCE 10007. In accordance with an aspect, there is provided a TO RELATED APPLICATIONS user interface, including: a first interface section configured to display phenomena derived from a first space; and a second 0001. This application is related to and claims domestic interface section configured to display phenomena derived priority benefits under 35 USC S119(e) from U.S. Provisional from a second space. Patent Application Ser. No. 61/748,468 filed on 3 Jan. 2013, 0008. In accordance with an aspect, there is provided a and from U.S. Provisional Patent Application Ser. No. sensing and display apparatus, including: a first phenomenon 61/916,773 filed Dec. 16, 2013, the entire contents of the interface configured to operatively interface with a phenom aforementioned applications are expressly and hereby incor enon and/or phenomena in a first space; and a second phe porated hereinto by reference. nomenon interface configured to operatively interface with a phenomenon and/or phenomena in a second space. BACKGROUND 0009. In accordance with an aspect, there are provided 0002 Sensor technology is employed in contexts to moni other aspects as identified in the claims. tor and/or measure information about an environment, such I0010. Other aspects and features of the non-limiting as the use of temperature sensors in industrial processes. embodiments (examples, options, etc.) may now become Mobile devices employ increasingly powerful processing apparent upon review of the following detailed description of capabilities to support various applications, such as telephony the non-limiting embodiments with the accompanying draw and other mobile communications. 1ngS. SUMMARY DETAILED DESCRIPTION OF THE DRAWINGS 0003. In accordance with an aspect, there is provided a 0011. The non-limiting embodiments may be more fully sensing and display apparatus, including: an interface assem appreciated by reference to the following description of the bly including: a first interface module configured to interface non-limiting embodiments when taken in conjunction with with a first computationally mediated space (for example, a the accompanying drawings, in which: scene, subject matter, or the like), the first interface module I0012 FIG. 1A depicts a schematic example of a sensor configured to exchange sensor signals and effector signals and display apparatus 1300 in one embodiment: with the first space; and a second interface module configured I0013 FIG. 1AA depicts a schematic example of a method to interface with a second computationally mediated space associated with the sensory and display apparatus 1300 of (for example, a user's own personal viewing space in or FIG. 1A in one embodiment; around their digital eye glass space), the second interface I0014 FIG. 1B depicts a schematic example of a sensory module configured to provide or exchange effector signals, and display apparatus 1300 in one embodiment; and, in Some embodiments, also sensor signals (for example, I0015 FIG. 1C depicts a schematic example of a sensory an eye-tracker, or the like) with the second space, and the and display apparatus 1300 in one embodiment; effector signals being user presentable, at least in part, in any I0016 FIG. 1D depicts a schematic example of a sensory one of the first space and the second space.