US 2016.0070439A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2016/0070439 A1 Bostick et al. (43) Pub. Date: Mar. 10, 2016 (54) ELECTRONIC COMMERCEUSING G06F 3/0 (2006.01) AUGMENTED REALITY GLASSES ANDA G06F 3/16 (2006.01) SMART WATCH (52) U.S. Cl. CPC ............ G06F 3/04842 (2013.01); G06F 3/017 (71) Applicant: International Business Machines (2013.01); G06F 3/013 (2013.01); G06F 3/167 Corporation, Armonk, NY (US) (2013.01); G06T 19/006 (2013.01); G06Q 30/0641 (2013.01) (72) Inventors: James E. Bostick, Cedar Park, TX (US); John M. Ganci, JR. Cary, NC (US); Sarbajit K. Rakshit, Kolkata (IN); (57) ABSTRACT Craig M.Trim, Sylmar, CA (US) In an approach for electronic commerce using augmented reality glasses and a Smart watch, a computer receives a (21) Appl. No.: 14/477,127 configuration associating a user gesture to a command. The computer determines whethera user of the augmented reality (22) Filed: Sep. 4, 2014 glasses selects an object in a first electronic commerce envi ronment and, responsive to determining the user selects an Publication Classification object, the computer determines whether the user performs a (51) Int. C. first gesture detectable by a Smart watch. The computer, then, G06F 3/0484 (2006.01) determines whether the first gesture matches the user gesture G06O 30/06 (2006.01) and, responsive to determining the first gesture matches the G06T 9/00 (2006.01) usergesture, the computer performs the associated command. -1 200 RECEIVESA CONFIGURATION OFA COMMAND ASSOCATED WITHAUSERGESTURE DETERMINES WHETHER AN OBJECT ISSELECTED RECEIVESA COMMAND BASED ONADETECTED USERGESTURE 216 DETERMINES WHETHER THE COMMAND EXECUTES THE DETERMINED RECEIVED PROCEEDS TO A SHOPPING CART COMMAND PROCEEDS TO SHOPPING CART Patent Application Publication Mar. 10, 2016 Sheet 1 of 3 US 2016/0070439 A1 AR GLASSES 12O 100 NN APPLICATIONeCOMMERCE eCOMMERCE DATABASE USER INTERFACE SENSOR 132 133 BAR CODE SCANNER 130 SMART WATCH SERVER 140 3 D DATABASE FIG. 1 Patent Application Publication Mar. 10, 2016 Sheet 2 of 3 US 2016/0070439 A1 -1 200 RECEIVESA CONFIGURATION OF A COMMAND ASSOCIATED WITHAUSERGESTURE 202 204 DETERMINES WHETHER AN OBJECT ISSELECTED DETERMINES THE OBJECT STORES THE DATAVIEWED BY THE USER RECEIVESA COMMAND BASED ONADETECTED USERGESTURE 216 DETERMINES WHETHER THE COMMAND EXECUTES THE RECEIVED PROCEEDS TO A DETERMINED SHOPPING CART COMMAND PROCEEDS TO SHOPPING CART FIG. 2 Patent Application Publication Mar. 10, 2016 Sheet 3 of 3 US 2016/0070439 A1 MEMORY C C 304 PERSISTENT STORAGE PROCESSOR(S) 30 320 IO DISPLAY INTERFACE(S) COMMUNICATIONS UNIT 318 EXTERNAL DEVICE(S) FIG. 3 US 2016/0070439 A1 Mar. 10, 2016 ELECTRONIC COMMERCEUSING 0007 FIG.3 depicts a block diagram of components of the AUGMENTED REALITY GLASSES ANDA augmented reality glasses executing the electronic commerce SMART WATCH application, in accordance with an embodiment of the present invention. BACKGROUND OF THE INVENTION DETAILED DESCRIPTION 0001. The present invention relates generally to the field of 0008 Embodiments of the present invention recognize augmented reality glasses, and more particularly to the use of that several electronic commerce (E-commerce) applications augmented reality glasses and a Smart watch for electronic for augmented reality glasses (AR glasses) have been devel COCC. oped using tactile and audio commands. Touch screens, in 0002 Augmented reality (AR) is a live, director indirect, Smartphones or touch sensors on the AR glasses, may be used view of a physical, real-world environment whose elements in conjunction to or as an alternative to audio sensors and are augmented by computer-generated sensory input Such as speech recognition to command AR glasses. Embodiments of visually perceivable content, including graphics, text, video, the present invention utilize gaze focal point detection to global position satellite (GPS) data or sound. Augmentation is identify an object by identifying a focal point in the user's conventionally in real-time and in semantic context with envi field of vision. Furthermore, embodiments of the invention ronmental elements, for example, the addition of current, use a Smart watch or other wearable computing device with real-time sports scores to a non-related news feed. Advanced one or more sensors which can detect one or more muscle augmentation Such as the use of computer vision, speech movements for a gesture such as a finger motion or a hand recognition and object recognition allows information about gesture. The Smart watch sends sensor data for a detected the Surrounding real-world to be interactive and manipulated gesture to AR glasses. The sensor data may include detected digitally. In many cases, information about the environment is muscle movement data foragesture. The gesture correlated to visually overlaid on the images of the perceived real-world. the sensor data of the muscle movements received by AR 0003. Some augmented reality devices, rely, at least in glasses may be configured to correspond to a user command. part, on a head-mounted display, for example, with sensors For example, a gesture associated with sensor data for one or for Sound recognition. An example of existing head-mounted more muscle movements may be configured to select an display technology or augmented reality glasses (AR glasses) object or product. uses transparent glasses which may include an electro-optic 0009 Embodiments of the invention provide a capability device and a pair of transparent lenses, which display infor to identify a selected object or product in an augmented mation or images displayed over a portion of a user's visual reality view, Such as an internet site or an on-line store data field while allowing the user to perceive the real-world. The base viewed using AR glasses. Embodiments of the present displayed information and/or images can provide Supplemen invention provide the ability to view or scan barcode data of tal information about a user's environment and objects in the a product in a real world environment Such as a brick and user's environment, in addition to the user's visual and audio mortar store. Additionally, embodiments of the present inven perception of the real-world. tion provide the ability to capture an image of an object in a real world environment such as a brick and mortar store for possible selection, identification, shopping cart addition, and SUMMARY other object related actions. Furthermore, embodiments of the present invention provide the capability to search product 0004. According to aspects of the present invention a data, to search product attributes, to search multiple websites, method, a computer product, and a computer system are local or on-line databases and real world environments, to disclosed for electronic commerce using augmented reality select an object or product, to move an object or product to an glasses and a Smart watch. A computer receives a configura on-line or augmented reality shopping cart for purchase and tion associating a user gesture to a command. The computer to store and retrieve selected products and search results using determines whether a user of the augmented reality glasses AR glasses and a Smart watch. Additionally, embodiments of selects an object in a first electronic commerce environment the present invention provide a memory management func and, responsive to determining the user selects an object, the tion for recall of data on previously viewed or searched computer determines whether the userperforms a first gesture objects or products such as product images, product identifi detectable by a smart watch. The computer, then, determines cation, product attributes, product type and product location. whether the first gesture matches the usergesture and, respon 0010. The present invention will now be described in sive to determining the first gesture matches the user gesture, detail with reference to the Figures. FIG. 1 is a functional the computer performs the associated command. block diagram illustrating an augmented reality data process ing environment, generally designated 100, in accordance BRIEF DESCRIPTION OF THE DRAWINGS with one embodiment of the present invention. FIG. 1 pro vides only an illustration of one implementation of the 0005 FIG. 1 is a functional block diagram illustrating an present invention and does not imply any limitations with augmented reality data processing environment, in accor regard to the environment in which different embodiments dance with an embodiment of the present invention; may be implemented. Modifications to the depicted environ 0006 FIG. 2 is a flowchart depicting operational steps of ment may be made by those skilled in the art without depart an electronic commerce application on augmented reality ing from the scope of the invention as recited by the claims. glasses for electronic commerce using augmented reality 0011 Augmented reality data processing environment glasses and a Smart watch within the augmented reality data 100 includes augmented reality glasses (AR glasses) 120, processing environment of FIG. 1, in accordance with an Smart watch 130 and server 140 all connected over network embodiment of the present invention; and 110. Network 110 can be, for example, a telecommunications US 2016/0070439 A1 Mar. 10, 2016 network, a local area network (LAN), a wide area network addition to digital image projection to the user in AR glasses (WAN), a virtual local area network (VLAN) such as the 120, creating the augmented reality standard in augmented Internet, or any combination of the three, and can include reality device technology. AR glasses 120 may be capable of wired, wireless, or fiber optic connections. In general, net sending and receiving data Such as sensor data from Smart work 110 can be any combination of connections and proto watch 130 via network 110. AR glasses 120 include E-com cols that will Support communications between AR glasses merce application 121, E-commerce database 125, and user 120, Smart watch 130 and server 140.