USOO8854.361B1
(12) United States Patent (10) Patent No.: US 8,854.361 B1 Smith (45) Date of Patent: Oct. 7, 2014
(54) VISUALLY AUGMENTING A GRAPHICAL 7,676.499 B2 3/2010 Dorsett, Jr. RENDERING OF ACHEMICAL STRUCTURE 7,707,206 B2 4/2010 Encina et al. 7,805.437 B1 9/2010 Andersson et al. REPRESENTATION OR BOLOGICAL 8.433,723 B2 4/2013 Smith et al. SEQUENCE REPRESENTATION WITH MULT-DIMIENSIONAL INFORMATION (Continued) (71) Applicant: CambridgeSoft Corporation, Waltham, FOREIGN PATENT DOCUMENTS MA (US) EP 15264.71 A1 4/2005 GB 249.3830 A 2, 2013 (72) Inventor: Robin Y. Smith, Boston, MA (US) (Continued) (73) Assignee: CambridgeSoft Corporation, Waltham, OTHER PUBLICATIONS MA (US) Ben Shine and Dana Tenneson, “ChemPad3—ATutorial”. May 21, (*) Notice: Subject to any disclaimer, the term of this 2008.* patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days. (Continued) (21) Appl. No.: 14/058,865 Primary Examiner — Kee MTung Assistant Examiner — Sing-Wai Wu (22) Filed: Oct. 21, 2013 (74) Attorney, Agent, or Firm — Choate, Hall & Stewart LLP Related U.S. Application Data (57) ABSTRACT (60) Provisional application No. 61/779,607, filed on Mar. 13, 2013. In certain embodiments, the invention relates to systems, methods, and apparatus that allow a user to visually augment (51) Int. Cl. a graphical rendering of either a chemical structure represen G06T I5/00 (2011.01) tation or a biological sequence representation with multi G09G 5/00 (2006.01) dimensional information. A user captures a video image using G06T 9/00 (2011.01) a computing device such as a hand-held Smartphone, com (52) U.S. Cl. puterized eye glasses or tablet computer. The video image CPC ...... G06T 19/006 (2013.01) includes information regarding at least one of a chemical USPC ...... 345/419; 34.5/629 structure and a biological sequence. A processor identifies, (58) Field of Classification Search within the video image, a graphical representation of at least CPC ...... G06T 7/0012: G06T 19/00; G06T 11/20; one of a chemical structure and a biological structure. The G06T 2207/10016: G06T 2210/41 processor augments the graphical representation with addi See application file for complete search history. tional information and provides the video data for presenta tion upon a display controlled by the computing device. The (56) References Cited computing device presents the video data in Substantially real time in relation to the capture of the video data by the com U.S. PATENT DOCUMENTS puting device. 7,250,950 B2 7/2007 Smith et al. 7,650,327 B2 1/2010 Remsen et al. 21 Claims, 6 Drawing Sheets
Receive Wided Caia Destinecif) o Presentation upon Jser Device at
ldentify a Graphical Representation within the video image 2.
Match Augmentation data for Wisually Augmenting the vided image 2
2it. swice Orieniaic Received?
Yes Adjust visual Representation of Portion of Augmentation data Responsive to evice Orientation
Apply Partion of Augmentation Data to Video Satarioi to Pressetiatics
s Uponiser evice 21:
Orientatio s ghargerecisived? User input received? es 2: p- icientify Follow-on Augmentation data for Alignmenting the video Data ; US 8,854.361 B1 Page 2
(56) References Cited Gonzalez-Villanueva et al., WallShare: A Collaborative Multi pointer System for Portable Devices, Nov. 19, 2012, 7 pages. U.S. PATENT DOCUMENTS International Search Report for PCT Application No. PCT/US2011/ 035070, mailed Oct. 6, 2011, 4 pages. 8,538,983 B2 9/2013 Huang et al. International Search Report, PCT/US2012/026574, dated Mar. 20, 2002fO161599 A1 10/2002 Faerman et al. 2013, 4pgs. 2004.0003000 A1 1/2004 Smith et al. iTunes Preview, Flick for iPhone, iPad, and iPod touch on the iTunes 2004/OOO6742 A1 1/2004 Slocombe App Store,
U.S. Patent Oct. 7, 2014 Sheet 2 of 6 US 8,854.361 B1
o Reseive Wide a 3 assic ir Presentation. Upon User Device 22
identify a Graphica Represeitation withii the Wideo image 34
viatoh Augieriatior Data for Visuay Augmenting the video image 26
evice Creigii R3Ceives?
Adjust Visual Representation of Portion of Augmentation Data Responsive to Davis Orientati 22
Apply Portion of Algientatio; Data to Video 333 is to P8ssigtin upon Jser Device f
{risistic User input Received Change Received? 22
identify Follow-On Augmentation Data for Augmenting the Video Data
... U.S. Patent Oct. 7, 2014 Sheet 3 of 6 US 8,854.361 B1
?zyswogaeson NQ@isensadosa U.S. Patent Oct. 7, 2014 Sheet 4 of 6 US 8,854.361 B1
o Receive Wide Daia estine if Presentation jpon Ser Device 2
identify a Graphica Representation within the Video image 44
identify Augmentation Features for Adding information to the video Data a
Provide Video image with One or more Controls for Selecting the Augmentation Features for Presentation pon iser Device 48
Featre Seiected
aii Yes Augment Video ata with Feature formatic f
identify One or more Sui-Features of the Selected Featie 44
Presert Wicie? aia with Feature formatici and One of More Controis for Selecting the Sub-Feature(s) As U.S. Patent US 8,854.361 B1
§§§
U.S. Patent Oct. 7, 2014 Sheet 6 of 6 US 8,854.361 B1
US 8,854,361 B1 1. 2 VISUALLY AUGMENTING A GRAPHICAL In an example embodiment, the computing device used to RENDERING OF ACHEMICAL STRUCTURE capture the video image accesses software configured to iden REPRESENTATION OR BOLOGICAL tify the captured chemical structure or biological sequence. SEQUENCE REPRESENTATION WITH The software is further configured to augment the video MULT-DIMIENSIONAL INFORMATION 5 image, prior to being displayed to the user (e.g., in a display area of the computing device or on a separate display), with RELATED APPLICATIONS information regarding the captured chemical structure or bio logical sequence. The information, in Some examples, can The present application claims priority to and the benefit include the chemical name, biological species name, proper of, U.S. Provisional Application No. 61/779,607 entitled 10 ties of the chemical compound or biological species, whether "Visually Augmenting a Graphical Rendering of a Chemical the identified chemical compound or biological species is available in a given lab or stock room, and (if identified in a Structure Representation or Biological Sequence Represen lab or stock room) a storage location of the identified chemi tation with Multi-Dimensional Information filed Mar. 13, cal compound or biological species. In another example, the 2013, the contents of which is hereby incorporated by refer 15 information can include a three-dimensional representation ence in its entirety. of the chemical structure or biological sequence. The display containing the augmented video image, in a BACKGROUND preferred embodiment, is presented in Substantially real-time, Such that the video is captured, augmented with additional Chemical structure rendering software is widely used by 20 information pertaining to a chemical structure or biological research and educational institutions to depict chemical struc sequence rendered therein, and displayed to the user while the tures and chemical reactions of interest. Unlike chemical user continues to capture the video image containing the formulas or chemical names, structural formulas provide a image of the chemical structure or biological sequence. The graphical representation of the molecular structure. A graphi additional information, for example, may be positionally cal chemical structure representation is capable of indicating 25 linked to the captured image of the chemical structure or the arrangements of atoms in a way that a chemical formula biological sequence (or other graphical representation) being Cannot. viewed. For example, the additional information may be Biological sequence and large molecule rendering soft Superimposed on, or placed in proximity to, the video image ware is widely used by research and educational institutions of a two-dimensional graphical chemical structure captured to depict biological sequences, including nucleotide and 30 within the video image. amino acid sequences of interest. A three-dimensional Call-out boxes may be displayed on the screen prompting graphical representation can be extrapolated from a nucle the user to choose to display particular additional information otide and/or amino acid sequence to depict the arrangements Such as, in Some examples, the name, storage location, and/or of atoms, nucleotides, and/or amino acid residues in a way properties of the identified chemical structure or biological that a simple sequence (e.g., a nucleotide and/or amino acid 35 species. The call-out boxes, for example, may be Superim sequence) cannot. posed on the live video image being viewed such that it is Augmented reality refers to augmenting the real world clear to the user what portion of the captured video image has environment with computer-generated input such as graph been identified by the software as a chemical structure or ics, video, or audio information. In handheld device aug biological sequence. For example, the additional information mented reality, digital objects can be overlaid on real world 40 may be displayed (i) on top of the video image of an identified Video data captured on the handheld computing device. graphical rendering of a chemical structure or biological Examples of handheld device augmented reality include the sequence, (ii) in a call-out box pointing to the identified LayarTM print media augmentation application by Layar of graphical rendering of the chemical structure or biological Amsterdam, the Netherlands, the Word Lens augmented real sequence, or (iii) Superimposed on top of (e.g., at least par ity translation application by Quest Visual of San Francisco, 45 tially overlapping) the identified graphical rendering of the Calif., and the WikitudeR Augmented Reality mobile plat chemical structure or biological sequence. The additional form by Wikitude GmbH of Salzburg, Austria. information, in Some embodiments, is rendered partially transparent Such that the captured video image is visible SUMMARY OF THE INVENTION beneath the additional information. The additional informa 50 tion, in this circumstance, may optionally be rendered opaque Described herein are various embodiments of systems, responsive to selection by a user (e.g., receiving a touch input methods, and apparatus that allow a user to visually augment corresponding to a transparent call-out box rendered upon a a graphical rendering of either a chemical structure represen touch screen device, etc.). If multiple chemical structure rep tation or a biological sequence representation with multi resentations and/or biological sequence representations are dimensional information. Initially, using a computing device 55 identified, the position of the additional information (or dis Such as a hand-held Smartphone, computerized eyeglasses or played information prompts), in some embodiments, indi tablet computer, a user captures a video image containing cates which identified chemical structure or biological information regarding at least one of a chemical structure and sequence correlates to which additional information. a biological sequence. In one example, the video image may A three-dimensional representation of the identified capture an illustration of a chemical structure, such as a 60 chemical structure or biological sequence may be added to two-dimensional rendering of a chemical structure drawn on the video image prior to display. The position of the three a piece of paper, white board, chalk board, or transparent dimensional representation may correspond to the position of drawing board as in a laboratory. Instead of a graphical ren the identified chemical structure or biological sequence (e.g., dering of a chemical structure, in another example, the video it may be Superimposed or partially Superimposed). In some image may capture a written chemical name, biological spe- 65 embodiments, the user may be provided the opportunity to cies name, formula, or other visual indication of a chemical interact with the three-dimensional representation. For compound or biological species. example, the user may be able to rotate, resize, and/or relocate US 8,854,361 B1 3 4 an added three-dimensional representation of the captured information to the graphical representation includes compar chemical structure or biological sequence as rendered upon ing the graphical representation to one or more stored graphi the display. The three-dimensional representation may be cal representations. In some embodiments, matching the rotated, for example, as a function of the position of the additional information to the graphical representation mobile device (e.g., tilting the device results in tilting the includes digitally refining the portion of the graphical repre three-dimensional representation). The three-dimensional sentation prior to comparing the graphical representation to representation, in some embodiments, rotates automatically the one or more stored graphical representations. (e.g., slowly spinning and/or tilting upon the display). In some embodiments, the method also includes receiving, In various embodiments, the systems, methods, and appa by the processor, an indication of selection of a first control of ratus utilize or include a tablet computer, a mobile phone 10 the one or more controls; and augmenting, by the processor, device, an augmented reality wearable computer, or any other subsequent video data with the portion of the additional infor computer device or system capable of capturing and present mation related to the first control. In some embodiments, ing video data. In further embodiments, the systems, meth augmenting the Subsequent video data includes adding audio ods, and apparatus utilize or include a laptop computer, desk data to the video data, wherein the audio data includes a top computer, notebook computer or other computer device 15 verbal description of the additional information. or system interfacing with a device capable of capturing and Another aspect of the invention relates to a processor and a presenting video data (e.g., through a wired or wireless con memory having instructions stored thereon. The instructions, nection). The systems, methods, and apparatus have applica when executed by the processor, cause the processor to tions in a wide variety of industries and environments that receive video data captured by a computing device, wherein work with graphical representations of chemical structural the video data includes a video image; identify, within the formulas, such as laboratories, research facilities, and class Video image, a visual identification of a chemical structure or room environments. a biological sequence; match additional information to the One aspect of the invention relates to a method including visual identification. The additional information includes at receiving video data captured by a first computing device, least one of (i) a graphical representation of the chemical wherein the video data includes a video image. The method 25 structure or biological sequence, (ii) a name of a) a chemical includes identifying, within the video image, by a processor compound represented by the chemical structure, or b) a of a second computing device, a graphical representation of biological species comprising a DNA molecule, an RNA one of a chemical structure and a biological sequence. The molecule, or a polypeptide represented by the biological method also includes matching, by the processor, additional sequence, and (iii) a plurality of properties of the chemical information to the graphical representation. The additional 30 compound or biological species. The instructions, when information includes at least one of (i) a three-dimensional executed by the processor, further cause the processor to representation of the chemical structure or biological augment the video image with at least one of (i) at least a sequence, (ii) a name of a) a chemical compound represented portion of the additional information, and (ii) one or more by the chemical structure, or b) a biological species compris controls configured, upon selection, to present at least a por ing a DNA molecule, an RNA molecule, or a polypeptide 35 tion of the additional information. Augmenting the video represented by the biological sequence, and (iii) a plurality of image includes positionally linking the at least one of (i) the properties of the chemical compound or biological species. portion of the additional information and (ii) the one or more The method also includes augmenting, by the processor, the controls to the visual identification Such that, upon movement Video image with at least one of (i) at least a portion of the of the visual identification within the video image, the at least additional information, and (ii) one or more controls config 40 one of (i) the portion of the additional information and (ii) the ured, upon selection, to present at least a portion of the addi one or more controls undergo a corresponding motion. The tional information. The method further includes, after aug instructions, when executed by the processor, further cause menting the video image, providing the video data for the processor to provide the video data for presentation upon presentation upon a display controlled by the first computing a display, wherein the video data is presented in Substantially device. The video data is presented by the first computing 45 real time in relation to the capture of the video data by the device in substantially real time in relation to the capture of computing device. the video data by the first computing device. In some embodiments, the computing device includes the In some embodiments, the first computing device includes processor. In some embodiments, the visual identification the second computing device. In some embodiments, the includes a two-dimensional drawing of a chemical structure graphical representation of the chemical structure includes a 50 or biological sequence. In some embodiments, the instruc two-dimensional drawing of the chemical structure. In some tions, when executed, cause the processor to: receive, from a embodiments, augmenting the video image includes deter user of the computing device, an indication of video freeze; mining an orientation of the first computing device; and ren and responsive to the indication of video freeze, provide for dering the three-dimensional representation based in part presentation a still image including a most recent augmented upon the orientation of the first computing device. In some 55 Video data frame. In some embodiments, the instructions, embodiments, the additional information further includes (iv) when executed, cause the processor to: receive, after provid a storage location. In some embodiments, matching addi ing the still image, selection of a first control of the one or tional information to the graphical representation includes more controls; apply, responsive to selection of the first con determining the storage location, wherein determining the trol, the portion of the additional information related to the storage location includes identifying the chemical compound 60 first control to the still image to determine a second still or biological species within a catalog. In some embodiments, image; and provide, for presentation upon the display, the augmenting the video image includes rendering a first control second still image. of the one or more controls as a semi-transparent image Another aspect of the invention relates to a non-transitory overlaid upon the video image. In some embodiments, aug computer readable medium having instructions stored menting the video image includes Superimposing the graphi 65 thereon. The instructions, when executed by a processor, cal representation with at least a portion of the additional cause the processor to: receive video data destined for display information. In some embodiments, matching the additional upon a computing device, wherein the video data includes a US 8,854,361 B1 5 6 Video image; identify, within the video image, a graphical developed using information from the embodiments representation of one of a chemical structure and a biological described herein. Adaptation and/or modification of the appa sequence; match additional information to the graphical rep ratus, systems, and methods described herein may be per resentation, wherein the additional information is arranged in formed by those of ordinary skill in the relevant art. a plurality of feature categories; augment the video image Throughout the description, where apparatus and systems with one or more controls configured, upon selection, to are described as having, including, or comprising specific presentat least a portion of the additional information belong components, or where processes and methods are described ing to a first category of the plurality of feature categories; as having, including, or comprising specific steps, it is con after augmenting the video image, provide the video data for presentation upon a display controlled by the computing templated that, additionally, there are apparatus and systems device; receive, responsive to selection of a first control of the 10 of the present invention that consist essentially of, or consist one or more controls by a user of the computing device, an of the recited components, and that there are processes and indication corresponding to the first category; and responsive methods according to the present invention that consistessen to the indication, augment Subsequent video data with at least tially of, or consist of the recited processing steps. a portion of the additional information belonging to the first It should be understood that the order of steps or order for category; and after augmenting the Subsequent video data, 15 performing certain actions is immaterialso long as the inven provide the Subsequent video data for presentation upon the tion remains operable. Moreover, two or more steps or actions display controlled by the computing device. may be conducted simultaneously. In some embodiments, the computing device includes the As used herein, the term “biological sequence” refers to the processor. In some embodiments, the video data was captured sequence of nucleotide or amino acid residues of a biological by the computing device; and the video data is presented by molecule (e.g., a DNA molecule, an RNA molecule, or a the computing device in Substantially real time in relation to polypeptide). A biological sequence can be graphically rep the capture of the video data by the computing device. resented in various ways, e.g., textually by a sequence of Elements of embodiments described with respect to a given letters (e.g., using a 1-letter nucleotide code or using a 1-letter aspect of the invention may be used in various embodiments or 3-letter amino acid code), or structurally (e.g., as a chemi of another aspect of the invention. For example, it is contem 25 cal structure, a ball-and-stick model, a ribbon diagram, a plated that features of dependent claims depending from one space-filling model, or an electrostatic model). independent claim can be used in apparatus, articles, systems, FIG. 1 is a block diagram of an example system 100 for and/or methods of any of the other independent claims. visually augmenting a graphical rendering of a chemical BRIEF DESCRIPTION OF THE DRAWINGS structure representation or biological sequence representa 30 tion with multi-dimensional information. The system 100 The foregoing and other objects, aspects, features, and includes a computing device 102 executing a software appli advantages of the invention will become more apparent and cation 124 for applying augmented reality data to captured may be better understood by referring to the following video information by identifying either a chemical structure description taken in conjunction with the accompanying or a biological sequence within the captured video and offer drawings, in which: 35 ing augmentation features and additional information regard FIG. 1 is a block diagram of an example system for visually ing the identified chemical structure or biological sequence. augmenting a graphical rendering of a chemical structure The computing device 102, for example, captures a video of representation or biological sequence representation with a chemical structure 106a drawn upon a piece of paper 104a, multi-dimensional information; identifies the chemical structure 106a as being a particular FIG. 2 is a flow chart of an example method for visually 40 chemical compound, matches additional information to the augmenting a graphical rendering of a chemical structure chemical compound for augmenting the video, and displays a representation or biological sequence representation with video image 104b of the piece of paper 104a augmented with multi-dimensional information; the additional information within a display area 112 of the FIG. 3 is a screen shot of an example user interface dem onstrating visual augmentation of a graphical rendering of a computing device 102. The additional information, for chemical structure representation; 45 example, includes a two-dimensional chemical structure FIG. 4 is a flow chart of an example method for deriving 106b overlaying the hand-drawn chemical structure 106a and and presenting multi-dimensional information related to a a three-dimensional graphical representation 122 of the graphical rendering of a chemical structure representation or chemical structure 106bpresented adjacent to the two-dimen biological sequence representation; sional chemical structure 106b. FIG. 5 is a block diagram of an example network environ 50 In some embodiments, the Software application 124 ment for visually augmenting a graphical rendering of a executing upon the computing device 102 accesses a remote chemical structure representation with multi-dimensional server 108 (e.g., via a network 110) to conduct a portion of the information; operations to achieve the augmented video presented within FIG. 6 is a block diagram of an example computing device the display 112. For example, the software application 124 and an example mobile computing device. 55 executing upon the computing device 102 may pass all or a The features and advantages of the present disclosure will portion of the video data to the server 108. In another become more apparent from the detailed description set forth example, the Software application 124 executing upon the below when taken in conjunction with the drawings, in which computing device 102 may pass user configuration settings like reference characters identify corresponding elements (e.g., as saved in a configuration data store 126), user inputs, throughout. In the drawings, like reference numbers gener 60 chemical structure information, chemical compound infor ally indicate identical, functionally similar, and/or structur mation, biological sequence information, biological species ally similar elements. information, or other data to the server 108. The server 108, in response, may provide data for augmenting the video data DESCRIPTION and/or an augmented video image to the computing device 65 102. It is contemplated that apparatus, systems, and methods of In some embodiments, the server 108 may include an the claimed invention encompass variations and adaptations image identification engine 114 for identifying a graphical US 8,854,361 B1 7 8 indication of a chemical structure or biological sequence embodiments, the video augmenting engine 128 presents a within the video data and correlating the chemical structure to portion of the augmentation data as semi-transparent images a chemical compound or the biological sequence to a biologi overlaying the original video image. For example, the hand cal species (e.g., DNA molecule, RNA molecule, polypep drawn chemical structure 106a may be partially visible tide, etc.). The chemical structure or biological sequence beneath the two-dimensional chemical structure 106b. The captured by the video data, in Some examples, can include a Video augmenting engine 128, in some implementations, pre two-dimensional graphical representation, a three-dimen sents a portion of the augmentation data in a manner that sional graphical representation, or another graphical indica “pins' the augmentation data to the video image of the chemi tion of a chemical compound or biological species. In some cal structure 106a. For example, should the user move the embodiments, the image identification engine 114 may rec 10 ognize a written name or formula of a biological species or computing device 102 or the paper 104 such that the align chemical compound within the video data. The chemical ment of the chemical structure 106a moves within the display structure or biological sequence captured within the video 112, a portion of the augmentation data may move with the data, in Some additional examples, may be illustrated or hand position of the chemical structure 106a within the video drawn on a flat or curved Surface Such as, in some examples, 15 image. Visual elements, in some embodiments, may tie the a white board, chalk board, or transparent hood of a labora portion of the augmentation data to the chemical structure tory experiment station. 106.a. For example, arrows, call-out boxes, or lead lines may In some embodiments, the server 108 may include an addi create a visual connection between the augmentation data and tional information matching engine 116 for matching addi the chemical structure 106b. This may be especially useful, tional information to the identified chemical compound or for example, if two or more chemical structures or biological biological species, such as, in some examples, a name of the sequences are illustrated within the video data of the display chemical compound or biological species, properties of the 112. In this manner, the user could immediately interpret chemical compound or biological species, or a graphical rep which augmentation data belonged to which chemical struc resentation (e.g., two dimensional, three dimensional, etc.) of ture or biological sequence. the chemical structure or biological sequence. In some imple 25 In addition to presenting the three-dimensional graphical mentations, the additional information includes a location of representation 122 and overlaying the hand-drawn chemical the chemical compound or biological species within a labo structure 106a with the two-dimensional chemical structure ratory, or reaction information regarding an ongoing experi 106b, in some implementations, the software application 124 ment involving the chemical structure or biological sequence. may be used to provide the user with multi-dimensional data The additional information, in some implementations, is 30 regarding the chemical structure 106. For example, the soft accessed by the additional information matching engine 116 ware application 124 may be configured to present a first set from one or more databases, such as a data store 120. In some of options (e.g., as a default or based upon the user configu implementations, a portion of the additional information is ration settings in the configuration store 126) as one or more accessed from a public repository, such as the PubChem selectable controls (not illustrated), each control representing Compound database maintained by the National Center for 35 a category of information related to the chemical structure Biotechnology Information (NCBI), the molecular spectral 106. In some examples, the categories may include chemical databases maintained by the National Institute of Standards properties, a storage location, or a name of the compound. and Technology (NIST), the Genbank sequence database Upon selecting a first selectable control, the Software appli maintained by NCBI, or the UniProt protein sequence data cation 124 may update the augmentation data presented base maintained by the UniProt consortium. In some imple 40 within the display 112 to include information identified as mentations, a portion of the additional information is pertaining to the selected category. The information, in some accessed from an electronic lab notebook (ELN) system. examples, may include text, graphics, video, and/or audio The server 108, in some implementations, includes a three information. In some embodiments, one or more sub-catego dimensional view engine 118 for presenting and adjusting a ries may be presented to the user upon selection of a primary three-dimensional graphical representation of the identified 45 category. For example, upon selection of a properties cat chemical compound or biological species. The three-dimen egory, Sub-categories of physical properties and chemical sional view engine 118, for example, may be used to render properties may be presented. In this manner, a user may drill the three-dimensional graphical representation 122 within the down to obtain more and more specific information regarding display 112. In some embodiments, the three-dimensional the chemical structure 106. view engine 118 modifies the orientation of the three-dimen 50 Although illustrated in a particular example system 100, in sional graphical representation 122 based upon orientation other implementations, the computing device 102, shown input, Such as a direct user input (e.g., one or more controls here as a handheld computing device, may be a tablet com linked to the three-dimensional graphical representation 122 puter, laptop computer, desktop computer, an augmented for rotating or tilting the aspect of the three-dimensional reality wearable computer Such as computerized glasses (e.g., graphical representation 122) or device orientation input 55 Google R. GlassTM by Google(R) Inc. of Mountain View, (e.g., as gleaned from one or more orientation sensors of the Calif.), or other computing device in communication with a computing device 102 Such as a gyroscope or accelerometer). Video capture device and a display. In some embodiments, the server 108 includes a video Additionally, in other implementations, one or more of the augmenting engine 128 for building an augmented video for image identification engine 114, additional information presentation within the display 112. The video augmenting 60 matching engine 116, three-dimensional view engine 118, engine 128, for example, may merge the video captured by and video augmenting engine 128 may reside within the the computing device 102 with augmentation data Such as, in computing device 102 (e.g., as part of the Software applica Some examples, a portion of the additional information (e.g., tion 124). In some implementations, the server 108 may obtained at least in part from the data store 120), selectable include two or more servers, such as a server farm or cloud controls for presenting additional information, and/or a two 65 computing service for video augmentation. Further details dimensional or three-dimensional graphical representation of and options for providing methods and systems for visually the chemical structure or biological sequence. In some augmenting a graphical rendering of either a chemical struc US 8,854,361 B1 10 ture representation or a biological sequence representation chemical compound or biological species. The additional with multi-dimensional information are described below. information, for example, may be arranged in categories and FIG. 2 is a flow chart of an example method 200 for visu Sub-categories Such that a user may drill down to obtain ally augmenting a graphical rendering of a chemical structure further details regarding the chemical compound or biologi representation with multi-dimensional information. The cal sequence. To aid in obtaining the additional information, method 200, in some implementations, may be performed by in Some implementations, the augmentation data includes the server 108 and/or the software application 124, as control information Such as one or more selectable call-out described in relation to FIG. 1. boxes configured, upon selection, to present a portion of the In some implementations, the method 200 begins with additional information. receiving video data destined for presentation upon a user 10 In some implementations, an indication of device orienta device (202). The video data, for example, may be captured tion is received (210). For example, data derived from an using a camera feature of computerized eye glasses or a orientation sensor (e.g., gyroscope, accelerometer, etc.) of a handheld computing device Such as the computing device 102 handheld computing device may be provided to aid in the described in relation to FIG. 1. In other examples, the video augmentation of the video data. If the device orientation is data may be captured using a video camera device connected 15 known (210), in Some implementations, a visual representa to a personal computer, or the video data may be provided in tion of a portion of the additional information is adjusted real time from a separate computing device (e.g., broadcast a responsive to the device orientation (212). For example, the number of students’ computing devices in a lecture hall). The orientation of text presented as additional information or Video includes a video image track. In some implementations, written upon a selectable control may be adjusted. If a three the video data further includes an audio image track and/or a dimensional graphical representation of the chemical struc metadata track. ture or biological sequence has been identified as augmenta In some implementations, a graphical representation of tion data, in some implementations, a rotation and/or either a chemical structure or a biological sequence is iden revolution of the three-dimensional graphical representation tified within the video image track of the video data (204). may be adjusted based upon the orientation of the device. For The graphical representation, for example, may be identified 25 example, the three-dimensional graphical representation may by the image identification engine 114, described in relation tilt with the tilting of the handheld device. The orientation of to FIG.1. The chemical structure or biological sequence, for the three-dimensional graphical representation may be example, may be a hand-drawn illustration or print illustra adjusted, for example, by the three-dimensional view engine tion representing a chemical compound or biological species. 118, described in relation to FIG. 1. In some implementations, the chemical structure or biologi 30 In some implementations, a portion of the augmentation cal sequence may be partially presented (e.g., one or more data is applied to the video data prior to presentation upon the bonds missing, a portion of an atom identifier cut off), user device (214). The augmentation data may be applied to obscured, or presented upon a curved surface. If the chemical the video data, for example, by the video augmenting engine structure or biological sequence is not complete or otherwise 128, described in relation to FIG. 1. The video data, for exactly matching a corresponding chemical compound or 35 example, may be merged with the augmentation data to create biological species, in some implementations, the chemical an augmented video data for presentation by the user device. compound or biological species is digitally refined to obtain In some embodiments, a portion of the augmentation data is an image for comparison with known chemical structures or presented as semi-transparent images overlaying the original biological sequences. For example, once a Suspected chemi Video image. In some implementations, a portion of the aug cal structure or biological sequence has been identified, the 40 mentation data is presented in a manner that “pins' the aug portion of the image containing the Suspected chemical struc mentation data to the graphical representation of the chemical ture may be pruned (e.g., Stray lines or text removed), image structure orbiological sequence. For example, should the user adjusted, or otherwise digitally altered in an attempt to match move the computing device or the item including the graphi the graphical representation of the chemical structure or bio cal representation of the chemical structure or biological logical sequence captured within the video image to a chemi 45 sequence Such that the alignment of the chemical structure or cal known chemical compound or biological species. In a biological sequence moves within the video image frame, a particular example, two or more video frames may be portion of the augmentation data may move with the position reviewed (e.g., merged, appended) in relation to each other to of the graphical representation of the chemical structure or identify the chemical compound or biological species. biological sequence. Visual elements, in Some embodiments, In other implementations, rather than identifying a graphi 50 may tie a portion of the augmentation data to the graphical cal representation of a chemical structure or biological representation of the chemical structure or biological sequence, a name of a chemical compound or biological sequence. For example, arrows, call-out boxes, or lead lines species, a chemical formula, or another representation of a may create a visual connection between the augmentation chemical compound or biological species may be identified. data and the graphical representation of the chemical struc In some implementations, augmentation data is matched to 55 ture or biological sequence. the chemical compound or biological species to augment the In Some implementations, user input may be received Video image (206). The augmentation data may be matched to (216). For example, a user may select one of the selectable the chemical compound or biological species, for example, controls presented as augmentation data, or a user may inter by the additional information matching engine 116 described act with a user-selectable three-dimensional graphical repre in relation to FIG.1. The augmentation data, in Some imple 60 sentation of the chemical structure or biological sequence mentations, includes image data Such as a two-dimensional or presented as augmentation data. If a user input other than three-dimensional representation of the chemical structure or orientation change is received (216), in Some implementa biological sequence. In some implementations, the augmen tions, follow-on augmentation data is identified for augment tation data includes additional information Such as properties ing the video data (218). For example, based upon the selec of the chemical compound or biological species, a storage 65 tion by the user of a category of additional information, location of the chemical compound or biological species, or corresponding image, text, and/or graphic data may be pre reaction information regarding an experiment involving the sented upon the video feed as augmentation data. Using the US 8,854,361 B1 11 12 same example, rather than or in addition to the additional or biological sequence (e.g., move the camera lens of the information, the user may be presented with one or more computing device 102 away from pointing at the paper 104a). selectable Sub-categories related to the selected category. In Some implementations, although the video has been frozen Whether or not follow-on augmentation data is identified, on a Snapshot image (e.g., current frame), the controls remain in some implementations, it is determined whether an orien active for pursuing additional information regarding the tation change has been received (220). In some implementa chemical structure or biological sequence illustrated within tions, the user alters the orientation of the computing device the Snapshot image. (e.g., physically tilts the device or rotates the display), caus The user interface 304 also includes a three-dimensional ing updated orientation information to be received. In some graphical representation 310 of the chemical structure 106a, implementations, the user manipulates the three-dimensional 10 presented adjacent to the two-dimensional graphical repre graphical representation of the chemical structure or biologi sentation 306. In some implementations, upon changing the cal sequence, causing orientation data specific to the three orientation of the computing device 302, the orientation of the dimensional graphical representation to be received. The three-dimensional graphical representation 310 may auto visual representation of at least a portion of the augmentation matically adjust (e.g., tilt or rotate). In some implementations, data, in some implementations, is adjusted responsive to the 15 the three-dimensional graphical representation 310 slowly orientation change (212). revolves and/or rotates as a moving image within the video Whether or not an orientation change was received at Step presented by the user interface 304. 220, as described previously in relation to step (214), in some At the bottom of the user interface 304, a search field 312 implementations, the augmentation data (and/or, if appli is presented. The search field 312 may be used, for example, cable, follow-on augmentation data) is applied to the Subse for searching for additional information related to the chemi quent (e.g., current) video data prior to presentation of the cal structure 106a. augmented Subsequent video data upon the user device (214). FIG. 4 is a flow chart of an example method 400 for deriv The method 200, in some implementations, continues to ing and presenting multi-dimensional information related to a update and adjust as additional video data frames, user inputs, graphical rendering of a chemical structure representation or and orientation changes are received. 25 biological sequence representation. The method 400, in some Although described in relation to a series of particular implementations, may be performed by the server 108 and/or steps, in Some implementations, one or more of the steps of the software application 124, as described in relation to FIG. the method 200 may be performed in a different order, or one 1. or more steps may be performed in parallel. In some imple In some implementations, the method 400 begins with mentations, one or more of the steps of the method 200 may 30 receiving video data destined for presentation upon a user be removed or modified, or one or more steps may be added device (402). The video data, for example, may be captured to the method 200, while staying within the scope and intent using a camera feature of computerized eye glasses or a of the method 200. handheld computing device Such as the computing device 102 FIG. 3 is a screen shot 300 of an example user interface described in relation to FIG. 1. In other examples, the video 304, rendered upon a computing device 302. The user inter 35 data may be captured using a video camera device connected face 304 demonstrates an example of visual augmentation of to a personal computer, or the video data may be provided in a graphical rendering of a chemical structure representation. real time from a separate computing device (e.g., broadcast a The user interface 304, for example, may be presented in number of students’ computing devices in a lecture hall). The relation to capturing video data regarding the hand-drawn Video includes a video image track. In some implementations, chemical structure 106a upon the paper 104a, as described in 40 the video further includes an audio image track and/or a relation to FIG. 1. metadata track. As illustrated within the user interface 304, the image of In some implementations, a graphical representation of the paper 104a containing the hand-drawn chemical structure either a chemical structure or a biological sequence is iden 106a has been obscured by semi-transparent augmentation tified within the video image track of the video data (404). data, including a two-dimensional graphical representation 45 The graphical representation, for example, may be identified 306 of the chemical structure 106a and a series of selectable by the image identification engine 114, described in relation call-out boxes 308 identifying categories of additional infor to FIG.1. The chemical structure or biological sequence, for mation pertaining to the chemical structure 106a Such as a example, may be a hand-drawn illustration or print illustra locations category call-out box 308a, a properties category tion representing a chemical formula or biological species. In call-out box 308b, and a name category call-out box 308c. 50 Some implementations, the chemical structure or biological Upon selection by a user of one of the selectable call-out sequence may be partially presented (e.g., one or more bonds boxes 308, for example, augmentation data regarding the missing, a portion of an atom identifier cut off), obscured, or selected category may be presented to the user in the display presented upon a curved surface. If the chemical structure or area of the computing device 302. In some implementations, biological sequence is not complete or otherwise exactly upon selection by a user of one of the selectable call-out boxes 55 matching a corresponding chemical compound or biological 308, one or more additional controls are presented to the user. species, in some implementations, the chemical compound or The additional controls, for example, may be configured to biological species is digitally refined to obtain an image for present, upon selection, information regarding one or more comparison with known chemical structures or biological Sub-categories of the identified category. sequences. For example, once a suspected chemical structure In some implementations, the one or more additional con 60 or biological sequence has been identified, the portion of the trols may include a control configured to allow the user to image containing the Suspected chemical structure may be freeze the video image containing the information regarding pruned (e.g., Stray lines or text removed), image-adjusted, or the identified category. For example, upon being presented otherwise digitally altered in an attempt to match the graphi with information relevant to the identified chemical structure cal representation of the chemical structure or biological or biological sequence, the user may desire to take a 'snap 65 sequence captured within the video image to a chemical shot' of the information for review without having to main known chemical compound or biological species. In a par tain video connection with the identified chemical structure ticular example, two or more video frames may be reviewed US 8,854,361 B1 13 14 (e.g., merged, appended) in relation to each other to identify the method 400 may be performed in a different order, or one the chemical compound or biological species. or more steps may be performed in parallel. In some imple In other implementations, rather than identifying a graphi mentations, one or more of the steps of the method 400 may cal representation of a chemical structure or biological be removed or modified, or one or more steps may be added sequence, a name of a chemical compound or biological to the method 400, while staying within the scope and intent species, a chemical formula, or another representation of a of the method 400. chemical compound or biological species may be identified. As shown in FIG. 5, an implementation of an exemplary In some implementations, augmentation features are iden cloud computing environment 500 for visually augmenting a tified for adding information related to the chemical com graphical rendering of either a chemical structure represen pound or biological species to the video data (406). The 10 tation or a biological sequence representation with multi augmentation features may be identified, for example, by the dimensional information is shown and described. The cloud additional information matching engine 116 described in computing environment 500 may include one or more relation to FIG. 1. The augmentation features, for example, resource providers 502a, 502b,502c (collectively, 502). Each may be arranged in categories and Sub-categories such that a resource provider 502 may include computing resources. In user may drill down to obtain further details regarding the 15 Some implementations, computing resources may include chemical compound or biological species. To aid in obtaining any hardware and/or software used to process data. For the additional information, in Some implementations, each example, computing resources may include hardware and/or augmentation feature is associated with a user interface con Software capable of executing algorithms, computer pro trol configured, upon selection, to present category informa grams, and/or computer applications. In some implementa tion related to the chemical compound or biological species. tions, exemplary computing resources may include applica In some implementations, the video image, including one tion servers and/or databases with storage and retrieval or more controls for selecting the augmentation features, is capabilities. Each resource provider 502 may be connected to provided for presentation to the user on the user computing any other resource provider 502 in the cloud computing envi device (408). The controls, in some examples, may include ronment 500. In some implementations, the resource provid any combination of call-out boxes, radio buttons, a configu 25 ers 502 may be connected over a computer network 508. Each ration listing, a search box, or a voice activated listing of resource provider 502 may be connected to one or more available options for further information. In some implemen computing device 504a, 504b, 504c (collectively, 504), over tations, the controls are pinned to the identified chemical the computer network 508. structure or biological sequence, such that, as the identified The cloud computing environment 500 may include a chemical structure or biological sequence moves within the 30 resource manager 506. The resource manager 506 may be display region of the user computing device (e.g., due to connected to the resource providers 502 and the computing movement of the camera relative to the chemical structure or devices 504 over the computer network 508. In some imple biological sequence being captured) the controls move within mentations, the resource manager 506 may facilitate the pro the display to be positioned relative to the identified chemical vision of computing resources by one or more resource pro structure or biological sequence. In some implementations, 35 viders 502 to one or more computing devices 504. The one or more controls are made visible or rendered opaque resource manager 506 may receive a request for a computing upon initial input from a user (e.g., identification of a touch by resource from a particular computing device 504. The the user upon a touch screen display, identification of a resource manager 506 may identify one or more resource "click” by the user via an input tool, etc.). providers 502 capable of providing the computing resource In some implementations, ifa feature is selected by the user 40 requested by the computing device 504. The resource man (410), the video data is augmented with information related to ager 506 may select a resource provider 502 to provide the the selected feature (412). For example, upon receiving an computing resource. The resource manager 506 may facili indication of a selection of a particular control, information tate a connection between the resource provider 502 and a related to that control (e.g., one or more of text, graphic, particular computing device 504. In some implementations, Video, and audio information) may be applied to Subsequent 45 the resource manager 506 may establish a connection Video data prior to presentation of the Subsequent video data. between a particular resource provider 502 and a particular In some implementations, in lieu of or in addition to pre computing device 504. In some implementations, the senting information related to the selected control, one or resource manager 506 may redirect a particular computing more sub-features of the selected features are identified (414). device 504 to a particular resource provider 502 with the For example, one or more Sub-categories may be identified 50 requested computing resource. related to a selected information category. Upon identifica FIG. 6 shows an example of a computing device 600 and a tion of the Sub-categories, in Some implementations, the mobile computing device 650 that can be used to implement Video data may be augmented with one or more controls for the techniques described in this disclosure. The computing selecting the one or more Sub-categories. If, instead, no Sub device 600 is intended to represent various forms of digital categories are available (e.g., the user has reached the finest 55 computers, such as laptops, desktops, workstations, personal granularity of information regarding the current topic), in digital assistants, servers, blade servers, mainframes, and Some implementations, one or more controls regarding a other appropriate computers. The mobile computing device higher level of category abstraction may be presented (e.g., 650 is intended to represent various forms of mobile devices, akin to a “back” or “related information' option). Such as personal digital assistants, cellular telephones, Smart In some implementations, the augmented Subsequent 60 phones, and other similar computing devices. The compo video data is presented with feature information and/or one or nents shown here, their connections and relationships, and more controls for selecting sub-features of the identified fea their functions, are meant to be examples only, and are not ture (416). The augmented Subsequent video data, for meant to be limiting. example, may be presented in a manner similar to that The computing device 600 includes a processor 602, a described in relation to step 408. 65 memory 604, a storage device 606, a high-speed interface 608 Although described in relation to a series of particular connecting to the memory 604 and multiple high-speed steps, in Some implementations, one or more of the steps of expansion ports 610, and a low-speed interface 612 connect US 8,854,361 B1 15 16 ing to a low-speed expansion port 614 and the storage device The mobile computing device 650 includes a processor 606. Each of the processor 602, the memory 604, the storage 652, a memory 664, an input/output device Such as a display device 606, the high-speed interface 608, the high-speed 654, a communication interface 666, and a transceiver 668, expansion ports 610, and the low-speed interface 612, are among other components. The mobile computing device 650 interconnected using various busses, and may be mounted on may also be provided with a storage device. Such as a micro a common motherboard or in other manners as appropriate. drive or other device, to provide additional storage. Each of The processor 602 can process instructions for execution the processor 652, the memory 664, the display 654, the within the computing device 600, including instructions communication interface 666, and the transceiver 668, are stored in the memory 604 or on the storage device 606 to interconnected using various buses, and several of the com display graphical information for a GUI on an external input/ 10 ponents may be mounted on a common motherboard or in output device, such as a display 616 coupled to the high-speed other manners as appropriate. interface 608. In other implementations, multiple processors The processor 652 can execute instructions within the and/or multiple buses may be used, as appropriate, along with mobile computing device 650, including instructions stored multiple memories and types of memory. Also, multiple com in the memory 664. The processor 652 may be implemented puting devices may be connected, with each device providing 15 as a chipset of chips that include separate and multiple analog portions of the necessary operations (e.g., as a server bank, a and digital processors. The processor 652 may provide, for group of blade servers, or a multi-processor system). example, for coordination of the other components of the The memory 604 stores information within the computing mobile computing device 650, such as control of user inter device 600. In some implementations, the memory 604 is a faces, applications run by the mobile computing device 650, Volatile memory unit or units. In some implementations, the and wireless communication by the mobile computing device memory 604 is a non-volatile memory unit or units. The 650. memory 604 may also be another form of computer-readable The processor 652 may communicate with a user through a medium, Such as a magnetic or optical disk. control interface 658 and a display interface 656 coupled to The storage device 606 is capable of providing mass stor the display 654. The display 654 may be, for example, a TFT age for the computing device 600. In some implementations, 25 (Thin-Film-Transistor Liquid Crystal Display) display or an the storage device 606 may be or contain a computer-readable OLED (Organic Light Emitting Diode) display, or other medium, Such as a floppy disk device, a hard disk device, an appropriate display technology. The display interface 656 optical disk device, or a tape device, a flash memory or other may comprise appropriate circuitry for driving the display similar solid state memory device, or an array of devices, 654 to present graphical and other information to a user. The including devices in a storage area network or other configu 30 control interface 658 may receive commands from a user and rations. Instructions can be stored in an information carrier. convert them for submission to the processor 652. In addition, The instructions, when executed by one or more processing an external interface 662 may provide communication with devices (for example, processor 602), perform one or more the processor 652, so as to enable near area communication of methods, such as those described above. The instructions can the mobile computing device 650 with other devices. The also be stored by one or more storage devices such as com 35 external interface 662 may provide, for example, for wired puter- or machine-readable mediums (for example, the communication in Some implementations, or for wireless memory 604, the storage device 606, or memory on the pro communication in other implementations, and multiple inter cessor 602). faces may also be used. The high-speed interface 608 manages bandwidth-inten The memory 664 stores information within the mobile sive operations for the computing device 600, while the low 40 computing device 650. The memory 664 can be implemented speed interface 612 manages lower bandwidth-intensive as one or more of a computer-readable medium or media, a operations. Such allocation of functions is an example only. Volatile memory unit or units, or a non-volatile memory unit In some implementations, the high-speed interface 608 is or units. An expansion memory 674 may also be provided and coupled to the memory 604, the display 616 (e.g., through a connected to the mobile computing device 650 through an graphics processor or accelerator), and to the high-speed 45 expansion interface 672, which may include, for example, a expansion ports 610, which may accept various expansion SIMM (Single In Line Memory Module) card interface. The cards (not shown). In the implementation, the low-speed expansion memory 674 may provide extra storage space for interface 612 is coupled to the storage device 606 and the the mobile computing device 650, or may also store applica low-speed expansion port 614. The low-speed expansion port tions or other information for the mobile computing device 614, which may include various communication ports (e.g., 50 650. Specifically, the expansion memory 674 may include USB, Bluetooth R, Ethernet, wireless Ethernet) may be instructions to carry out or Supplement the processes coupled to one or more input/output devices, such as a key described above, and may include secure information also. board, a pointing device, a scanner, or a networking device Thus, for example, the expansion memory 674 may be pro Such as a Switch or router, e.g., through a network adapter. vide as a security module for the mobile computing device The computing device 600 may be implemented in a num 55 650, and may be programmed with instructions that permit ber of different forms, as shown in the figure. For example, it secure use of the mobile computing device 650. In addition, may be implemented as a standard server 620, or multiple secure applications may be provided via the SIMM cards, times in a group of such servers. In addition, it may be imple along with additional information, Such as placing identify mented in a personal computer Such as a laptop computer 622. ing information on the SIMM card in a non-hackable manner. It may also be implemented as part of a rack server system 60 The memory may include, for example, flash memory and/ 624. Alternatively, components from the computing device or NVRAM memory (non-volatile random access memory), 600 may be combined with other components in a mobile as discussed below. In some implementations, instructions device (not shown), such as a mobile computing device 650. are stored in an information carrier, that the instructions, Each of Such devices may contain one or more of the com when executed by one or more processing devices (for puting device 600 and the mobile computing device 650, and 65 example, processor 652), perform one or more methods. Such an entire system may be made up of multiple computing as those described above. The instructions can also be stored devices communicating with each other. by one or more storage devices, such as one or more com US 8,854,361 B1 17 18 puter- or machine-readable mediums (for example, the The term machine-readable signal refers to any signal used to memory 664, the expansion memory 674, or memory on the provide machine instructions and/or data to a programmable processor 652). In some implementations, the instructions processor. can be received in a propagated signal, for example, over the To provide for interaction with a user, the systems and transceiver 668 or the external interface 662. techniques described here can be implemented on a computer The mobile computing device 650 may communicate wire having a display device (e.g., a CRT (cathode ray tube) or lessly through the communication interface 666, which may LCD (liquid crystal display) monitor) for displaying infor include digital signal processing circuitry where necessary. mation to the user and a keyboard and a pointing device (e.g., The communication interface 666 may provide for commu a mouse or a trackball) by which the user can provide input to nications under various modes or protocols, such as GSM 10 the computer. Other kinds of devices can be used to provide voice calls (Global System for Mobile communications), for interaction with a user as well; for example, feedback SMS (Short Message Service). EMS (Enhanced Messaging provided to the user can be any form of sensory feedback Service), or MMS messaging (Multimedia Messaging Ser (e.g., visual feedback, auditory feedback, or tactile feed vice), CDMA (code division multiple access), TDMA (time 15 back); and input from the user can be received in any form, division multiple access), PDC (Personal Digital Cellular), including acoustic, speech, or tactile input. WCDMA (Wideband Code Division Multiple Access), The systems and techniques described here can be imple CDMA2000, or GPRS (General Packet Radio Service), mented in a computing system that includes a back end com among others. Such communication may occur, for example, ponent (e.g., as a data server), or that includes a middleware through the transceiver 668 using a radio-frequency. In addi component (e.g., an application server), or that includes a tion, short-range communication may occur, Such as using a front end component (e.g., a client computer having a graphi Bluetooth R., Wi-FiTM, or other such transceiver (not shown). cal user interface or a Web browser through which a user can In addition, a GPS (Global Positioning System) receiver interact with an implementation of the systems and tech module 670 may provide additional navigation- and location niques described here), or any combination of such back end, related wireless data to the mobile computing device 650, 25 middleware, or frontend components. The components of the which may be used as appropriate by applications running on system can be interconnected by any form or medium of the mobile computing device 650. digital data communication (e.g., a communication network). The mobile computing device 650 may also communicate Examples of communication networks include a local area audibly using an audio codec 660, which may receive spoken network (LAN), a wide area network (WAN), and the Inter 30 net. information from a user and convert it to usable digital infor The computing system can include clients and servers. A mation. The audio codec 660 may likewise generate audible client and server are generally remote from each other and Sound for a user, such as through a speaker, e.g., in a handset typically interact through a communication network. The of the mobile computing device 650. Such sound may include relationship of client and server arises by virtue of computer Sound from Voice telephone calls, may include recorded 35 programs running on the respective computers and having a Sound (e.g., voice messages, music files, etc.) and may also client-server relationship to each other. include Sound generated by applications operating on the It should also be noted that embodiments of the present mobile computing device 650. disclosure may be provided as one or more computer-read The mobile computing device 650 may be implemented in able programs embodied on or in one or more articles of a number of different forms, as shown in the figure. For 40 manufacture. The article of manufacture may be any Suitable example, it may be implemented as a cellular telephone 680. hardware apparatus, Such as, for example, a floppy disk, a It may also be implemented as part of a Smart-phone 682, hard disk, a CD ROM, a CD-RW, a CD-R, a DVD ROM, a personal digital assistant, or other similar mobile device. DVD-RW, a DVD-R, a flash memory card, a PROM, a RAM, Various implementations of the systems and techniques a ROM, or a magnetic tape. In general, the computer-readable described here can be realized in digital electronic circuitry, 45 programs may be implemented in any programming lan integrated circuitry, specially designed ASICs (application guage. Some examples of languages that may be used include specific integrated circuits), computer hardware, firmware, C, C++, or Java. The Software programs may be further trans software, and/or combinations thereof. These various imple lated into machine language or virtual machine instructions mentations can include implementation in one or more com and stored in a program file in that form. The program file may puter programs that are executable and/or interpretable on a 50 then be stored on or in one or more of the articles of manu programmable system including at least one programmable facture. processor, which may be special or general purpose, coupled Certain embodiments of the present invention were to receive data and instructions from, and to transmit data and described above. It is, however, expressly noted that the instructions to, a storage system, at least one input device, and present invention is not limited to those embodiments, but at least one output device. 55 rather the intention is that additions and modifications to what These computer programs (also known as programs, soft was expressly described herein are also included within the ware, Software applications or code) include machine instruc scope of the invention. Moreover, it is to be understood that tions for a programmable processor, and can be implemented the features of the various embodiments described herein in a high-level procedural and/or object-oriented program were not mutually exclusive and can exist in various combi ming language, and/or in assembly/machine language. As 60 nations and permutations, even if Such combinations or per used herein, the terms machine-readable medium and com mutations were not made express herein, without departing puter-readable medium refer to any computer program prod from the spirit and scope of the invention. In fact, variations, uct, apparatus and/or device (e.g., magnetic discs, optical modifications, and other implementations of what was disks, memory, Programmable Logic Devices (PLDs)) used described herein will occur to those of ordinary skill in the art to provide machine instructions and/or data to a program 65 without departing from the spirit and the scope of the inven mable processor, including a machine-readable medium that tion. As such, the invention is not to be defined only by the receives machine instructions as a machine-readable signal. preceding illustrative description. US 8,854,361 B1 19 20 In view of the structure, functions and apparatus of the 6. The method of claim 5, wherein matching additional systems and methods described here, in some implementa information to the graphical representation comprises deter tions, systems and methods for visually augmenting a graphi mining the storage location, wherein determining the storage cal rendering of a chemical structure representation with location comprises identifying the chemical compound or multi-dimensional information are provided. Having biological species within a catalog. described certain implementations of visual augmentation of 7. The method of claim 1, wherein the rendering and the a graphical rendering of a chemical structure representation merging comprises rendering a first graphical control of the or biological sequence representation with multi-dimen one or more graphical controls as a semi-transparent image sional information, it will now become apparent to one of skill and overlaying the semi-transparent image upon the first in the art that other implementations incorporating the con 10 video frame. cepts of the disclosure may be used. Therefore, the disclosure should not be limited to certain implementations, but rather 8. The method of claim 1, wherein merging the video data should be limited only by the spirit and scope of the following with the rendered augmentation data comprises Superimpos claims. ing the graphical representation with at least a portion of the 15 additional information. The invention claimed is: 9. The method of claim 1, wherein matching the additional 1. A method comprising: information to the graphical representation comprises com receiving video data captured by a first computing device, paring the graphical representation to one or more stored wherein the video data comprises a sequence of images graphical representations. comprising a first video frame; 10. The method of claim 9, wherein matching the addi identifying, within the first video frame, by a processor of tional information to the graphical representation comprises a second computing device, a graphical representation digitally refining the portion of the graphical representation of one of a chemical structure and a biological sequence; prior to comparing the graphical representation to the one or matching, by the processor, additional information to the more stored graphical representations. graphical representation, wherein the additional infor 25 11. The method of claim 1, further comprising: mation comprises at least one of receiving, by the processor, an indication of selection of a (i) a three-dimensional representation of the chemical first graphical control of the one or more graphical con structure or biological sequence, trols; (ii) a name of rendering, by the processor, augmentation data corre a) a chemical compound represented by the chemical 30 sponding to the additional information related to the first Structure, or b) a biological species comprising a DNA molecule, graphical control; and an RNA molecule, or a polypeptide represented by merging, by the processor, Subsequent video data with the the biological sequence, and portion of the rendered augmentation data related to the (iii) a plurality of properties of the chemical compound 35 first graphical control. or biological species; 12. The method of claim 11, wherein merging the subse rendering, by the processor, augmentation data corre quent video data comprises adding audio data to the video sponding to the additional information, wherein the aug data, wherein the audio data comprises a verbal description of mentation data comprises one or both of (i) and (ii): the additional information. (i) graphics corresponding to at least a portion of the 40 13. The method of claim 1, wherein merging the video data additional information, and with the rendered augmentation data comprises positioning (ii) one or more graphical controls configured, upon the augmentation data in proximity to the graphical represen selection, to present at least a portion of the additional tation of the chemical structure or the biological sequence information; and identified within the first video frame. merging, by the processor, the video data with the rendered 45 14. A system comprising: augmentation data to positionally link the augmentation a processor; and data to the graphical representation of the chemical a memory having instructions stored thereon, wherein the structure or the biological sequence identified within the instructions, when executed by the processor, cause the first video frame and providing the video data for pre processor to: sentation upon a display controlled by the first comput 50 receive video data captured by a computing device, ing device, wherein the video data is presented by the wherein the video data comprises a sequence of first computing device in Substantially real time in rela images comprising a first video frame; tion to the capture of the video data by the first comput identify, within the first video frame, a visual identifica ing device. tion of a chemical structure or a biological sequence; 2. The method of claim 1, wherein the first computing 55 match additional information to the visual identification, device comprises the second computing device. wherein the additional information comprises at least 3. The method of claim 1, wherein the graphical represen one of tation of the chemical structure comprises a two-dimensional (i) a graphical representation of the chemical structure drawing of the chemical structure. or biological sequence, 4. The method of claim 1, wherein merging the video data 60 (ii) a name of with the rendered augmentation data comprises: a) a chemical compound represented by the chemi determining an orientation of the first computing device; cal structure, or and b) a biological species comprising a DNA mol rendering the three-dimensional representation based in ecule, an RNA molecule, or a polypeptide rep part upon the orientation of the first computing device. 65 resented by the biological sequence, and 5. The method of claim 1, wherein the additional informa (iii) a plurality of properties of the chemical com tion further comprises (iv) a storage location. pound or biological species; US 8,854,361 B1 21 22 render augmentation data corresponding to the addi provide, for presentation upon the display, the second still tional information, wherein the augmentation data image. comprises one or both of (i) and (ii): 19. A non-transitory computer readable medium having (i) graphics corresponding to at least a portion of the instructions stored thereon, wherein the instructions, when additional information, and 5 executed by a processor, cause the processor to: (ii) one or more graphical controls configured, upon receive video data destined for display upon a computing selection, to present at least a portion of the addi device, wherein the video data comprises a sequence of tional information; and images comprising a first video frame; merge the Video data with the rendered augmentation identify, within the first video frame, a graphical represen data to positionally link the augmentation data to the 10 visual representation of the chemical structure or the tation of one of a chemical structure and a biological biological sequence identified within the first video Sequence; frame, wherein match additional information to the graphical representa merging the Video data with the rendered augmenta tion, wherein the additional information is arranged in a tion data comprises positionally linking the at least plurality of feature categories: 15 render augmentation data corresponding to the additional one of (i) the portion of the additional information information, wherein the augmentation data comprises and (ii) the one or more graphical controls to the one or more graphical controls configured, upon selec Visual identification such that, upon movement of tion, to present at least a portion of the additional infor the visual identification within the first video mation belonging to a first category of the plurality of frame, the at least one of (i) the portion of the feature categories; additional information and (ii) the one or more merge the Video data with the one or more graphical con graphical controls undergo a corresponding trols to positionally link the augmentation data to the motion; and graphical representation of the chemical structure or the provide the video data for presentation upon a display; biological sequence identified within the first video wherein the video data is presented in substantially real 25 time in relation to the capture of the video data by the frame; computing device. after augmenting the video data, provide the video data for 15. The system of claim 14, wherein the computing device presentation upon a display controlled by the computing comprises the processor. device; receive, responsive to selection of a first graphical control 16. The system of claim 14, wherein the visual identifica 30 of the one or more graphical controls by a user of the tion comprises a two-dimensional drawing of a chemical computing device, an indication corresponding to the Structure or biological sequence. first category; and 17. The system of claim 14, wherein the instructions, when responsive to the indication, augment subsequent video executed, cause the processor to: data with at least a portion of the additional information receive, from a user of the computing device, an indication 35 of video freeze; and belonging to the first category; and responsive to the indication of video freeze, provide for after augmenting the subsequent video data, provide the presentation a still image comprising a most recent aug Subsequent video data for presentation upon the display mented video data frame. controlled by the computing device. 20. The computer readable medium of claim 19, wherein 18. The system of claim 17, wherein the instructions, when 40 executed, cause the processor to: the computing device comprises the processor. receive, after providing the still image, selection of a first 21. The computer readable medium of claim 19, wherein: graphical control of the one or more graphical controls; the Video data was captured by the computing device; and apply, responsive to selection of the first graphical control, the video data is presented by the computing device in the portion of the additional information related to the substantially real time in relation to the capture of the first graphical control to the still image to determine a 45 video data by the computing device. Second still image; and ck ck ck ck ck