Visualizing 3D molecular structures using an augmented reality app

Kristina Eriksen, Bjarne E. Nielsen, Michael Pittelkow

5 Department of , University of Copenhagen, Universitetsparken 5, DK-2100 Copenhagen, Denmark. E-mail: [email protected]

ABSTRACT 10 We present a simple procedure to make an augmented reality app to visualize any 3D chemical model.

The molecular structure may be based on data from crystallographic data or from computer modelling.

This guide is made in such a way, that no programming skills are needed and the procedure uses free

and is a way to visualize 3D structures that are normally difficult to comprehend in the 2D

15 space of paper. The process can be applied to make 3D representation of any 2D object, and we envisage

the app to be useful when visualizing simple stereochemical problems, when presenting a complex 3D

structure on a poster presentation or even in audio-visual presentations. The method works for all

including small molecules, supramolecular structures, MOFs and biomacromolecules.

GRAPHICAL ABSTRACT

20

KEYWORDS

Augmented reality, Unity, Vuforia, Application, 3D models.

25

Journal 5/18/21 Page 1 of 14

INTRODUCTION

Conveying information about three-dimensional (3D) structures in two-dimensional (2D) space, such as

on paper or a screen can be difficult. Augmented reality (AR) provides an opportunity to visualize 2D

30 structures in 3D. Software to make simple AR apps is becoming common and ranges of free software

now exist to make customized apps. AR has transformed in computer games and films,

but the technique is distinctly under-used in (chemical) science.1

In chemical science the challenge of visualizing in 3D exists at several levels ranging from teaching of

stereo chemistry problems at freshman university level to visualizing complex molecular structures at

35 the forefront of chemical research. Visualization can be especially challenging since molecules are getting

larger and more complex and span 3D. An elegant way to visualize molecules in 3D is to 3D print the

desired structure, and protocols of how to do this starting from molecular structures have recently been

described.2 To describe the geometry and symmetry of complex molecules chemists are often forced to

draw molecules in simplified or schematic ways and thus neglect information. One example of a highly

40 complex that is difficult to display in 2D is the Molecular Borromean Rings prepared by

Stoddart and co-workers (Figure 1a). In their structural representation some atoms and labelling of

atoms are omitted to simplifying the structure.3 In the simplified 2D image bonds and atoms are

overlapping and thus still make it difficult to visualize the geometry and symmetry. Many chemists even

draw the same molecule twice in different formats in the same paper to better explain the connections

45 of the different elements and its geometry. This is illustrated with the porphyrin nano- by Anderson

(Figure 1b) and the supramolecular complex between biotin[6]uril and the iodide anion (Figure 1d).4-5 It

can be challenging to come up with a new synthetic route for complicated molecules such as Paclitaxel

(Figure 1c) because it is hard to visualize how sterically congested regions effects each other.6 For these

types of problems described above, a simple way to visualize molecules in 3D would be beneficial.

50

Journal 5/18/21 Page 2 of 14

Figure 1. Examples of complex molecules that are difficult to comprehend in 2D. Download the AR app on your device (phone or tablet) 55 to view these structures in AR. The app can be found via the QR code or the link in Figure 2. a) The Borromean rings are shown in two different ways to emphasize the geometry and symmetry of it. b) The porphyrin nanoball is presented in two different ways to highlight the geometry and the connection of the different elements in the nanoball. c) Paclitaxel is complex molecule with many stereo centers. A 3D view of it helps to appreciate the complexity and may aid retrosynthetic analysis. d) A topview and sideview of the structure of the biotin[6]uril. Download the app by following the link below (Figure 2) and see how the AR works. 60

In this contribution we describe how to make a simple AR app for a mobile device (e.g. phones and

tablets) to visualize molecules in 3D. It is important to emphasize that this guide is made in such a way,

that no programming skills are needed, and that only free software is used. When the app is made it is

free to transfer the app via an USB cable from the computer to the mobile devices. However, if you want

65 to publish the app in Google Play it requires a one-time payment to Google Play of 25 $.

This guide must be followed tightly, because a series of programs are needed. When the AR app is made

and transferred to a mobile device, a camera opens and it recognises an image. The image may be on a

poster, in a book or on a screen. The recognition leads to a 3D model (of one or more molecules of your

Journal 5/18/21 Page 3 of 14

own choice) is opening as a part of the real world through your mobile device. When all the software is

70 installed correctly, the app is simple to make and can be used at a poster session or in classroom.

Download an example of the AR app (follow the link or QR code below) on your android mobile device

and see how it works and what it looks like (in the real augmented world). Once the app is downloaded

and opened, then point the camera at Figure 1 and 2 and a 3D model of the molecules will appear in

AR.

75 Link to the AR app: https://play.google.com/store/apps/details?id=com.UniCPH.Android.MoleculAR

P Figure 2. A 2D image of Biotin[6]uril. Download the app from the link below or the QR code and point the mobile device to see the structure in 3D through the camera. https://play.google.com/store/apps/details?id=com.UniCPH.Android.MoleculAR 80

85

Journal 5/18/21 Page 4 of 14

THE HOW TO GUIDE:

STEP 1: MAKING A 3D MODEL OF THE MOLECULE

If you don’t have a specific molecule or just want to make the

90 app to visualize a common molecule, then skip this step and go

to “Step 2: Working with ”.

There are many programs that allow you to draw molecules in

3D. The most realistic 3D models for the given molecule are either from a crystal structure or a high

95 level optimization calculation by using software such as “”. It is also possible to draw the

molecule in chemical drawing software, such as “Chem3D” in the “ChemDraw” software package.

When the geometry of your molecule is satisfying, then save it at the desktop as a “mol”-file. Only the

geometry of the molecule is needed to be correct at this state. The format of your molecule is added in

the next step.

100 STEP 2: WORKING WITH JMOL

The format of your molecule is changed by using the software

“Jmol”. If you don’t have Jmol then download it here:

http://jmol.sourceforge.net/download/ and unfold the “Jmol-

105 14.29.46-binary.zip” in a known folder. Click at “jmol.bat” and

Jmol will open. If you have any problems by running Jmol,

then make sure that you are running the latest version from Sun. You find and install it here:

www.java.com.

110 If you do not have a mol-file from Step 1 then click File→Create MOL and write e.g. “porphyrin”. Then

a porphyrin molecule will appear on your screen in Jmol.

If you saved a mol-file on the desktop from Step 1 then open it in Jmol. When it is opened click at

File→Console. The console is a terminal where you write commands and thereby change the design of

115 your molecule. Here is a list of useful commands to change your design of your molecule:

Journal 5/18/21 Page 5 of 14

http://cbm.msoe.edu/teachingResources/images/jmolTraining/changingColorsAndDisplayFormats.p

df

If you want the spacefill format of your 3D molecule, then write “spacefill on” in the console window and

press Enter. The “Enter” activates any code in the console window. If you want the ball-and-stick format

120 then write “spacefill 20%,wireframe 0.15”and press Enter.

If you want all the nitrogen atoms to be blue, then write “color nitrogen blue” or if you want the bonds

to be black then write “color bonds black” etc.

When you are happy with the design of your molecule then write “write molecule.obj” in the console,

125 press Enter and wait ca. 60 seconds until the console reports that the data is saved properly. Check

that your molecule is saved as an MTL- and an OBJ-file in the unfolded Jmol folder. As a check you can

open the “molecule.obj”-file and determine whether the format of your molecule looks satisfying.

STEP 3: WORKING WITH UNITY 130 The software “Unity” is made by Unity Technologies and is

a commercial multi-platform 3D game development tool and

can be used at all levels of computer skills. It has a free

version for people who are not using Unity at a professional

135 level. Unity runs on Windows and macOS but the app is

only made for “Android” devices in this paper.

Download “Unity Hub”, create an account and activate the

license. Install the program “Unity 2019.2.4f1” and in “Add

modules to your installation” select “Android Build

140 Support” and the two subprograms “Android SDK” and

“Open JDK”! The subprograms becomes visible by clicking at the grey arrow next to “Android Build

Support”.

Journal 5/18/21 Page 6 of 14

When Unity is downloaded then create a new 3D project under “Projects” in Unity Hub. A new project is

now open. Click (bottom left) at Project→Create→Folder and name the new folder “molecule”.

145

Now, drag the molecule.MTL file into the new folder “molecule” in Unity and then the molecule.OBJ file.

The order is important! The MTL file stores the design and the OBJ file stores the geometry. A new small

box showing your molecule should appear in the folder “molecule” in Unity. Drag the box onto the “Scene

window” (in the middle of the screen).

150

STEP 4: WORKING WITH VUFORIA

“Vuforia” is an augmented reality software

development kit (AR-SDK) for mobile

155 devices. It recognizes planar images

through a camera of the mobile device and

the recognition leads to the chosen 3D

model is displayed as a part of the real-

world scene.

160 There are different ways to implement

Vuforia in Unity. In the newest version of

Unity (from 2019.2) is automatically

installing Vuforia but it needs to be

activated. Vuforia is activated in Unity by

165 clicking: Edit→Project settings→Player→XR Settings and check the” Vuforia AR supported” box.

For implementing an image into your app go to the website: https://developer.vuforia.com. Click on

“Register” and go through the Creating an Account process. At the same website click on “Develop” in

the Navigation Bar and click “Get Development Key”. Type any name you want in the box “License name”,

confirm the agreement and click at “confirm” in the end. In the new opened window click on the name

170 you entered before and copy the license key in the grey box. Now click at “Target Manager” in Navigation

Bar (next to “License Manager” in the Navigation Bar) and press “Add Database”. Give the database a

Journal 5/18/21 Page 7 of 14

name, select “Device” and click “Create”. A new Database is now created. Now find the image (choose an

image with high quality) you want your app to recognize and save it on your desktop. The image should

be the one you want on your poster and ideally just a picture of your chosen molecule. Return to Vuforia

175 and click at the new Database you just made and press “Add target”. Select the image you just saved on

the desktop and upload it. Add a name for the Database and click “Add”. Finally click at “Download

Database (All)”, select “Unity Editor” as a development platform and save the unity package onto the

Desktop and open it directly after downloading it. When it is opened, a new window in Unity is opening

too. In the new opned window in Unity click “Import”.

180 STEP 5: USING VUFORIA IN UNITY

Step 5 guides you through how to build the AR app

in Unity by using Vuforia.

In the Hierarchy panel (top left) right click at “Main

185 Camera” and press Delete. We'll only use the AR

Camera. Now click at GameObject→ Vuforia

Engine→AR camera and an AR camera shows up

in the Hierarchy panel.

Click at “AR camera” in the Hierarchy panel and a

190 new small window named “Inspector” will appear. In Inspector click at “Vuforia Behaviour (Script)” and

then “Open Vuforia Engine configuration”. In the new window, paste the License Key in the App License

Key Box. You copied the license key in Step 4.

Now, click at GameObject→Vuforia Engine→Image and your Image target shows up in the Scene

window. Back in the Hierarchy panel, click at ImageTarget. In the new Inspector Tab, under the “Image

195 Target Behaviour (Script)”, click at the Database dropdown and select “[name] Database” instead of

empty. Click at “molecule” and in the Hierarchy panel and find the Transform section where you can

modify the position, rotation, and scale of your 3D model relative to the Image target. Change the size

and position of the molecule so it is proportional to the image. Do the same for the AR camera, so the

Journal 5/18/21 Page 8 of 14

small box showing the AR camera’s view is acutally pointing at the molecule and image in the Scene

200 window,

Finally in the Hierarchy panel, click and drag the “molecule” object exactly onto the ImageTarget so it

becomes a child of it. The AR app is now one step from being done. You can choose between building

the AR app on your mobile device (go to Step 6A) or the more complicated way where it is shown how to

publish the app in Goggle Play (go to Step 6B). As mentioened the last step (Step 6B) requries an

205 development account in Google Play and costs 25 $ (2019).

Journal 5/18/21 Page 9 of 14

THE STEP 6A: TRANSFERRING THE DATA TO THE MOBILE DEVICES

Step 6 explains how you install your app

210 on your mobile device. First you have to

allow installing the app on your mobile

device via a USB cable connecting your

device and the computer. At your mobile

device go to “Settings” and find “Build

215 numbers”. Tap several times on “build

numbers” until a new window appears

saying “you are now a developer!”. Go

back to Settings and find “Developer

option” on your mobile device. Scroll

220 down and find and activate “USB

debugging”.

Connect your mobile device via an USB cable to your computer. A new window pops up on your device:

check the box to allow debugging again. In Unity at your computer click:

File→BuildSettings→Android→Switch Platform and then “Build and Run”. Save the project as an apk-

225 file on your computer. The app is now built on your mobile device. If you have any problems with Unity

recognizing the device then it is must likely because your computer needs a USB-adb-driver. This is not

an issue for windows 10, but later versions should install an universal USB-adb-driver. One example

on a free universal USB-adb-driver can be found here:

https://software.intel.com/en-us/xdk/docs/installing-android-debug-bridge-adb-usb-driver-on-

230 windows

When it is completed you can remove the connection to the computer and open your new app on your

mobile device. Point the camera on your image target and enjoy the AR view.

Journal 5/18/21 Page 10 of 14

235 STEP 6B: PUBLISHING THE APP IN GOOGLE PLAY

To publish an app to Google play, the app

must comply with the design rules set by

Google play. One of these rules is that the

240 app must be provided in a package

containing boths package for the 32 bit

and 64 bit architecture. To include 64 bit

package navigate in Unity to the Player

settings by clicking File→BuildSettings

245 →Android→Player Settings. In the Player

setting expand the “Other Options” section

and under configuration locate the

scripting backed and change this from

“mono” to “IL2CPP”. Now activate the

250 “Arm64” check box. When publishing an

app to Google play, the app cannot be

signed with an developer key. To change the signing of the package expand the “Publishing Settings”,

and click the “Keystore manager” button. If this is the first app you are creating then press “Create new

keystore” and fill in the required information. Close the Player Settings and return to Build Settings and

255 check the box “Build app bundle”. This generates the aab-file which is needed to upload to Google play.

To publish an app to Google play a developer account is needed, this requires a one-time payment of 25

$ (2019). Find the website by searching “Google play developer console”, pay for the account and log in.

A new app is created by clicking the “Create application” button and filling in the name and default

language, finish by clicking “Create” and the “Store listing” menu opens. In this menu fill in the following

260 information: 1. Short description, 2. Full description, 3. Upload an image to create the icon for the app,

this image must have a resolution of 512x512. 4. Upload at least two screen shots. 5. Upload a “feature

graphic” this image must have a resolution of 1024x500. 6. Chose the application type from the

Journal 5/18/21 Page 11 of 14

dropdown menu. 7. Choose the category from the dropdown menu. 8. Fill in the contact email for the

app author. 9. As this app have access to the camera on the device you are also required to provide a

265 link to a privacy statement describing how you handle personal informaiton. Finish the Store listing

section by clicking Save.

Open the “App releases” dialog via the left hand navigation menu, create a new app release by clicking

“manage” in the “Production track” section and press “Create release”. In this windows carry out the

following steps. 1. Choose to let Google manage and protect your app signing key. Please note that this

270 means you need the same keystore as used when signing creating the intial .AAB to sign any updates.

2. Browse the .aab file you created in unity in “Android App Bundles and APKs to add” section. 3. Update

the release notes for the app in the “What's new in this release?” section. Complete this step by clicking

Save.

The left hand navigation menu contains the two mandatory steps “Content rating” and “App content”

275 both of these are questionaires pertaining to the nature of the content, complete these by filling in the

required information.

Now navigate to the “Pricing & distribution” and perform the following steps. 1. Choose “FREE” app. 2.

Select the countries that you want the app to be avaliable in. 3. Verify that you app complies with content

guidelines and US export laws by checking the boxes in the “Consent” section. Finalise by pressing “save

280 draft”.

To publish the app navigate back to the App eleases section, press the “Edit release” button in the

“Production track” section. Finalise by clicking the review button. Google play will now review that all

the necesary information have been provide and the app complies with registration. This can take up to

seven days.

285 NOTE : PERFORMANCE IMPROVEMENTS

If very large structures or slower android devices are used performance of the app can be a little

sluggish. It is not withing the scope of this paper to describe the full set of rendering optimizations one

can do in Unity but the following two hints should get you a good portion of the way.

290

Journal 5/18/21 Page 12 of 14

Remove excess strutures:

When Jmol is used to generate the obj-file, an addition sphere is created for each atom, even though

these spheres is located at the same spot as the original atom sphere unity takes up processing power

calculating positions of these. To remove these extra sphere open the unity project again. In the

295 Hierarchy panel expand the hierachy until the structure is expanded, right click at the structure and

select “Unpack prefab”. Now scroll down to the children named “SprereXXX” where XXX is a integer,

select them all and press “delete” this should not affect the 3D structure of the molecule but

signigicantly increase the performance.

300 CONCLUSION

We have described how to make a simple augmented reality app to visualize any 3D chemical model

using free . The method works for all molecules including small molecules, supramolecular

structures, MOFs and biomacromolecules.

305

AUTHOR INFORMATION Corresponding Author *E-mail: [email protected]

ACKNOWLEDGMENTS 310 We acknowledge support from the Danish Council for Independent Research (DFF 4181-00206) and

from the University of Copenhagen.

315

Journal 5/18/21 Page 13 of 14

REFERENCES 320

1. Maier, P.; Klinker, G. Augmented Chemical Reactions: 3D Interaction Methods for Chemistry.

iJOE. 2013, 9 (S8), 80−82. b) Yang, S.; Mei, B.; Yue, X. Mobile Augmented Reality Assisted

Chemical Education: Insights from Elements 4D. J. Chem. Educ. 2018, 1060-1062. c) Wolle,

P.; Müller, M. P.; Rauh, D. Augmented reality in Scientific Publications – Taking the

325 Visulization of 3D Structures to the Next Level. ACS Chem. Biol., 2018, 13, 496-499.

2. Chen, T.-H.; Lee, S.; Flood, A. H.; Miljanic, O. S., How to print a crystal structure model in

3D, CrystEngComm 2014, 16, 5488–5493. B) Scalfani, V. F.; Vaid, T. P. 3D Printed Molecules

and Extended Solid Models for Teaching Symmetry and Point Groups. J. Chem. Educ., 2014,

91, 1174-1180.

330 3. Chichak, K. S.; Cantrill, S. J.; Pease, A. R.; Chiu, S-H.; Cave, G. W. V.; Atwood, J. L.; Stoddart, J. F. Molecular Borromean Rings. Science. 2004, 304, 1308-1312. 4. Cremers, J.; Haver, R.; Rickhaus, M.; Gong, J. Q.; Favereau, L.; Peeks, M. D.; Claridge, T. D.

W.; Herz, L. M.; Anderson, H. L. Template-Directed Synthesis of a Conjugated Zinc Porphyrin

Nanoball. J. Am. Chem. Soc. 2018, 140, 5352-5355.

335 5. Lisbjerg, M.; Jessen, B. M.; Rasmussen, B.; Nielsen, B. E.; Madsen, A. Oe.; Pittelkow, M.

Discovery of a cyclic 6+6 hexamer of D-Biotin and formaldehyde. Chem. Sci. 2014, 5, 2647-

2650.

6. Holton, R. A.; Somoza, C.; Kim, H.-B.; Liang, F.; Biediger, R. J.; Boatman, P. D.; Shindo, M.;

Smith, C. C.; Kim, S.; Nadizadeh, H.; Suzuki, Y.; Tao, C.; Vu, P.; Tang, S.; Zhang, P.; Murthi, K.

340 K.; Gentile, L. N.; Liu, J. H. First Total Synthesis of Taxol. 1. Functionalization of the B Ring.

J.Am. Chem. Soc., 1994, 116, 1597-1598.

Journal 5/18/21 Page 14 of 14