Creating Augmented Reality USDZ Files to Visualize 3D Objects in the Classroom on Student Phones Babak Sanii*

Supplemental Information for: Creating augmented reality USDZ files to visualize 3D objects in the classroom on student phones Babak Sanii*

Keck Science Department of Claremont McKenna, Pitzer and Scripps colleges. 925 N. Mills Ave, Claremont, CA, USA.

Detailed steps for producing and distributing AR files ...... 2 Downloading Software ...... 2 USDPython (needed for all three methods) ...... 2 PyMol (needed for method 1) ...... 2 MeshLab (needed for method 1)...... 2 Blender (needed for method 2)...... 2 Octave (needed for method 3) ...... 2 Steps to make Augmented Reality Objects ...... 2 Method 1: Importing molecules via PyMol, e.g., from the protein database ...... 3 Method 2: Composing scenes via Blender...... 3 Method 3: Scripting mathematical representations via Octave...... 4 All 3 methods: Making the USDZ file ...... 4 Distributing USDZ files ...... 5 Brief Laboratory Materials...... 6 Detailed steps for producing and distributing AR files

Downloading Software

USDPython (needed for all three methods)

The last step in all the object-generation methods outlined in the manuscript is to convert an\ .OBJ file to a .USDZ file. There are several available ways to do this, including some online paid options, and Python libraries. Our current preferred method is to use Apple’s free

USDPython libraries provided by Apple. They can be downloaded from:

https://developer.apple.com/download/more/?=USDPython

For reference we used version 0.61. Once decompressed note the location of the folder.

PyMol (needed for method 1)

Pymol can be downloaded for free at: https://pymol.org/

But after a trial period it will require a license. For educational purposes free licensing can be found here: https://pymol.org/edu/?q=educational

MeshLab (needed for method 1)

MeshLab can be downloaded for free from http://www.meshlab.net .

Blender (needed for method 2)

Blender can be downloaded for free from https://www.blender.org

Octave (needed for method 3)

Octave can be downloaded for free from:

https://www.gnu.org/software/octave/download.html

Steps to make Augmented Reality Objects Method 1: Importing molecules via PyMol, e.g., from the protein database

1. Start the program PyMol

2. From the “File” menu select “Get PDB”

3. Enter your desired PDB ID code (e.g., 1gfl) and click on “Download”

4. From the “File” menu select “Export Image As…” then “VRML 2”

5. Choose a location and enter a filename (e.g., “1gfl”) and press “save”

6. Start the program MeshLab (you may close or minimize PyMol)

7. From the “File” menu select “Import Mesh…”

8. Select the file you just downloaded (it should have a .wrl extension) and press

“Open”

9. Note: for large files like proteins you will have to wait a moment for it to load.

10. From the “File” menu select “Export Mesh As…”

11. Give it a filename (e.g., “1gfl.wrl.obj”)and the pull-down menu select the format

“Alias Wavefront Object (*.obj)”, note where the file will be saved and press “Save”.

Click “Ok” on the panel of saving options.

12. Follow the steps below titled “Making the USDZ file”

Method 2: Composing scenes via Blender

1. Start the program Blender

2. Compose your scene (there are many tutorials here:

https://www.blender.org/support/tutorials/). Alternatively, open up a scene like

the one included in the supplementary information by going to the “File” menu and

selecting “Open…” and selecting the file (e.g., “nucleus_size.blend”).

3. In the “File” menu select “Export” and then “Wavefront (.obj)”

4. Choose a location (note where the file is) and give it a filename (e.g.,

“nucleus_size.obj”) and click the small “Export OBJ” button in the top-right of the

window. 5. Follow the steps below titled “Making the USDZ file”

Method 3: Scripting mathematical representations via Octave

1. Start the program Octave

2. Generate your 3D object (this will require scripting). Alternatively you can run the

script we included in the supplemental information (makeorbital.m), which is an

amalgam of scripts written by Peter van Alem to generate electron orbitals, and Anders

Sandberg to generate a .obj file. Run the script by being in the same directory as the

script (via the “cd” command) and typing the name of the script (e.g., “makeorbital”).

3. Follow the steps below titled “Making the USDZ file”

All 3 methods: Making the USDZ file

1. On a mac, go to the folder of the USDPython tool you downloaded from Apple.

Double-click on “USD.command” and give it permission to run (you can also right-

click on it and select “open”). It will open a terminal window you can type in. You

may close or minimize MeshLab.

2. In this terminal type “usdzconvert “ but do not press enter.

3. From the Finder (i.e. the File folders) find the .obj file you generated earlier, and

drag that file into the Terminal window. This should add the path to the file to your

command.

4. Back to the Terminal, press the space bar and type your desired name of a file,

adding a “.usdz” extension. (e.g., 1gfl.usdz).

5. Add a space bar and add the following text “-iOS12 -diffuseColor 1,1,1” to make the

resulting file compatible with more phones, and give the object a white color.

6. Your total command should look something like this (with your filenames/paths): usdzconvert /Users/babaksanii/Documents/1gfl.obj 1gfl.usdz -iOS12 -

diffuseColor 1,1,1

Press enter.

7. In the terminal type “open .” to produce a Finder window where the usdz file you

created is saved. You can now open it in Preview to confirm it is a 3D object.

8. Email/Airdrop/Message the file to someone with an iPhone with a recent operating

system (iOS 12).

9. On the phone, open the file and view it as a finger-manipulatable object. Press the

AR button on the top and slowly pan the camera of your phone over the edge of a

table until the AR object appears.

Distributing USDZ files

Distributing the files to students can be achieved multiple ways. Per the instructions above they can be sent directly (e.g., via email). Alternatively they may be hosted on a website.

However, this may require whoever is hosting your website to add a line for the file to be recognized. There are detailed instructions here:

https://webkit.org/blog/8421/viewing-augmented-reality-assets-in-safari-for-ios/

The configuration for a common web serving system, Apache, would be:

AddType model/vnd.usdz+zip usdz # All files ending in .usdz served as USD. 1

The HTML code on the webpage itself should include rel=”ar” as follows:

. 1 Brief Laboratory Materials  Prelab:

o Download and install PyMol: https://pymol.org/2/ . Also register for the PyMol educational license: https://pymol.org/edu/?q=educational/ . When you run pymol it will prompt you to find the license file o Download and install MeshLab: http://www.meshlab.net/#download o Ensure you can successfully start both programs o If you can, bring a 3-button mouse. If not be sure to learn how to right-click with your computer. If your computer does not have a USB-A (the "standard" USB port) and you have the appropriate adapter please bring it.

 Learning goal: passing familiarity with tools to visualize complex biomolecules (and perhaps connect their structures to their functions).

In lab you will have three tasks in the Pymol program:

1. Build a simple molecule 2. View a molecular simulation made by advanced lab students [omitted] 3. Study a DNA-recognition protein from the Protein Data Base

Much of this will be learning the software by experimentation, and can be frustrating. Instructors often won't have "answers" and the software is complex enough that there are usually several ways to achieve any task. It's ok to search online for guidance, and if you ever want to "start fresh" go to the menu File > Reinitialize > Everything. Another useful menu command is Display > Zoom > All, which will let you see everything at once.

1. Build and save a molecule

o Start PyMol o Top-right quadrant: click on the "Builder" button o You can choose any of the three types (chemical, protein, nucleic acid) and experiment as you like, but I recommend following this tutorial first: https://pymolwiki.org/index.php/Builder . Note that on many computers you may select a point by right-clicking on that part of the molecule (on some computers that corresponds to clicking while holding down "control."). o Once you've made a molecule of your liking, explore it from all sides in Pymol. The bottom-right gives you a shortcut for which mouse buttons with which keys control what aspects of the viewing, but experiment with dragging with various mouse buttons and holding down shift and control until you are comfortable with it. o Experiment with ways to depict your molecule. On the right-panel the "H" stands for "hide" - use it to hide everything. Then use the "S" to show Lines, Sticks, or Cartoons, or Spheres. Try each to find your favorite. o With your favorite depiction, find your favorite view, and let's save a picture. . File > Export Image As > PNG o Let's also save a version of the file that keeps all the 3D details by doing . File > Export Image As > VRML2 o If you'd like to save a copy of the session to keep working on it later, do File > Save Session...

2. View a molecular simulation made by advanced lab students [omitted]

3. Study a DNA-recognition protein from the Protein Data Base

o Choose a DNA-recognition protein from the list I made on their webpage (https://www.rcsb.org), and write down its 4-digit PDB (protein data base) code:

PDB_dnacomplexes.pdf [a list of human mixed DNA and protein structures from PDB, generated from a search: dna recognition and Chain Type: mixed and TAXONOMY is just Homo sapiens (human) on https://www.rcsb.org]

o In Pymol, reinitialize o File > Get PDB... and enter the 4 letter code - Pymol will download the model from the database. You are now looking at real crystallographic data - a big part of someone's PhD. As part of your post-lab you will have to look up the paper they published this in, and report on what is known about the structure and function of this protein. o Visually explore the structure, particularly how it recognizes DNA. o Like above, explore its depiction and view angle, and save out both a PNG image and a VRML2 3D file. o Given its complexity, it can be helpful to explore it in Augmented Reality. . Convert the VRML2 file (a .dae file) to an AR-compatible OBJ file by . "File > Import Mesh" in MeshLab . "File > Export Mesh As... " and select the output as an "Alias Wavefront Object (.obj)" and save it. Make sure the filename has no space bars or strange characters, has your name and the 4 letter pdf code in it. . Upload the .obj file to my web server which will convert it to AR. For this to work you must be logged into the Claremont-WPA wireless network. Then upload it at http://bsanii.local/convertAR . . On an iOS device (e.g., iPhone, iPad, etc...) log onto the Claremont-WPA wireless network. On your phone, in Safari, go to: http://bsanii.local/uploads and click on your newly made usdz file. . Use AR to get a better understanding of the binding geometry between the protein and the DNA. This may be helpful to understand the molecules structure/function relationship for the postlab, where you will be reading the paper that revealed this structure. Take a picture of your protein in AR, for your postlab.

 Postlab to be submitted individually. Submit the following images: your built molecule, your protein, and the AR view of your protein.

Look up your protein's PDB entry on http://www.rcsb.org by its 4 letter code. Find the literature that revealed its structure and look up the complete paper. Use that paper to write a short paragraph, on the order of a figure caption, that explains:

o What method was used to determine the protein's structure o What is known about the protein's function o How/if the structure determines the protein's functionality

Include a citation to the paper, using the Nature citation style (https://paperpile.com/s/nature- citation-style/). 1. Jackson, D. Viewing Augmented Reality Assets in Safari for iOS., https://webkit.org/blog/8421/viewing-augmented-reality-assets-in-safari-for-ios/ (2018).