Madeline Wrable GIS Specialist CLSTL’19

pngtree.com 1 Content

• GIS & Data Lab • VR Pilot Project Story • What is VR • VR hard/software • Getting started

pngtree.com 2 Content

• GIS & Data Lab • VR Pilot Project Story • What is VR • VR hard/software • Getting started

pngtree.com 3 Massachusetts Institute of Technology

Getty Images 4 Rotch Library

Getty Images 5 MIT GIS Services: GIS & Data Lab

• GIS Lab • Additional software • GIS data repository, GeoWeb • One-on-one help • Workshops

Credit: Lisa Sweeney 6 Content

• GIS & Data Lab • VR Pilot Project Story • What is VR • VR hard/software • Getting started

pngtree.com 7 VR Pilot Project Story

Adapted from “Reality Bytes: VR and AR” – 8 Matt Bernhardt, MIT Libraries Jesussanz/Getty Creative 8 VR Pilot Project Story

The MIT Libraries Experimental Collections Fund Supports projects using library datasets in new ways.

I used it for a drone-pilot-project. Do you have any ideas?

Adapted from “Reality Bytes: VR and AR” – 9 Matt Bernhardt, MIT Libraries Jesussanz/Getty Creative 9 VR Pilot Project Story

The MIT Libraries Experimental Collections Fund Supports projects using library I wonder if virtual reality (VR) datasets in new ways. would be a good fit…

I used it for a drone-pilot-project. Do you have any ideas?

Adapted from “Reality Bytes: VR and AR” – 10 Matt Bernhardt, MIT Libraries Jesussanz/Getty Creative 10 Content

• GIS & Data Lab • VR Pilot Project Story • What is VR • VR hard/software • Getting started

pngtree.com 11 Facebook F8 developer’s conference, 2015 12 History of Immersive Reality

Breakthroughs

21

Results and Future Directions

34

oculus quest image: facebook

Content

• GIS & Data Lab • VR Pilot Project Story • What is VR • VR hard/software • Getting started VR Pipeline

Content Software Content Creation + Interface Viewer 2D  3D 3D/VR headsets VR Pipeline

Content Software Content Creation + Interface Viewer 2D  3D 3D/VR headsets

Content Viewing Platform Examples • Google Cardboard Mobile Devices • Google Daydream • Samsung Gear VR

• Oculus Rift Laptop/Desktops • HTC Vive

Game Consoles • Sony PlayStation VR

Standalone HMDs • FOVE VR & Eye Tracking • Google Hololens … • … 46 Mobile Device: Google Cardboard

• ~$15 simple headset • Free or paid smartphone apps • “Head in a beachball” content • Phone gyroscopes used to determine view direction • Only click in center of your vision • https://vr.google.com/cardboar d/developers/

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries Mobile Device: Google Daydream

• ~$80 less simple headset • Free or paid smartphone apps • “Head and hand in beachball” • Hand-held wand to separate looking and clicking • https://developers.google.co m/vr/daydream/overview

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries Mobile Device: Samsung Gear VR

• ~$100, requires a Samsung phone • Free or paid smartphone apps • Comes with handheld remote • Limited to Oculus software library • http://developer.samsung.com/str eam-code101/gear-vr

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries Personal Computer: Oculus Rift

• ~$500 hardware kit • Powerful computer required • Free or paid software applications • Cubicle-sized space • One body sensors Handheld wand or controller Headset with speakers • https://developer.oculus.com/

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries Personal Computer : HTC Vive

• ~$800 hardware kit • Powerful computer required • Free or paid software applications • Cubicle or Room-sized space – Two body sensors – Two handheld wands – Headset with speakers • Note: now has a new version (Pro) • https://developer.viveport.com/us/ develop_portal/

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries Console: Sony PlayStation VR

• $300 headset & $300 console • 45 million PlayStation 4 owners makes this an accessible path to VR • Easy adoption • https://www.playstation.com /en-us/develop/

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries Standalone: FOVE VR

• ~$600 hardware kit • Founded by a former game producer at Sony • Headset with eye tracking • https://www.getfove.com/ developers/

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries Standalone: HoloLens

• ~$3000 hardware kit • Mixed Reality • Development kits started selling February 29, 2016 • https://developer.microsoft. com/en-us/windows/mixed- reality/install_the_tools

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries full full reality virtuality mixed reality full full reality virtuality high presence

full full reality virtuality

low presence high presence tethered

full full reality virtuality

untethered low presence high presence vive pro eye tethered pimax vr vive oculus quest oculus rift htc vive cosmos meta 2 windows vr magic leap oculus dk2 hololens gear vr daydream full oculus dk1 full google glass reality virtuality

cardboard

smartphone smartphone smartphone untethered low presence high presence vive pro eye tethered pimax vr vive oculus quest oculus rift htc vive cosmos meta 2 windows vr magic leap oculus dk2 hololens gear vr daydream full oculus dk1 full google glass reality virtuality

cardboard

smartphone smartphone smartphone untethered low presence other tracking optics / displays rendering telecom senses

no tracking: 3DOF larger FOV (~ 200°) greater processor greater bandwidth better ergonomic V greater resolution more efficient GPU V haptic interfaces external tracking: 6DOF greater refresh rate ray tracing ultra low latency dynamic binaural V opto electronics foveated rendering wireless local audio inside out tracking: 6DOF variable focus lenses optimization of V other senses inputs V eye tracking rendering pipeline ultra low latency general locomotion inside out tracking + location new 3D graphic techniques wireless universal tracking: 6DOF + coordinates AI in rendering processes AI in content creation

+ better batteries VR Pipeline

Content Software Content Creation + Interface Viewer 2D  3D 3D/VR headsets VR Pipeline

Content Software Content Creation + Interface Viewer 2D  3D 3D/VR headsets Content Creation Platforms

• Photo Spheres • Photogrammetry • 3D Scanning • 3D Modeling • Game Engines • …

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries Photo Sphere: Ricoh Theta

• ~$300 digital camera • Twin fisheye lenses • Still photographs or video • Includes streaming capability • Suitable for Cardboard or Daydream-style viewers • Output supported by Facebook, Flickr, YouTube, etc

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries Photo Sphere: VUZE

• ~$800 digital camera • Four pairs of fisheye lenses • Still photographs or video • Stereo 3d photo spheres • Suitable for Cardboard or Daydream-style viewers • Output suitable for YouTube

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries Photogrammetry: Agisoft PhotoScan

• $179 Standard & $3,500 Pro (license for only one computer) • Able to reconstruct 3D models from drone acquired photos

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries Photogrammetry: ArcGIS Drone2Map

• Prohibitively expensive without educational licensing, even then ~$1,000 per seat • Part of a spatial software suite • Somewhat lacking in 3D model capabilites

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries 3D Scanning: Sense

• $400 3d scanner • Time of flight & white light scanner • Useful for generating 3d models to import into VR/AR content

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries capturing modeling rendering agisoft photoscan Autodesk Revit Unreal Engine 360 video suite Solidworks CryEngine Adobe Premiere SketchUp Unity ... Rhinoceros ... Autodesk 3DS Max Autodesk Maya Strata Blender Cinema 4D ...

3D Modeling Software

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries interactive

synthetic photogrammetry simulations videogrammetry 3D scanning representing capturing

CGI / Digital immersive Animation video

non interactive hand built proceduraly built AI built

VR Pipeline

Content Software Content Creation + Interface Viewer 2D  3D 3D/VR headsets VR Pipeline

Content Software Content Creation + Interface Viewer 2D  3D 3D/VR headsets Game Engine: Unity

• Free for hobbyists and beginners • All purpose game engine • Poor internal editing options • https://unity3d.com/learn/tutori als/topics/virtual-reality/getting- started-vr-development Game Engine: Unreal Engine

• Free up to $3,000 revenue, then charges a 5% royalty • Game engine authored by Epic Games (BioShock, Assassin’s Creed) • https://docs.unrealengine.com/lat est/INT/Platforms/index.html Game Engine: Cry Engine

• $10/month but no royalty fees • Game Engine designed by the German developer Crytek (Far Cry) • Licensed by Amazon

• http://docs.cryengine.com/pages/v iewpage.action?pageId=25536773 VR Pipeline

Content Software Content Creation + Interface Viewer 2D  3D 3D/VR headsets VR Pipeline Content Creation Software Content Viewer

Content Creation + 2D  3D VR Pipeline Future Direction

Content Creation Platforms

Software Development Kits (SDKs)

Content Viewing Platforms Content

• GIS & Data Lab • VR Pilot Project Story • What is VR • VR hard/software • Getting started MIT GIS Services

• GIS Lab • Additional software • GIS data repository, GeoWeb • One-on-one help • Workshops MIT GIS Services

• GIS Lab » powerful computers & graphics cards (content viewer) • Additional software » 3D modeling software (content creation) • GIS data repository, GeoWeb » available datasets (content) • One-on-one help » exposure to the community • Workshops » teaching experience • Perfect for a VR-pilot-project MIT GIS Services + headset • GIS Lab » powerful computers & graphics cards (content viewer) • Additional software » 3D modeling software (content creation) • GIS data repository, GeoWeb » available datasets (content) • One-on-one help » exposure to the community • Workshops » teaching experience • Perfect for a VR-pilot-project VR Applications

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries Jeremy Bailenson: On good candidates for VR

• Expensive: If it would cost a lot to do something in real life, like visit the statue of David in Florence, Italy, it might make sense to do it virtually. • Dangerous: Climbing Mount Kilimanjaro might be the adventure of a lifetime, but it could also be your last. Several people die on its slopes every year. • Impossible: You can’t travel back in time, grow a third arm, or experience life as a person of a different race or gender. But VR can give you a surprisingly visceral taste of what it would be like if you could. • Rare: You could go whale-watching a dozen times without seeing a humpback breach right next to your boat. Or you could do it once in VR.

Adapted from “Reality Bytes: VR and AR” – Matt Bernhardt, MIT Libraries image: mike kelley image: ubisoft image: gary wright MIT GIS Lab: 3D & VR Applications

Pixabay.com Applications Overview

Source Imagery 3D Software VR Software&Hardware

Drone

Floor plan

Kisspng.com, MIT Libraries, Dreamstime.com, & Pixabay.com The GIS Lab’s drone collected imagery of geological formations near Death Valley National Park, CA.

Pixabay.com Credit: Daniel Sheehan Processed imagery using Agisoft Photoscan software:

Point Cloud Solid Mesh Textured Mesh

Credit: Daniel Sheehan Textured mesh Clicking the center button allows teleporting & imported into Unity Pulling the trigger button allows flying

HTV Vive Useful for those who: Controller • couldn’t go to the field • would like to review findings • analyze the 3D model further

Credit: Madeline Wrable Applications Overview

Source Imagery 3D Software VR Software&Hardware

Drone

Floor plan

Kisspng.com, MIT Libraries, Dreamstime.com, & Pixabay.com Started with a floor plan in AutoCAD

Credit: Madeline Wrable Traced 3D shapes from floorplan…

…and extruded them in AutoCAD

Credit: Madeline Wrable Exported 3D model as an .fbx file Clicking center button allows teleporting and imported it into Unity Useful for space planning and lab design

Credit: Madeline Wrable Workflow Resources

Drone Floorplan • Drone model: Phantom 4 • Floorplans : MIT Facilities • Flight planning: DroneDeploy • 3D software: AutoCAD • Photo stitching: PhotoScan

• Exported as FBX files, and imported into Unity • Used SteamVR & Virtual Reality Toolkit (VRTK)

Credit: Madeline Wrable VR Pipeline Content Creation Software Content Viewer

Content Creation + 2D  3D

Download software, most are & follow the tutorials for display in free for educational purposes available for VR headset Rhino Unity Blender Unreal SketchUp CryEngine Autodesk VirtualReality Toolkit (VRTK)

 Future applications 

• Launch XR checkout • AR/MR applications • Create 360° library tour • Invite community experts • More learning materials

Pixabay.com Tips & Tricks

• Find where you already have some equipment (e.g. computer, graphics card, software) • Make sure whatever plug comes off the headset plugs directly into your device (phone/computer) • Explore other’s apps. & watch VRTK tutorials • Think of a simple application to get started with • Set a date to teach others & make learning materials

32 Game Design Examples

115

Unity Demo

121