COMS W4172 Introduction
Steven Feiner Department of Computer Science Columbia University New York, NY 10027
www.cs.columbia.edu/graphics/courses/csw4172 (available soon)
January 12, 2021
Sadri et al., ISMAR 2019 Sukan et al., SUI 20161
Please turn on your camera!
2
Feiner, COMS W4172, Spring 2021 Goal
. Learn how to design, develop, and evaluate effective 3D user Google ARCore Trimble Sketchup interfaces . Emphasis on Augmented Reality (AR) and Virtual Reality (VR)
Elvezio, Sukan, Oda, Tversky, & Feiner, SIGGRAPH 2017 Elvezio, Sukan, Oda, Tversky, & Feiner, SIGGRAPH 2017
3
Virtual Reality (VR)
. Computer-generated world of virtual media . 3D . Interactive . Tracked relative to user
Oculus Rift 4
Feiner, COMS W4172, Spring 2021 Augmented Reality (AR)
. Computer-generated world of virtual media . 3D . Interactive . Tracked relative to user . Registered in 3D with the perceptible real world
Meta 2 Dev Kit5
What is Augmented Reality?
. Augmenting the real world with Real computer-generated virtual content (addressing any sense) 1. Combine real and virtual
. Much work addresses only visual AR Virtual 2. Interactive in real time 3. Registered in 3D Note: These three criteria are part of a definition used by most researchers, first articulated by R. Azuma [Presence, 1997] . Unlike virtual reality (VR) Real + Virtual . Supplement rather than replace real world . Design virtual world to complement real
world A. Webster et al., Proc. ASCE Congress on Computing in Civil Engineering, 1996
6
Feiner, COMS W4172, Spring 2021 Combining Real and Virtual
. Variations . Diminished reality . Remove real objects
. Mediated reality S. Mann and J. Fung, ISMR 2001 . Modify real objects Original Diminished Reality Mediated Reality
V. Lepetit and M.-O. Berger, ISMR 2001 https://www.youtube.com/watch?v=DGNtUeNiwBM 7
Why Now?
. Commodity devices are finally sufficiently . Powerful . Small . Inexpensive
2001 Columbia Mobile Wikitude AR, Layar,… Augmented Reality Vuforia, ARKit, ARCore…, System, 1996 – 2008 –
8
Feiner, COMS W4172, Spring 2021 Why Now? Commodity VR HWDs
HTC VIVE Pro Oculus Quest 2
HP Reverb G2 Valve Index 9
Why Now? Commodity AR HWDs
Epson Moverio BT-35E
HTC VIVE Pro (Video See-Through)
Magic Leap 1
Nreal Light Microsoft HoloLens 2 10
Feiner, COMS W4172, Spring 2021 Why Now?
https://trends.google.com 19
Approach
. Lectures . Design, development, and evaluation assignments . Unity 2019–2020 . Game development environment O. Oda et al., Columbia . PTC Vuforia Engine 9.6, ARCore v1.22, ARKit 4 . Camera-based 3D position and orientation tracking . Windows 10 64-bit or macOS 10.13+ for development . Android OS 4.4+ or iOS 11+ for deployment
20
Feiner, COMS W4172, Spring 2021 Approach
. Headsets . Oculus Quest 1 and 2 . Windows Mixed Reality headsets
21
Approach
. Team final projects . Teams proposed by Allison Costa, Cory Robertson, Mark Rosenlof, 2019 . You . Topics proposed by . Domain advisors . You
Adam Carpentieri, Michelle Quin, Kristina Saha, Janane Sekaran, 2020
Maxwell Bennett, Lea Broudo, Sarah Leventhal, 2020
22
Feiner, COMS W4172, Spring 2021 Professor
. Steve Feiner feiner cs.columbia.edu . Director, Computer Graphics and User Interfaces Lab . HCI . 3D UIs . AR . VR . Wearable/mobile computing . Office hours . After class . Monday/Wednesday 1–2pm . Email to arrange other times
23
IAs
. Carmine Elvezio carmine cs.columbia.edu . PhD student, Computer Graphics and User Interfaces Lab . AR, VR, 3D graphics . 3D systems and development environments . Open-source frameworks . Office hours: . Thursday, 10–11am
24
Feiner, COMS W4172, Spring 2021 IAs
. Lea Broudo lrb2167 cs.columbia.edu . CS senior, SEAS . Computer graphics, VR . Animation in Maya . VR and AR in Unity . Office hours: . Tuesday, noon–1pm
25
IAs
. Janane Sekaran jgs2175 cs.columbia.edu . CS MS student, SEAS . VR and AR development . Multiplayer AR games in Unity . Office hours: . Friday, 1pm–2pm
26
Feiner, COMS W4172, Spring 2021 Prereqs
. COMS W4160 (Computer Graphics) or equivalent or
COMS W4170 (User Interface Design) or equivalent or Ask me!
. Math? . Covered in class
27
Textbooks
. J. LaViola Jr., E. Kruijff, R. McMahan, D. Bowman, and I. Poupyrev. 3D User Interfaces: Theory and Practice, 2nd Edition. Addison-Wesley, Boston, 2017, ISBN-13 978-0-13-403432-4 [required] . D. Schmalstieg and T. Höllerer. Augmented Reality: Principles and Practice. Addison-Wesley, Boston, 2016, ISBN-13 978-0-32-188357-5 [required] . For this week through 1/19: LaViola et al. Chaps 1–2 Schmalstieg & Höllerer Chap 1
28
Feiner, COMS W4172, Spring 2021 Lateness Policy
. All assignments due at 1:10pm ET on scheduled due date . Four “late days” allowed during semester for which lateness is not penalized . None can be used for final project . Only one can be used for first assignment . Anything turned in past 1:10pm until 11:59pm ET the next day is one day late . Every (partial) day thereafter that an assignment is late (including weekends and holidays) counts as an additional late day . Absolutely no late work accepted beyond that accounted for by late days . If not done on time, turn in whatever you have completed on time to receive partial credit
31
Academic Honesty Policy
. Department of Computer Science Policies and Procedures Regarding Academic Honesty . www.cs.columbia.edu/education/honesty . All use of GitHub or similar collaborative code dev sites must be done using private repos, open only to appropriate parties . Videos of your assignments must also be private . Infractions will be referred to the CUCS Academic Committee and the Deans
32
Feiner, COMS W4172, Spring 2021 Syllabus www.cs.columbia.edu/graphics/courses/csw4172 (available soon)
. Intro and history
. Design principles (reality, metaphor, magic) Olwal . Case studies . 3D math . Development tools . 3D perception, displays, and devices . Selection . Manipulation . Travel
. Wayfinding Benko and Ishak . Control: menus ↔ multimodal . Symbolic input . Tracking . Design issues Two-handed, whole-body, immersion, presence . Evaluation . Tangible user interfaces
Bell and Höllerer
33
Early History
http://www.nationalmuseum.af.mil/Visit/Museum-Exhibits/Fact- . Flight simulators Sheets/Display/Article/196852/link-trainer/
. Mechanical . Link Trainer, 1930s
TL39 simulator (Moscow Aviation Institute)
. Analog video . Full-motion cameras “flown” over 3D terrain models/photos, 1950s https://airandspace.si.edu/collection-objects/terrain-board- segment-lubbock-texas-flight-simulator
34
Feiner, COMS W4172, Spring 2021 Early History
. Flight simulators . Digital . NASA space program 1960s: General Electric realtime textured surface generator (1964) + realtime General Electric` color solid object scene generator (1967) [40 objects with 240 edges] 1970s: Evans & Sutherland NASA Visual System (1976) [900 polys @ 25 fps]
R. Weinberg. Computer graphics in support of Space Shuttle simulation. Proc. SIGGRAPH 78, 82–86. https://doi.org/10.1145/800248.807375 Evans & Sutherland
35
Early History
. Sensorama (Mort Heilig, 1962) . Stereo film . Audio . Olfactory . Haptics
M. Heilig, Designed 1957 www.mortonheilig.com
See an interview and demo: http://www.youtube.com/watch?v=vSINEBZNCks
36
Feiner, COMS W4172, Spring 2021 Early History
. Sensorama (Mort Heilig, 1962) . Stereo film . Audio . Olfactory . Haptics
M. Heilig, Designed 1957 www.mortonheilig.com
See an interview and demo: http://www.youtube.com/watch?v=vSINEBZNCks
37
Early History
. Timothy Johnson, Sketchpad III (1963) . Built using code from Ivan Sutherland’s 2D Sketchpad . 3D CAD with 2D interaction devices Johnson . Larry Roberts, “Lincoln Wand” (1966) . 3D interaction device . Tracks 3D position of tip . Ultrasonic Roberts
38
Feiner, COMS W4172, Spring 2021 Early History
. Hugo Gernsback, “Teleyeglasses” concept only, Short Wave Craft, Radio Craft, March 1949 https://www.americanradiohistory.com/Archive-Radio- October 1936 Craft/1940s/Radio-Craft-RE-1949-Mar.pdf . Pair of CRTs viewed through lenses
(A stereo head-worn video display, but without interactive graphics or head tracking) . Pictures shown here are later . 1963 photo shows mock-up of version with built-in TV receiver, made that year for Life magazine article A. Eisenstaedt
Life, July 1963 https://archive.org/details/LifeV55n0419630726/page/n7 9/mode/1up 39
Feiner, COMS W4172, Spring 2021