COMS W4172 Introduction Steven Feiner Department of Computer Science Columbia University New York, NY 10027 www.cs.columbia.edu/graphics/courses/csw4172 (available soon) January 12, 2021 Sadri et al., ISMAR 2019 Sukan et al., SUI 20161 Please turn on your camera! 2 Feiner, COMS W4172, Spring 2021 Goal . Learn how to design, develop, and evaluate effective 3D user Google ARCore Trimble Sketchup interfaces . Emphasis on Augmented Reality (AR) and Virtual Reality (VR) Elvezio, Sukan, Oda, Tversky, & Feiner, SIGGRAPH 2017 Elvezio, Sukan, Oda, Tversky, & Feiner, SIGGRAPH 2017 3 Virtual Reality (VR) . Computer-generated world of virtual media . 3D . Interactive . Tracked relative to user Oculus Rift 4 Feiner, COMS W4172, Spring 2021 Augmented Reality (AR) . Computer-generated world of virtual media . 3D . Interactive . Tracked relative to user . Registered in 3D with the perceptible real world Meta 2 Dev Kit5 What is Augmented Reality? . Augmenting the real world with Real computer-generated virtual content (addressing any sense) 1. Combine real and virtual . Much work addresses only visual AR Virtual 2. Interactive in real time 3. Registered in 3D Note: These three criteria are part of a definition used by most researchers, first articulated by R. Azuma [Presence, 1997] . Unlike virtual reality (VR) Real + Virtual . Supplement rather than replace real world . Design virtual world to complement real world A. Webster et al., Proc. ASCE Congress on Computing in Civil Engineering, 1996 6 Feiner, COMS W4172, Spring 2021 Combining Real and Virtual . Variations . Diminished reality . Remove real objects . Mediated reality S. Mann and J. Fung, ISMR 2001 . Modify real objects Original Diminished Reality Mediated Reality V. Lepetit and M.-O. Berger, ISMR 2001 https://www.youtube.com/watch?v=DGNtUeNiwBM 7 Why Now? . Commodity devices are finally sufficiently . Powerful . Small . Inexpensive 2001 Columbia Mobile Wikitude AR, Layar,… Augmented Reality Vuforia, ARKit, ARCore…, System, 1996 – 2008 – 8 Feiner, COMS W4172, Spring 2021 Why Now? Commodity VR HWDs HTC VIVE Pro Oculus Quest 2 HP Reverb G2 Valve Index 9 Why Now? Commodity AR HWDs Epson Moverio BT-35E HTC VIVE Pro (Video See-Through) Magic Leap 1 Nreal Light Microsoft HoloLens 2 10 Feiner, COMS W4172, Spring 2021 Why Now? https://trends.google.com 19 Approach . Lectures . Design, development, and evaluation assignments . Unity 2019–2020 . Game development environment O. Oda et al., Columbia . PTC Vuforia Engine 9.6, ARCore v1.22, ARKit 4 . Camera-based 3D position and orientation tracking . Windows 10 64-bit or macOS 10.13+ for development . Android OS 4.4+ or iOS 11+ for deployment 20 Feiner, COMS W4172, Spring 2021 Approach . Headsets . Oculus Quest 1 and 2 . Windows Mixed Reality headsets 21 Approach . Team final projects . Teams proposed by Allison Costa, Cory Robertson, Mark Rosenlof, 2019 . You . Topics proposed by . Domain advisors . You Adam Carpentieri, Michelle Quin, Kristina Saha, Janane Sekaran, 2020 Maxwell Bennett, Lea Broudo, Sarah Leventhal, 2020 22 Feiner, COMS W4172, Spring 2021 Professor . Steve Feiner feiner cs.columbia.edu . Director, Computer Graphics and User Interfaces Lab . HCI . 3D UIs . AR . VR . Wearable/mobile computing . Office hours . After class . Monday/Wednesday 1–2pm . Email to arrange other times 23 IAs . Carmine Elvezio carmine cs.columbia.edu . PhD student, Computer Graphics and User Interfaces Lab . AR, VR, 3D graphics . 3D systems and development environments . Open-source frameworks . Office hours: . Thursday, 10–11am 24 Feiner, COMS W4172, Spring 2021 IAs . Lea Broudo lrb2167 cs.columbia.edu . CS senior, SEAS . Computer graphics, VR . Animation in Maya . VR and AR in Unity . Office hours: . Tuesday, noon–1pm 25 IAs . Janane Sekaran jgs2175 cs.columbia.edu . CS MS student, SEAS . VR and AR development . Multiplayer AR games in Unity . Office hours: . Friday, 1pm–2pm 26 Feiner, COMS W4172, Spring 2021 Prereqs . COMS W4160 (Computer Graphics) or equivalent or COMS W4170 (User Interface Design) or equivalent or Ask me! . Math? . Covered in class 27 Textbooks . J. LaViola Jr., E. Kruijff, R. McMahan, D. Bowman, and I. Poupyrev. 3D User Interfaces: Theory and Practice, 2nd Edition. Addison-Wesley, Boston, 2017, ISBN-13 978-0-13-403432-4 [required] . D. Schmalstieg and T. Höllerer. Augmented Reality: Principles and Practice. Addison-Wesley, Boston, 2016, ISBN-13 978-0-32-188357-5 [required] . For this week through 1/19: LaViola et al. Chaps 1–2 Schmalstieg & Höllerer Chap 1 28 Feiner, COMS W4172, Spring 2021 Lateness Policy . All assignments due at 1:10pm ET on scheduled due date . Four “late days” allowed during semester for which lateness is not penalized . None can be used for final project . Only one can be used for first assignment . Anything turned in past 1:10pm until 11:59pm ET the next day is one day late . Every (partial) day thereafter that an assignment is late (including weekends and holidays) counts as an additional late day . Absolutely no late work accepted beyond that accounted for by late days . If not done on time, turn in whatever you have completed on time to receive partial credit 31 Academic Honesty Policy . Department of Computer Science Policies and Procedures Regarding Academic Honesty . www.cs.columbia.edu/education/honesty . All use of GitHub or similar collaborative code dev sites must be done using private repos, open only to appropriate parties . Videos of your assignments must also be private . Infractions will be referred to the CUCS Academic Committee and the Deans 32 Feiner, COMS W4172, Spring 2021 Syllabus www.cs.columbia.edu/graphics/courses/csw4172 (available soon) . Intro and history . Design principles (reality, metaphor, magic) Olwal . Case studies . 3D math . Development tools . 3D perception, displays, and devices . Selection . Manipulation . Travel . Wayfinding Benko and Ishak . Control: menus ↔ multimodal . Symbolic input . Tracking . Design issues Two-handed, whole-body, immersion, presence . Evaluation . Tangible user interfaces Bell and Höllerer 33 Early History http://www.nationalmuseum.af.mil/Visit/Museum-Exhibits/Fact- . Flight simulators Sheets/Display/Article/196852/link-trainer/ . Mechanical . Link Trainer, 1930s TL39 simulator (Moscow Aviation Institute) . Analog video . Full-motion cameras “flown” over 3D terrain models/photos, 1950s https://airandspace.si.edu/collection-objects/terrain-board- segment-lubbock-texas-flight-simulator 34 Feiner, COMS W4172, Spring 2021 Early History . Flight simulators . Digital . NASA space program 1960s: General Electric realtime textured surface generator (1964) + realtime General Electric` color solid object scene generator (1967) [40 objects with 240 edges] 1970s: Evans & Sutherland NASA Visual System (1976) [900 polys @ 25 fps] R. Weinberg. Computer graphics in support of Space Shuttle simulation. Proc. SIGGRAPH 78, 82–86. https://doi.org/10.1145/800248.807375 Evans & Sutherland 35 Early History . Sensorama (Mort Heilig, 1962) . Stereo film . Audio . Olfactory . Haptics M. Heilig, Designed 1957 www.mortonheilig.com See an interview and demo: http://www.youtube.com/watch?v=vSINEBZNCks 36 Feiner, COMS W4172, Spring 2021 Early History . Sensorama (Mort Heilig, 1962) . Stereo film . Audio . Olfactory . Haptics M. Heilig, Designed 1957 www.mortonheilig.com See an interview and demo: http://www.youtube.com/watch?v=vSINEBZNCks 37 Early History . Timothy Johnson, Sketchpad III (1963) . Built using code from Ivan Sutherland’s 2D Sketchpad . 3D CAD with 2D interaction devices Johnson . Larry Roberts, “Lincoln Wand” (1966) . 3D interaction device . Tracks 3D position of tip . Ultrasonic Roberts 38 Feiner, COMS W4172, Spring 2021 Early History . Hugo Gernsback, “Teleyeglasses” concept only, Short Wave Craft, Radio Craft, March 1949 https://www.americanradiohistory.com/Archive-Radio- October 1936 Craft/1940s/Radio-Craft-RE-1949-Mar.pdf . Pair of CRTs viewed through lenses (A stereo head-worn video display, but without interactive graphics or head tracking) . Pictures shown here are later . 1963 photo shows mock-up of version with built-in TV receiver, made that year for Life magazine article A. Eisenstaedt Life, July 1963 https://archive.org/details/LifeV55n0419630726/page/n7 9/mode/1up 39 Feiner, COMS W4172, Spring 2021.