COMS W4172 Introduction

Steven Feiner Department of Computer Science Columbia University New York, NY 10027

www.cs.columbia.edu/graphics/courses/csw4172 (available soon)

January 12, 2021

Sadri et al., ISMAR 2019 Sukan et al., SUI 20161

Please turn on your camera!

2

Feiner, COMS W4172, Spring 2021 Goal

. Learn how to design, develop, and evaluate effective 3D user Google ARCore Trimble Sketchup interfaces . Emphasis on (AR) and (VR)

Elvezio, Sukan, Oda, Tversky, & Feiner, SIGGRAPH 2017 Elvezio, Sukan, Oda, Tversky, & Feiner, SIGGRAPH 2017

3

Virtual Reality (VR)

. Computer-generated world of virtual media . 3D . Interactive . Tracked relative to user

Oculus Rift 4

Feiner, COMS W4172, Spring 2021 Augmented Reality (AR)

. Computer-generated world of virtual media . 3D . Interactive . Tracked relative to user . Registered in 3D with the perceptible real world

Meta 2 Dev Kit5

What is Augmented Reality?

. Augmenting the real world with Real computer-generated virtual content (addressing any sense) 1. Combine real and virtual

. Much work addresses only visual AR Virtual 2. Interactive in real time 3. Registered in 3D Note: These three criteria are part of a definition used by most researchers, first articulated by R. Azuma [Presence, 1997] . Unlike virtual reality (VR) Real + Virtual . Supplement rather than replace real world . Design to complement real

world A. Webster et al., Proc. ASCE Congress on Computing in Civil Engineering, 1996

6

Feiner, COMS W4172, Spring 2021 Combining Real and Virtual

. Variations . Diminished reality . Remove real objects

. Mediated reality S. Mann and J. Fung, ISMR 2001 . Modify real objects Original Diminished Reality Mediated Reality

V. Lepetit and M.-O. Berger, ISMR 2001 https://www.youtube.com/watch?v=DGNtUeNiwBM 7

Why Now?

. Commodity devices are finally sufficiently . Powerful . Small . Inexpensive

2001 Columbia Mobile Wikitude AR, Layar,… Augmented Reality Vuforia, ARKit, ARCore…, System, 1996 – 2008 –

8

Feiner, COMS W4172, Spring 2021 Why Now? Commodity VR HWDs

HTC VIVE Pro Quest 2

HP Reverb G2 Valve Index 9

Why Now? Commodity AR HWDs

Epson Moverio BT-35E

HTC VIVE Pro (Video See-Through)

Magic Leap 1

Nreal Light Microsoft HoloLens 2 10

Feiner, COMS W4172, Spring 2021 Why Now?

https://trends.google.com 19

Approach

. Lectures . Design, development, and evaluation assignments . Unity 2019–2020 . Game development environment O. Oda et al., Columbia . PTC Vuforia Engine 9.6, ARCore v1.22, ARKit 4 . Camera-based 3D position and orientation tracking . Windows 10 64-bit or macOS 10.13+ for development .  Android OS 4.4+ or  iOS 11+ for deployment

20

Feiner, COMS W4172, Spring 2021 Approach

. Headsets . 1 and 2 . Windows headsets

21

Approach

. Team final projects . Teams proposed by Allison Costa, Cory Robertson, Mark Rosenlof, 2019 . You . Topics proposed by . Domain advisors . You

Adam Carpentieri, Michelle Quin, Kristina Saha, Janane Sekaran, 2020

Maxwell Bennett, Lea Broudo, Sarah Leventhal, 2020

22

Feiner, COMS W4172, Spring 2021 Professor

. Steve Feiner feiner cs.columbia.edu . Director, Computer Graphics and User Interfaces Lab . HCI . 3D UIs . AR . VR . Wearable/mobile computing . Office hours . After class . Monday/Wednesday 1–2pm . Email to arrange other times

23

IAs

. Carmine Elvezio carmine cs.columbia.edu . PhD student, Computer Graphics and User Interfaces Lab . AR, VR, 3D graphics . 3D systems and development environments . Open- frameworks . Office hours: . Thursday, 10–11am

24

Feiner, COMS W4172, Spring 2021 IAs

. Lea Broudo lrb2167 cs.columbia.edu . CS senior, SEAS . Computer graphics, VR . Animation in Maya . VR and AR in Unity . Office hours: . Tuesday, noon–1pm

25

IAs

. Janane Sekaran jgs2175 cs.columbia.edu . CS MS student, SEAS . VR and AR development . Multiplayer AR games in Unity . Office hours: . Friday, 1pm–2pm

26

Feiner, COMS W4172, Spring 2021 Prereqs

. COMS W4160 (Computer Graphics) or equivalent or

COMS W4170 (User Interface Design) or equivalent or Ask me!

. Math? . Covered in class

27

Textbooks

. J. LaViola Jr., E. Kruijff, R. McMahan, D. Bowman, and I. Poupyrev. 3D User Interfaces: Theory and Practice, 2nd Edition. Addison-Wesley, Boston, 2017, ISBN-13 978-0-13-403432-4 [required] . D. Schmalstieg and T. Höllerer. Augmented Reality: Principles and Practice. Addison-Wesley, Boston, 2016, ISBN-13 978-0-32-188357-5 [required] . For this week through 1/19: LaViola et al. Chaps 1–2 Schmalstieg & Höllerer Chap 1

28

Feiner, COMS W4172, Spring 2021 Lateness Policy

. All assignments due at 1:10pm ET on scheduled due date . Four “late days” allowed during semester for which lateness is not penalized . None can be used for final project . Only one can be used for first assignment . Anything turned in past 1:10pm until 11:59pm ET the next day is one day late . Every (partial) day thereafter that an assignment is late (including weekends and holidays) counts as an additional late day . Absolutely no late work accepted beyond that accounted for by late days . If not done on time, turn in whatever you have completed on time to receive partial credit

31

Academic Honesty Policy

. Department of Computer Science Policies and Procedures Regarding Academic Honesty . www.cs.columbia.edu/education/honesty . All use of GitHub or similar collaborative code dev sites must be done using private repos, open only to appropriate parties . Videos of your assignments must also be private . Infractions will be referred to the CUCS Academic Committee and the Deans

32

Feiner, COMS W4172, Spring 2021 Syllabus www.cs.columbia.edu/graphics/courses/csw4172 (available soon)

. Intro and history

. Design principles (reality, metaphor, magic) Olwal . Case studies . 3D math . Development tools . 3D perception, displays, and devices . Selection . Manipulation . Travel

. Wayfinding Benko and Ishak . Control: menus ↔ multimodal . Symbolic input . Tracking . Design issues Two-handed, whole-body, immersion, presence . Evaluation . Tangible user interfaces

Bell and Höllerer

33

Early History

http://www.nationalmuseum.af.mil/Visit/Museum-Exhibits/Fact- . Flight simulators Sheets/Display/Article/196852/link-trainer/

. Mechanical . Link Trainer, 1930s

TL39 simulator (Moscow Aviation Institute)

. Analog video . Full-motion cameras “flown” over 3D terrain models/photos, 1950s https://airandspace.si.edu/collection-objects/terrain-board- segment-lubbock-texas-flight-simulator

34

Feiner, COMS W4172, Spring 2021 Early History

. Flight simulators . Digital . NASA space program  1960s: General Electric realtime textured surface generator (1964) + realtime General Electric` color solid object scene generator (1967) [40 objects with 240 edges]  1970s: Evans & Sutherland NASA Visual System (1976) [900 polys @ 25 fps]

R. Weinberg. Computer graphics in support of Space Shuttle simulation. Proc. SIGGRAPH 78, 82–86. https://doi.org/10.1145/800248.807375 Evans & Sutherland

35

Early History

. (Mort Heilig, 1962) . Stereo film . Audio . Olfactory . Haptics

M. Heilig, Designed 1957 www.mortonheilig.com

See an interview and demo: http://www.youtube.com/watch?v=vSINEBZNCks

36

Feiner, COMS W4172, Spring 2021 Early History

. Sensorama (Mort Heilig, 1962) . Stereo film . Audio . Olfactory . Haptics

M. Heilig, Designed 1957 www.mortonheilig.com

See an interview and demo: http://www.youtube.com/watch?v=vSINEBZNCks

37

Early History

. Timothy Johnson, Sketchpad III (1963) . Built using code from Ivan Sutherland’s 2D Sketchpad . 3D CAD with 2D interaction devices Johnson . Larry Roberts, “Lincoln Wand” (1966) . 3D interaction device . Tracks 3D position of tip . Ultrasonic Roberts

38

Feiner, COMS W4172, Spring 2021 Early History

. Hugo Gernsback, “Teleyeglasses” concept only, Short Wave Craft, Radio Craft, March 1949 https://www.americanradiohistory.com/Archive-Radio- October 1936 Craft/1940s/Radio-Craft-RE-1949-Mar.pdf . Pair of CRTs viewed through lenses

(A stereo head-worn video display, but without interactive graphics or head tracking) . Pictures shown here are later . 1963 photo shows mock-up of version with built-in TV receiver, made that year for Life magazine article A. Eisenstaedt

Life, July 1963 https://archive.org/details/LifeV55n0419630726/page/n7 9/mode/1up 39

Feiner, COMS W4172, Spring 2021