COMS W4172 Case Study: 3D Windows/Desktops 2

Steven Feiner Department of Computer Science Columbia University New York, NY 10027

www.cs.columbia.edu/graphics/courses/csw4172

February 4, 2021

1

The Task Gallery https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d--manager

. Active task is on stage (far wall) . User can . Move tasks to desired locations on room surfaces . Select a task to make it the active task . Study shows users . Prefer tasks on walls, not floor/ceiling . Remember near–far order better than left–right order

2

Feiner, COMS W4172, Spring 2021 The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager

. Navigation . Designed to avoid confusion of unconstrained 3D travel . User can move only . forward or backward . one room at a time . always facing forward 3

The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager

. Navigation . On-screen Only user icon is visible until mouse controls hovers over it . Keyboard accelerators

4

Feiner, COMS W4172, Spring 2021 The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager

. Glancing to left reveals palette of apps and docs . Add to existing task or create new task . Designed to exploit spatial memory 5

The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager

. Glancing to right reveals file browser

6

Feiner, COMS W4172, Spring 2021 The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager

. Task windows organized into . Selected set . Loose stack . Can be moved anywhere on stage, scaling based on distance . On stands . Ordered stack . Inactive windows . On podium

7

The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager

. Selection . Replace or add to selected set . Automated layout . Move selected window to loose stack or ordered stack

8

Feiner, COMS W4172, Spring 2021 The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager

1:38-3:36 https://www.youtube.com/watch?v=4uCt2criawQ

9

Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d

. Originally internal research testbed, 2003 . Then .net open source project for /Solaris

https://www.youtube.com/watch?v=JXv8VlpoK_g 2:10

10

Feiner, COMS W4172, Spring 2021 Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d

. Avoids 3D navigation by supporting only rotation among a surrounding set of desktops

. Switching virtual desktops by panning

11

Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d

. Avoids 3D navigation by supporting only rotation among a surrounding set of desktops

. Switching virtual desktops by panning

12

Feiner, COMS W4172, Spring 2021 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d

. Avoids 3D navigation by supporting only rotation among a surrounding set of desktops

. Switching virtual desktops by panning

13

Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d

. Browsing a 3D jukebox

14

Feiner, COMS W4172, Spring 2021 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d

. Rotating a window

15

Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d

. Rotating a window

16

Feiner, COMS W4172, Spring 2021 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d

. Rotating a window

17

Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d

. Rotating a window to interact with info on back

18

Feiner, COMS W4172, Spring 2021 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d

. Adding a note on the back of a window

19

Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d

. Viewing windows at an angle with labeled edges

20

Feiner, COMS W4172, Spring 2021 BumpTop A. Agarawala and R. Balakrishnan www..com

. Originally research project (2006), later product, then acquired by Google (2010), now open source . Piles . Selective use of physics . Pen-based interaction

22

BumpTop A. Agarawala and R. Balakrishnan www.bumptop.com

http://www.youtube.com/watch?v=M0ODskdEPnQ . Piles . Selective use of physics . Pen-based interaction

23

Feiner, COMS W4172, Spring 2021 SpaceTop J. Lee, A. Olwal, H. Ishii, C. Boulanger, CHI 2013

. Transparent display: luminous (OLED) vs. occlusive with true black (LCD) . Depth cameras track head and hands . Main window appears in front . When user raises hands from kbd, main window slides down to allow user to select another window from “virtual cabinet” behind it to take its place

http://www.leejinha.com/spacetop.html https://www.youtube.com/watch?v=b8MQxD0ozlc http://dx.doi.org/10.1145/2470654.2470680 https://vimeo.com/130447706

25

SpaceTop J. Lee, A. Olwal, H. Ishii, C. Boulanger, CHI 2013

. Transparent display: luminous (OLED) vs. occlusive with true black (LCD) . Depth cameras track head and hands . Main window appears in front . When user raises hands from kbd, main window slides down to allow user to select another window from “virtual cabinet” behind it to take its place

http://www.leejinha.com/spacetop.html https://www.youtube.com/watch?v=b8MQxD0ozlc http://dx.doi.org/10.1145/2470654.2470680 https://vimeo.com/130447706

26

Feiner, COMS W4172, Spring 2021 Personal Cockpit B. Ens, R. Finnegan, P. Irangi, CHI 2014

. Exploration of layout and interaction with arrays of windows in AR . Accounting for . Asymmetric interaction around the body . Transfer between body-fixed and world-fixed (1:14–1:31) . Limited FOV (2:08–2:50)

http://hci.cs.umanitoba.ca/publications/details/personal-cockpit https://www.youtube.com/watch?v=L0ZjmEP-c1E 27

Microsoft, Apple, Linux, Android

. Compositing window managers (e.g., Microsoft “”) appear 2001–2007 . Each window is rendered to a separate off-screen buffer Flip 3D [Windows . Window manager 7, but not 8…] composites buffers to create final desktop, so user can 3D [OS X . View one window’s Leopard,…] pixels through another window (and process the pixels; e.g., to blur) . 3D possibilities . Texture-map 3D polygon(s) with a window’s texture Cover Flow . Transform window’s [was in iP{hone,od}, Os X] polygon(s) in 3D

30

Feiner, COMS W4172, Spring 2021 Full 3D vs. Desktop 3D

. Will there be an advantage for “3D windows” in a “full 3D” environment (large FOV, immersive,…)?

32

2D Windows in 3D Space Redux

. Industrial research reinvigorated by commercial AR/VR HWD dev kits and products

Microsoft HoloLens (composited image)

Meta 2

33

Feiner, COMS W4172, Spring 2021 2D Windows in 3D Space Redux

vSpatial https://www.vspatial.com/ 34

2D Windows in 3D Space Redux

Facebook Infinite Office https://www.youtube.com/watch?v=5_bVkbG1ZCo 35

Feiner, COMS W4172, Spring 2021 2D Windows in 3D Space Redux

Facebook Infinite Office https://www.youtube.com/watch?v=5_bVkbG1ZCo 36

2D Windows in 3D Space Redux

Facebook Infinite Office https://www.youtube.com/watch?v=5_bVkbG1ZCo 37

Feiner, COMS W4172, Spring 2021