COMS W4172 Case Study: 3D Windows/Desktops 2 Steven Feiner Department of Computer Science Columbia University New York, NY 10027 www.cs.columbia.edu/graphics/courses/csw4172 February 4, 2021 1 The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager . Active task is on stage (far wall) . User can . Move tasks to desired locations on room surfaces . Select a task to make it the active task . Study shows users . Prefer tasks on walls, not floor/ceiling . Remember near–far order better than left–right order 2 Feiner, COMS W4172, Spring 2021 The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager . Navigation . Designed to avoid confusion of unconstrained 3D travel . User can move only . forward or backward . one room at a time . always facing forward 3 The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager . Navigation . On-screen Only user icon is visible until mouse controls hovers over it . Keyboard accelerators 4 Feiner, COMS W4172, Spring 2021 The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager . Glancing to left reveals palette of apps and docs . Add to existing task or create new task . Designed to exploit spatial memory 5 The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager . Glancing to right reveals file browser 6 Feiner, COMS W4172, Spring 2021 The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager . Task windows organized into . Selected set . Loose stack . Can be moved anywhere on stage, scaling based on distance . On stands . Ordered stack . Inactive windows . On podium 7 The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager . Selection . Replace or add to selected set . Automated layout . Move selected window to loose stack or ordered stack 8 Feiner, COMS W4172, Spring 2021 The Task Gallery Microsoft https://www.microsoft.com/en-us/research/publication/the-task-gallery-a-3d-window-manager 1:38-3:36 https://www.youtube.com/watch?v=4uCt2criawQ 9 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d . Originally internal research testbed, 2003 . Then java.net open source project for Linux/Solaris https://www.youtube.com/watch?v=JXv8VlpoK_g 2:10 10 Feiner, COMS W4172, Spring 2021 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d . Avoids 3D navigation by supporting only rotation among a surrounding set of desktops . Switching virtual desktops by panning 11 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d . Avoids 3D navigation by supporting only rotation among a surrounding set of desktops . Switching virtual desktops by panning 12 Feiner, COMS W4172, Spring 2021 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d . Avoids 3D navigation by supporting only rotation among a surrounding set of desktops . Switching virtual desktops by panning 13 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d . Browsing a 3D jukebox 14 Feiner, COMS W4172, Spring 2021 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d . Rotating a window 15 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d . Rotating a window 16 Feiner, COMS W4172, Spring 2021 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d . Rotating a window 17 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d . Rotating a window to interact with info on back 18 Feiner, COMS W4172, Spring 2021 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d . Adding a note on the back of a window 19 Project Looking Glass Sun https://web.archive.org/web/20130609081034/https://java.net/projects/lg3d . Viewing windows at an angle with labeled edges 20 Feiner, COMS W4172, Spring 2021 BumpTop A. Agarawala and R. Balakrishnan www.bumptop.com . Originally research project (2006), later product, then acquired by Google (2010), now open source . Piles . Selective use of physics . Pen-based interaction 22 BumpTop A. Agarawala and R. Balakrishnan www.bumptop.com http://www.youtube.com/watch?v=M0ODskdEPnQ . Piles . Selective use of physics . Pen-based interaction 23 Feiner, COMS W4172, Spring 2021 SpaceTop J. Lee, A. Olwal, H. Ishii, C. Boulanger, CHI 2013 . Transparent display: luminous (OLED) vs. occlusive with true black (LCD) . Depth cameras track head and hands . Main window appears in front . When user raises hands from kbd, main window slides down to allow user to select another window from “virtual cabinet” behind it to take its place http://www.leejinha.com/spacetop.html https://www.youtube.com/watch?v=b8MQxD0ozlc http://dx.doi.org/10.1145/2470654.2470680 https://vimeo.com/130447706 25 SpaceTop J. Lee, A. Olwal, H. Ishii, C. Boulanger, CHI 2013 . Transparent display: luminous (OLED) vs. occlusive with true black (LCD) . Depth cameras track head and hands . Main window appears in front . When user raises hands from kbd, main window slides down to allow user to select another window from “virtual cabinet” behind it to take its place http://www.leejinha.com/spacetop.html https://www.youtube.com/watch?v=b8MQxD0ozlc http://dx.doi.org/10.1145/2470654.2470680 https://vimeo.com/130447706 26 Feiner, COMS W4172, Spring 2021 Personal Cockpit B. Ens, R. Finnegan, P. Irangi, CHI 2014 . Exploration of layout and interaction with arrays of windows in AR . Accounting for . Asymmetric interaction around the body . Transfer between body-fixed and world-fixed (1:14–1:31) . Limited FOV (2:08–2:50) http://hci.cs.umanitoba.ca/publications/details/personal-cockpit https://www.youtube.com/watch?v=L0ZjmEP-c1E 27 Microsoft, Apple, Linux, Android . Compositing window managers (e.g., Microsoft “Desktop Window Manager”) appear 2001–2007 . Each window is rendered to a separate off-screen buffer Flip 3D [Windows . Window manager 7, but not 8…] composites buffers to create final desktop, so user can 3D dock [OS X . View one window’s Leopard,…] pixels through another window (and process the pixels; e.g., to blur) . 3D possibilities . Texture-map 3D polygon(s) with a window’s texture Cover Flow . Transform window’s [was in iP{hone,od}, Os X] polygon(s) in 3D 30 Feiner, COMS W4172, Spring 2021 Full 3D vs. Desktop 3D . Will there be an advantage for “3D windows” in a “full 3D” environment (large FOV, immersive,…)? 32 2D Windows in 3D Space Redux . Industrial research reinvigorated by commercial AR/VR HWD dev kits and products Microsoft HoloLens (composited image) Meta 2 33 Feiner, COMS W4172, Spring 2021 2D Windows in 3D Space Redux vSpatial https://www.vspatial.com/ 34 2D Windows in 3D Space Redux Facebook Infinite Office https://www.youtube.com/watch?v=5_bVkbG1ZCo 35 Feiner, COMS W4172, Spring 2021 2D Windows in 3D Space Redux Facebook Infinite Office https://www.youtube.com/watch?v=5_bVkbG1ZCo 36 2D Windows in 3D Space Redux Facebook Infinite Office https://www.youtube.com/watch?v=5_bVkbG1ZCo 37 Feiner, COMS W4172, Spring 2021.