COMS W4172 Perception, Displays, and Devices 3

Steven Feiner Department of Computer Science Columbia University New York, NY 10027

www.cs.columbia.edu/graphics/courses/csw4172

February 16, 2021

Stereoscopic Viewing Wheatstone, 1838 Passive . Spatial multiplexing . “Free viewing” http://www.luminous-lint.com/app/image/06751152706639519425853376/ . Stereoscope . Presents each eye with its own view

Brewster, 1839 Holmes, 1861

http://www.gilai.com/product_763/Holms-Wood-Stereoscope-with-Green-Velvet-Lining-and-Stereocard.

Feiner, COMS W4172, Spring 2021 Stereoscopic Viewing Passive . Spatial multiplexing . “Free viewing” . Stereoscope presents each eye with its own view

What’s wrong with these pictures?

Feiner, COMS W4172, Spring 2021 Commercial versions (w/o camera support): • www.amazon.com/Hasbro-Viewer-touch- Stereoscopic Viewing iPhone-Black/dp/B004T7VI2Y • www.samsung.com/global/galaxy/gear-vr For commodity • arvr.google.com/cardboard • arvr.google.com/daydream/smartphonevr Passive mobile devices • holokit.io/ (optical see-through)

. Spatial multiplexing . VR: Stereoscopic viewer for smartphone/tablet . AR: 1. Cheat: Separate left/right stereoscopic views of virtual objects combined with L–R shifted copies of a monoscopic camera view 2. Full video see-through stereo using additional camera lens (e.g., www.kula3d.com/kula-bebe) Stereo viewer used with Columbia’s Goblin XNA and Nokia Lumia 800 3. Optical see-through stereo using 2012 additional optics (e.g., holokit.io)

Google Cardboard, 2014 https://arvr.google.com/cardboard/

Stereo viewer for Nokia Lumia 800 courtesy of USC ICT MxR Lab, 2012 https://web.archive.org/web/20190314093037/http://projects.ict.usc.edu/mxr/diy/fov2go-viewer/

Autostereoscopic Viewing Passive . Spatial multiplexing . Parallax barrier approaches . Parallax stereogram (F. Ives, 1901) . Viewer’s eyes need to be positioned properly for each to see only its parts of the pair

http://paulbourke.net/stereographics/lenticular/ 6

Feiner, COMS W4172, Spring 2021 Autostereoscopic Viewing Passive RRRLL L . Spatial multiplexing . Parallax barrier approaches . Parallax panoramagram L R Y B (C. Kanolt, 1915) …Y  Presents different stereo G… pairs at different locations . Can use physically moving mechanical or virtually moving liquid crystal barrier to create active display for tracked user . mrl.nyu.edu/~perlin/demos/autostereo.html

http://paulbourke.net/stereographics/lenticular/ 7

Autostereoscopic Viewing Passive . Spatial multiplexing . Lenticular approach

Note: Refraction (H. Ives, 1930) not modeled properly in figure! . Array of cylindrical lenses . Brighter than barrier

Every stereo postcard you’ve seen 

https://earthx.org/product/turtle-postcard/

http://paulbourke.net/stereographics/lenticular/ 8

Feiner, COMS W4172, Spring 2021 Autostereoscopic Viewing Reconfigurable from mono to stereo . Goal: Display that can switch HTC Evo 3D Nintendo 3DS from Was used in SHARP 3D desktop displays and laptops autostereoscopic 3D to 2D . Solution: Parallax barrier made from switching LC panel that can be turned on or off

http://sharp-world.com/products/lcd_monitor/sharp_lcd/wide_variety/index.html 9

Stereoscopic Viewing Passive . Spectral multiplexing . “Red–cyan” (earlier “red-blue”) anaglyph stereo . Display overlapped left/right images in red/cyan . Can create by  Displaying only red channel of left image and green/blue channels of right image  Converting left (right) image to greyscale and displaying as red (cyan) http://www.stereoscopy.com/3d-images/photography.html

http://mars.jpl.nasa.gov/MPF/mpf/anaglyph-arc.html 10

Feiner, COMS W4172, Spring 2021 Stereoscopic Viewing Passive . Spectral multiplexing . ColorCode 3-D . Display overlapped left/right images in amber/blue  Left eye provides color (mostly)  Right eye provides second eye’s geometry (mostly) . Better color rendition than red/cyan anaglyph . Imagery looks better without eyewear . www.colorcode3d.com . Other color pairs used by other vendors 11

Stereoscopic Viewing Passive . Spectral multiplexing . Infitec . Display overlapped left/right images using different RGB primaries for each eye  Special narrow bandpass filters used on projector and in eyewear . Passive eyewear (expensive) . Metallized screen not needed . Very low crosstalk . Slight L–R color shift . www.infitec.net

http://www.infitec.net

http://infitec.net/wp-content/uploads/2015/04/60.5L_Stereoscopic-laser-display-system-using-six-primary-colors-.pdf 12

Feiner, COMS W4172, Spring 2021 Stereoscopic Viewing Passive . Spectral multiplexing . Infitec Excellence [improved version] . Display overlapped left/right images using different RGB primaries for each eye  Special narrow bandpass filters used on projector and in eyewear R, G, B: one band each for Right; two bands each for Left. Long B/short G and long G/short . Passive eyewear R are combined, totalling four bands for Left. (expensive) Each single-band primary is a metamer of its . Metallized screen two-band primary! not needed . Very low crosstalk . Much lower L–R color shift . www.infitec.net 12 3 4

http://www.infitec.net

https://infitec.net/wp-content/uploads/2015/04/SIDDIG_2011.pdf 13

Stereoscopic Viewing Passive (single image)

. Spectral multiplexing . Chromostereoscopy . Different colors focus at slightly different points on retina (red closer to temple than blue for most people) . Caused by chromatic aberration

Perceived fused depth (blue) Left eye seen from above Perceived fused depth (red)

Actual position of object (black) and optically shifted positions due to red and blue focusing at different offsets on retina. Optical axis Fovea is off center relative to eye’s optical axis. Index of refraction varies with wavelength (red < blue) Exaggerated! http://users.ph.tum.de/cucke/ftp/lectures/netduis.pdf

Feiner, COMS W4172, Spring 2021 Stereoscopic Viewing Passive (single image) . Spectral multiplexing . ChromaDepthTM stereo . Inspired by chromostereoscopy . Refractive/diffractive optics over right eye shift colors differentially to the left (red > blue)

Actual position (black) and optically shifted positions Perceived fused depth (blue) Perceived fused depth (red)

Optics Effect is more extreme if optics are used for both eyes Exaggerated! www.chromatek.com 15

Stereoscopic Viewing Passive (single moving image) . Intensity multiplexing . Pulfrich effect https://the3dmarket.com/products/pulfrich-3d-glasses . Dark filter over one eye causes perceptual delay . Note: Works with moving images only (e.g., http://michaelbach.de/ot/lum-Pulfrich/index.html http://www.youtube.com/watch?v=1mnWI_u_zBg) . Dark filter on eye at leading edge of horizontal motion brings it forward Perceived depth

Left–right motion Right–left motion

Actual position (black) and Actual position (black) and perceived position (grey) perceived position (grey) Perceived depth

Optics Optics

https://youtu.be/1mnWI_u_zBg Use filter over left eye! 16

Feiner, COMS W4172, Spring 2021 Stereoscopic Viewing Passive . Polarization state multiplexing . Linear

(horizontal/vertical) http://reald.com . Circular (left- handed/right-handed)

http://www.barco.com/VirtualReality/en/stereoscopic/passive.asp

Polarized light has the tip of the electric field vector following a particular path 17

Stereoscopic Viewing Passive . Linearly polarized

Polarized light has the tip of the electric field vector following a particular path https://en.wikipedia.org/wiki/Polarizer 18

Feiner, COMS W4172, Spring 2021 See the 19th C. machine Wheatstone designed to demonstrate the propagation of polarized light: https://www.youtube.com/watch?v=bw4R5qXaIww Stereoscopic Viewing Passive . Circularly polarized

Polarized light has the tip of the electric field vector following a particular path https://en.wikipedia.org/wiki/Polarizer 19

Stereoscopic Viewing Passive . Polarization state multiplexing . Linear (rotated by 90°) . Circular (left- handed/right-handed) Linearly . Robust to head roll polarized . Screen must preserve polarization (metallized) Circularly polarized

https://en.wikipedia.org/wiki/Polarizer 21

Feiner, COMS W4172, Spring 2021 Stereo Content Caveats

. Geometry . Avoid window violations: Objects at screen edges that extend in front of the screen and are cut off by the edge . Avoid extreme parallax: Difficult to verge and make L–R cross-talk obvious . Avoid rapid/discontinuous changes in depth over time . Color coding for anaglyph . Avoiding objects whose light is sent to only a single eye after filtration (e.g., red or blue objects in simplistic coding approaches) . Some anaglyph approaches (https://www.flickr.com/groups/anachrome3d/) allow slight leakage of red into the cyan filter to support redness cues in both eyes

Image with front of landscape protruding from screen, violating window edges (NASA Mars Pathfinder)

Image after shifting L and R views horizontally (and cropping out edge portions with only one view) to adjust front of landscape to be behind screen

http://en.wikipedia.org/wiki/Anaglyph_3D

Stereoscopic Viewing Active

. Temporal multiplexing . Liquid crystal shutter eyewear . Right/left lenses alternatively opacify in

synchrony with display of CrystalEyes LC shutter eyewear left/right views  L R L R … . Synchronized to wireless (e.g., infrared, Bluetooth) emitter

https://lennylipton.wordpress.com/2009/02/24/thank-you-puff-the-magic-dragon/ 23

Feiner, COMS W4172, Spring 2021 Stereoscopic Viewing Active

. Temporal multiplexing . Liquid crystal shutter eyewear . Can support multiple users at decreased fps

 L1 L2 R1 R2 … CrystalEyes LC shutter eyewear

Two-user stereo workbench

Left user’s view Right user’s view Right user’s view seen from left

M. Agrawala et al., The Two User Responsive Workbench, SIGGRAPH 1997 24

Stereoscopic Viewing Active

. Temporal multiplexing . Liquid crystal shutter eyewear

. Combined with spatial multiplexing Kitamura et al., IllusionHole to support multiple users at full fps

Two-user stereo workbench

Left user’s view Right user’s view Right user’s view seen from left

Y. Kitamura et al., http://www.icd.riec.tohoku.ac.jp/project/displays-and-interface/IllusionHole/IllusionHole.html 25

Feiner, COMS W4172, Spring 2021 Stereoscopic Viewing Head-Worn Display

. Binocular head-worn Sutherland (worn by Vickers), ca. 1970 displays (HWDs)—AKA head-mounted displays (HMDs) . Head-tracked

. Private Saab, ca. 2004 . Opaque or see-through

Meta, 2016 26

Classification of AR Displays R. Raskar (MIT)

projector

retinal display head- worn display

hand-held display

spatial see-through display

projector

Head-attached Body-attached Spatial

Feiner, COMS W4172, Spring 2021 Classification of AR Displays S. Feiner

. Optical see-through . Projection Display Virtual Virtual Graphics Graphics

Optics Combined in optics

Display Combined in . Video see-through environment

Camera(s) Virtual Graphics . Display can be . Worn Display . Held Combined in . In environment computer

Feiner, COMS W4172, Spring 2021