Zhengyou Zhang Research Manager/Principal Researcher Research, Redmond, WA

Jian Sun’s team at MSRA

Very many sources of Image variability

[Slide credit : John Winn] Scene type Street scene Scene geometry

[Slide credit : John Winn] Scene type Street scene Scene geometry Sky Sidewalk Bicycle Bollard Object classes Building×3 Tree×3 Car×5 Road Person×4 Bench

[Slide credit : John Winn] Scene type Street scene Scene geometry Sky Sidewalk Bicycle Bollard Object classes Building×3 Tree×3 Car×5 Road Person×4 Bench Object position Object orientation

[Slide credit : John Winn] SceneScene type type Street scene SceneScene geometry geometry ObjectObject classes classes ObjectObject position position ObjectObject orientation orientation Object shape

[Slide credit : John Winn] SceneScene typetype SceneScene geometrygeometry ObjectObject classesclasses ObjectObject positionposition ObjectObject orientationorientation ObjectObject shapeshape Depth/occlusions

[Slide credit : John Winn] SceneScene typetype SceneScene geometrygeometry ObjectObject classesclasses ObjectObject positionposition ObjectObject orientationorientation ObjectObject shapeshape Depth/occlusionsDepth/occlusions Object appearance

[Slide credit : John Winn] Scene type Scene geometry Object classes Object position Object orientation Object shape Depth/occlusions Object appearance Illumination Shadows

[Slide credit : John Winn] Scene type Scene geometry Object classes Object position Object orientation Object shape Depth/occlusions Object appearance Illumination Shadows

[Slide credit : John Winn] Scene type Scene geometry Object classes Object position Object orientation Object shape Depth/occlusions Object appearance Illumination Shadows Motion blur Camera effects

[Slide credit : John Winn]

Collaborative Office Space

Microsoft Hub ViiBoard: Vision-enhanced Immersive Interaction with Touch Board

Experimental Setup Surface Hub Display

Kinect

Big Touch Board (Surface Hub) + RGB-D Sensor () leads to more natural and immersive interaction with touch boards VTouch ImmerseBoard Natural and Rich Interaction Beyond Immersive Remote Collaboration Touch

reference point same space gaze intention VTouch

(A) (B) (A) (B)

(C) (D)

(A) (B) (A) (B) (C)

Menu Buttons Menu Buttons

(A) (B)

Preference for Vision-enabled Importance for Vision- UI enabled UI

Body Following Hand Gesture

Hover

Distinguishing Hands Distinguishing Users

NOT Prefer Strongly Strongly Prefer Disagree Agree Vision-enabled UI is easy to use and remember

Body Following Hand Gesture

Hover

Distinguishing Hands Distinguishing Users

Strongly Strongly Disagree Agree ImmerseBoard Person Person 1 2 Face, Eye gaze, Gestures, Proxemics, etc Content Creation Person Person 1 2 Face

Content Content Creation Content Creation RGBD Sensor (Kinect) + Touch Board (Surface Hub) = Immersive Remote Collaboration

as if writing on a physical whiteboard side-by-side

• Seeing the reference point • Sharing the same space • Being aware of gaze • Predicting intention Side-by-side writing on a whiteboard on a mirror ImmerseBoard: Implemented Conditions

Setup Hybrid

Mirror Tilt Board

Big Touch Board (Surface Hub) + RGB-D Sensor (Kinect) leads to more natural and immersive interaction with touch boards

VTouch ImmerseBoard Natural and Rich Interaction Beyond Touch Immersive Remote Collaboration

reference point same space gaze intention

Varun Ramakrishna Richard Stebbing Aaron Hertzmann Sameh Khamis Toby Sharp David Kim Cem Keskin Christoph Rhemann Duncan Robertson Yichen Wei Jonathan Taylor Daniel Freedman Jamie Shotton Eyal Krupka Ido Leichter Andrew Fitzgibbon Alon Vinnikov Shahram Izadi Reinitializer Batch Rendering

... Hand Detector Region of Interest …

Stochastic Optimizer Batch Golden Energy Computation

Understanding Reality for Generating Credible Augmentations

Microsoft Research [With Shapovalov et al. CVPR ’12 ]

Inference Machine = Extension of Random Forests

Colours represent different object categories [Silberman, Shapira, Gal, Kohli, ECCV 2014] [Kim, Kohli, Saverese, ICCV 2013] [Silberman, Hoiem, Kohli, Fergus, ECCV 2012]

Interacting with objects requires understanding of support relationships!!

Can I move the book? [With Oxford Brookes, Shahram Izadi, TOG 2015]

Video