Event-Based Computer Vision at Sony AVS CVPR 2021

Sony Group Corporation

Game & Network Sony Interactive Entertainment Services

Sony Music Group(Global) (Sony Music Entertainment, Sony Music Publishing) Music Sony Music Entertainment Japan

Sony Pictures Entertainment Pictures

Electronics Products & Sony Corporation Solutions

Imaging & Sensing Sony Semiconductor Solutions Solutions

Sony Financial Holdings Financial Services

Image Sensor Sales

Design/development bases Manufacturing bases ALT Semiconductor Finland (Finland) Shanghai Sales bases

Weybridge

Altair Semiconductor France Sony Depthsensing Solutions (France) (Brussel) Seoul

Sony Advanced Visual Sensing (Schlieren) Sony Semiconductor Israel Taiwan Branch San Jose Midokura Ibérica (Hsinchu) (Barcelona)

Taipei

Sony Semiconductor Israel (Israel) Hong Kong

Sony Precision Devices Sony Device Technology (Huizhou) (Thailand) （Huizhou）（Bangkadi） Singapore Optical pickup Image sensor assembly etc.

We built and build Event-Based Computer Vision Algorithms

... Let’s Talk Technology

A temporal contrast event is a tuple of an address with polarity 푝෬ and timestamp 푡Ǎ. Such an event is created when …

푡ም 푡 푂푁 1 푓표푟 න 퐶푥,푦 푡 + 푁ሶ푥,푦 푡 푑푡 ≥ 훩 푇 푗−1 푒 = 푥, 푦, 푝෬, 푡Ǎ : ∃푒 [푗] 푝෬ = 푥,푦 푡ም 푡 푂퐹퐹 0 푓표푟 න 퐶푥,푦 푡 + 푁ሶ푥,푦 푡 푑푡 ≤ 훩 푇 푗−1

… of the temporal contrast… … is below … … the integral since the last timestamp… … and the noise … …or above a threshold

Temporal contrast is the rate of illumination change…

푑 (푙푛 퐼푥,푦 푡 ቁ 푡 1 푑 퐼푥,푦 푡 퐶푥,푦 푡 = = 퐼푥,푦(푡൯ 푑푡 푑푡

… normalized by the absolute intensity… … which is equivalent to the log intensity change rate.

Integrating temporal contrast results in …

푡1 퐼푥,푦 푡1 න 퐶푡 푡 푑푡 = ∆퐶푡 푡 , 푡 = 푙푛 퐼 푡 − 푙푛 퐼 푡 = 푙 푛 푥,푦 푥,푦 0 1 푥,푦 1 푥,푦 0 퐼 푡 푡0 푥,푦 0

… a “temporal contrast step”, … … or a fixed step factor.

… which is a fixed step in log intensity …

The “event magnitude” is … reflectance change 푇 푗 푡 푑푙표푔(퐼(푘, 푡)) 푑푅 푑푅 푑푥 푑푅 푑푦 퐸푀푥,푦 푗 = ∆퐶푥,푦 푡0, 푡1 = න + − − 푑푡 푇[푗−1] 푑푡 푑푡 푑푥 푑푡 푑푦 푑푡 illumination change spatial contrast optical flow

… the temporal contrast step encoded by one event, …

… which in turn is made up of different components.

reflectance change 푇 푗 푑푙표푔(퐼(푘, 푡)) 푑푅 푑푅 푑푥 푑푅 푑푦 퐸푀푥,푦[푗] = න + − − 푑푡 푇[푗−1] 푑푡 푑푡 푑푥 푑푡 푑푦 푑푡 event magnitude illumination change spatial contrast optical flow ?

Entropy: Active Lighting: Reconstruction: Tracking: - Smart Triggering - Structured Light - Imaging - Camera: - Smart Processing - Active Marker Tracking - Mapping - VIO - Visible Light Classification: - World: Communication - Object - User - Objects

푇 푗 푑푙표푔(퐼(푘, 푡)) 푑푅 푑푅 푑푥 푑푅 푑푦 퐸푀푥,푦[푗] = න + − − 푑푡 푇[푗−1] 푑푡 푑푡 푑푥 푑푡 푑푦 푑푡 illumination change

Active Lighting

푇 푗 푑푙표푔(퐼(푘, 푡)) 푑푅 푑푅 푑푥 푑푅 푑푦 퐸푀푥,푦[푗] = න + − − 푑푡 푇[푗−1] 푑푡 푑푡 푑푥 푑푡 푑푦 푑푡 optical flow

Tracking

푇 푗 푑푙표푔(퐼(푘, 푡)) 푑푅 푑푅 푑푥 푑푅 푑푦 퐸푀푥,푦[푗] = න + − − 푑푡 푇[푗−1] 푑푡 푑푡 푑푥 푑푡 푑푦 푑푡 spatial contrast

Reconstruction

Predicted depth frames Predicted depth frames 푡0 푡1 푡2 푡3

Predict depth Generate using events Replace depth 푡0 푡1 푡2 푡3 additional depth measurements 푡0 푡1 푡2 푡3 estimates High-speed depth Trigger depth less often → save power

https://linkedin.com/company/sony-advanced-visual-sensing

Or Have A Look At Sony Jobs Every Now And Then:

https://www.sonyjobs.com/jobs.html