Event-Based Computer Vision at Sony AVS CVPR 2021
Total Page:16
File Type:pdf, Size:1020Kb
Event-Based Computer Vision at Sony AVS CVPR 2021 Christian Brändli, CEO, Sony AVS 2 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation So Who Is Sony Semiconductor? Sony Group Corporation Game & Network Sony Interactive Entertainment Services Sony Music Group(Global) (Sony Music Entertainment, Sony Music Publishing) Music Sony Music Entertainment Japan Sony Pictures Entertainment Pictures Electronics Products & Sony Corporation Solutions Imaging & Sensing Sony Semiconductor Solutions Solutions Sony Financial Holdings Financial Services 3 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation We Build The World’s Best Image Sensors Image Sensor Sales 4 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation Who is Sony AVS? Design/development bases Manufacturing bases ALT Semiconductor Finland (Finland) Shanghai Sales bases Weybridge Altair Semiconductor France Sony Depthsensing Solutions (France) (Brussel) Seoul Sony Advanced Visual Sensing (Schlieren) Sony Semiconductor Israel Taiwan Branch San Jose Midokura Ibérica (Hsinchu) (Barcelona) Taipei Sony Semiconductor Israel (Israel) Hong Kong Sony Precision Devices Sony Device Technology (Huizhou) (Thailand) (Huizhou) (Bangkadi) Singapore Optical pickup Image sensor assembly etc. 5 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation And What Does Sony AVS Do? We built and build Event-Based Computer Vision Algorithms 6 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation Enough PR ... ... Let’s Talk Technology 7 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation What Is A Temporal Contrast Event? A temporal contrast event is a tuple of an address with polarity 푝෬ and timestamp 푡Ǎ. Such an event is created when … 푡ም 푡 푂푁 1 푓표푟 න 퐶푥,푦 푡 + 푁ሶ푥,푦 푡 푑푡 ≥ 훩 푇 푗−1 푒 = 푥, 푦, 푝෬, 푡Ǎ : ∃푒 [푗] 푝෬ = 푥,푦 푡ም 푡 푂퐹퐹 0 푓표푟 න 퐶푥,푦 푡 + 푁ሶ푥,푦 푡 푑푡 ≤ 훩 푇 푗−1 … of the temporal contrast… … is below … … the integral since the last timestamp… … and the noise … …or above a threshold 8 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation What is Temporal Contrast? Temporal contrast is the rate of illumination change… 푑 (푙푛 퐼푥,푦 푡 ቁ 푡 1 푑 퐼푥,푦 푡 퐶푥,푦 푡 = = 퐼푥,푦(푡൯ 푑푡 푑푡 … normalized by the absolute intensity… … which is equivalent to the log intensity change rate. 9 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation So What is Encoded By an Event? Integrating temporal contrast results in … 푡1 퐼푥,푦 푡1 න 퐶푡 푡 푑푡 = ∆퐶푡 푡 , 푡 = 푙푛 퐼 푡 − 푙푛 퐼 푡 = 푙 푛 푥,푦 푥,푦 0 1 푥,푦 1 푥,푦 0 퐼 푡 푡0 푥,푦 0 … a “temporal contrast step”, … … or a fixed step factor. … which is a fixed step in log intensity … 10 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation Then What Makes Up a Temporal Contrast Step? The “event magnitude” is … reflectance change 푇 푗 푡 푑푙표푔(퐼(푘, 푡)) 푑푅 푑푅 푑푥 푑푅 푑푦 퐸푀푥,푦 푗 = ∆퐶푥,푦 푡0, 푡1 = න + − − 푑푡 푇[푗−1] 푑푡 푑푡 푑푥 푑푡 푑푦 푑푡 illumination change spatial contrast optical flow … the temporal contrast step encoded by one event, … … which in turn is made up of different components. 11 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation So What To Do With Events reflectance change 푇 푗 푑푙표푔(퐼(푘, 푡)) 푑푅 푑푅 푑푥 푑푅 푑푦 퐸푀푥,푦[푗] = න + − − 푑푡 푇[푗−1] 푑푡 푑푡 푑푥 푑푡 푑푦 푑푡 event magnitude illumination change spatial contrast optical flow ? Entropy: Active Lighting: Reconstruction: Tracking: - Smart Triggering - Structured Light - Imaging - Camera: - Smart Processing - Active Marker Tracking - Mapping - VIO - Visible Light Classification: - World: Communication - Object - User - Objects 12 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation Some Examples 13 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation So What To Do With Events 푇 푗 푑푙표푔(퐼(푘, 푡)) 푑푅 푑푅 푑푥 푑푅 푑푦 퐸푀푥,푦[푗] = න + − − 푑푡 푇[푗−1] 푑푡 푑푡 푑푥 푑푡 푑푦 푑푡 illumination change Active Lighting 14 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation High-Speed 3D 15 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation So What To Do With Events 푇 푗 푑푙표푔(퐼(푘, 푡)) 푑푅 푑푅 푑푥 푑푅 푑푦 퐸푀푥,푦[푗] = න + − − 푑푡 푇[푗−1] 푑푡 푑푡 푑푥 푑푡 푑푦 푑푡 optical flow Tracking 16 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation Very Fast Real-Time Tracking Of Dots 17 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation Better For Fast Motion than Open CV 18 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation And More Robust On Repetitive Structures 19 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation So What To Do With Events 푇 푗 푑푙표푔(퐼(푘, 푡)) 푑푅 푑푅 푑푥 푑푅 푑푦 퐸푀푥,푦[푗] = න + − − 푑푡 푇[푗−1] 푑푡 푑푡 푑푥 푑푡 푑푦 푑푡 spatial contrast Reconstruction 20 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation Sensor Fusion of iToF and EVS for Efficient Depth Sensing Predicted depth frames Predicted depth frames 푡0 푡1 푡2 푡3 Predict depth Generate using events Replace depth 푡0 푡1 푡2 푡3 additional depth measurements 푡0 푡1 푡2 푡3 estimates High-speed depth Trigger depth less often → save power 21 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation To Capture What Our Engineers Are Doing In the Office 22 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation If You Like To Work With The Latest And Greatest Sensors Follow us on LinkedIn: https://linkedin.com/company/sony-advanced-visual-sensing Or Have A Look At Sony Jobs Every Now And Then: https://www.sonyjobs.com/jobs.html 23 16-Jun-21 Research Division 1 – Sony Advanced Visual Sensing AG © 2021 Sony Semiconductor Solutions Corporation .