AIT Inline Computational Imaging: Geometric calibration and image rectification B. Blaschitz, S. Štolc and S. Breuss AIT Austrian Institute of Technology GmbH Center for Vision, Automation & Control Vienna, Austria www.ait.ac.at/hpv Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging INLINE COMPUTATIONAL IMAGING: WORKING PRINCIPLE Multi-line scan camera Camera IlluminationConstant illumination Inspected object Inspected object Transport stage Transport stage Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 2 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging INLINE COMPUTATIONAL IMAGING: WORKING PRINCIPLE Multi-line scan camera Constant illumination Inspected object Transport stage Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 3 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging AIT ICI LIGHT FIELD: MULTIPLE VIEWING & ILLUMINATION ANGLES y Scientific xVision Days 2018, 7-8 November | Stuttgart, Germany pin grid array Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging AIT INLINE COMPUTATIONAL IMAGING: INDUSTRIAL USE CASES Metal parts Electronic parts Product packaging Coins Printed circuit boards Measurement Security print / OVD 1064μm 2381μm 5 Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging ICI SOFTWARE MODULES FOR 2D/3D TASKS Acquisition Stereo matching Coarse refinement . Light field data . High speed . Fast discrete acquisition multi-view / depth . Camera, light field denoising / illumination, stereo regularization and transport matching preserving control discontinuities Refined 2D/3D data ICI Sensor System . Fine depth model . Camera (depth map, point . Rectification Feature extraction Fine refinement Illumination cloud) . Transport stage . Depth measurement . Camera lens . Extraction of . Fast continuous confidence undistortion & robust image depth / . Refined 2D texture rectification features regularization images . Transport . Extraction of . Fusion with (all-in-focus, gloss / misalignment local surface additional depth shadow suppression) correction features cues . Fine 2.5D surface map Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 6 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging MODELLING SKEW TRANSPORT Linescan image n (x3,y3) Line 1 Sensor (x2,y2) v Line n u (x1,y1) v u Linescan Lens time image 1 Input Image Stack Corresponding points in different linescan images have different u-coordinate Goal: Rectified Image Stack (for easier matching) where transport appears aligned Transport direction Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 7 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging GOAL: RECTIFIED IMAGE STACK (FOR EASIER MATCHING) WHERE TRANSPORT APPEARS ALIGNED Corresponding points in different linescan images should have same u-coordinate Rectification . Camera lens undistortion & uncalibrated calibrated rectification . Transport misalignment correction Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 8 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging CALIBRATION PRINCIPAL Transport direction x‘ t=(t1,t2,t3) v u I v u c Focal length f Sensor plane Camera center Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 9 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging VIRTUAL IMAGE PLANE I‘ x‘ x‘‘ t=(t1,t2,t3) v u H Coordinate system of new image plane I’ I‘ v‘ u‘ c‘ Parallel to transport direction f Minimal rotation Same distance to camera center Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 10 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging VIRTUAL IMAGE PLANE Project sensor lines x‘ x‘‘ t=(t1,t2,t3) v v‘ u u‘ H I‘ Homography from I to I’ v‘ u‘ c‘ f Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 11 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging VIRTUAL IMAGE PLANE Resample in I’ x‘‘ x‘‘‘ Equal u’-coordinates H R x‘‘‘ x‘‘ v‘ Equidistant spacing u‘ Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 12 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging VIRTUAL IMAGE PLANE Pull back to I x‘‘‘ x‘‘‘‘ x‘ t=(t1,t2,t3) v u H R H-1 I v I‘ v‘ u u‘ c c‘ f f Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 13 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging VIRTUAL IMAGE PLANE Interpolate color values x‘‘‘ x‘‘‘‘ x‘ v u H R H-1 M(u,v) is the color value at position (u,v) (u-1,v) (x‘‘‘‘,v) x=(u,v) 1 - s s v u Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 14 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging CALIBRATION IMPROVES STEREO MATCHING Acquisition Stereo matching Coarse refinement . Light field data . High speed . Fast discrete acquisition multi-view / depth . Camera, light field denoising / illumination, stereo regularization and transport matching preserving control discontinuities Refined 2D/3D data ICI Sensor System . Fine depth model . Camera (depth map, point . Rectification Feature extraction Fine refinement Illumination cloud) . Transport stage . Depth measurement . Camera lens . Extraction of . Fast continuous confidence undistortion & robust image depth / . Refined 2D texture rectification features regularization images . Transport . Extraction of . Fusion with (all-in-focus, gloss / misalignment local surface additional depth shadow suppression) correction features cues . Fine 2.5D surface map Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 15 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging IMPROVED CORRESPONDENCE ANALYSIS Stereo matching . High speed multi-view / light field stereo matching … Refined 2D/3D data . Fine depth model Rectification Feature extraction (depth map, point cloud) . Depth measurement . Camera lens . Extraction of confidence undistortion & robust image . Refined 2D texture rectification features images . Transport . Extraction of (all-in-focus, gloss / misalignment local surface shadow suppression) correction features . Fine 2.5D surface map Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 16 Blaschitz Blaschitz et al. Scientific Vision Days 2018, 7 SCENE – Geometriccalibration and image rectification of Inline Computational Imaging COMPARISON - 8 November | Stuttgart, Germany Reconstructed z value after depth denoising Maximal pixel intensity Uncalibrated – IMAGE GRADIENT Reconstructed z value after depth denoising Maximal pixel intensity Calibrated 17 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging SCENE COMPARISON – 3D RECONSTRUCTION Uncalibrated Calibrated 18 Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging EXAMPLE: STAIRCASE – INPUT LIGHTFIELD STACK CAD model of the 3d printed model staircase (print tolerance approx. 0.0615mm) Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 19 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging EXAMPLE: STAIRCASE – 3D RECONSTRUCTION Distance from camera center Height difference of two steps Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 20 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging EXAMPLE: EURO CENT COINS – IMAGE STACK 10 Euro Cent coin 2 Euro Cent coin Diameter 19.75mm Diameter 18.75mm Thickness 1,93mm Thickness 1,67mm Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 21 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging EXAMPLE: EURO CENT COINS - UNCALIBRATED Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 22 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging EXAMPLE: EURO CENT COINS - CALIBRATED Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 23 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging EXAMPLE: EURO CENT COINS – ERROR MEASURE Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 24 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging TAKE-HOME MESSAGE New calibration method turns AIT‘s Inline Computational Imaging system, which uses a single multi-line scan camera to generate 3d light field stacks into • a measurement device in µm-scale • at industrial inline production speed For industrial applications, scientific work and our patents visit www.ait.ac.at/hpv High speed transport Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 25 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging THANK YOU FOR YOUR ATTENTION! Bernhard Blaschitz [email protected] 26.