Geometric Calibration and Image Rectification of Inline Computational Imaging

AIT Inline Computational Imaging: Geometric calibration and image rectification

B. Blaschitz, S. Štolc and S. Breuss

AIT Austrian Institute of Technology GmbH Center for Vision, Automation & Control Vienna, Austria www.ait.ac.at/hpv Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

INLINE COMPUTATIONAL IMAGING: WORKING PRINCIPLE

Multi-line scan camera Camera

IlluminationConstant illumination Inspected object Inspected object

Transport stage Transport stage

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 2 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

INLINE COMPUTATIONAL IMAGING: WORKING PRINCIPLE

Multi-line scan camera

Constant illumination Inspected object

Transport stage

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 3 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

AIT ICI LIGHT FIELD: MULTIPLE VIEWING & ILLUMINATION ANGLES

Scientific xVision Days 2018, 7-8 November | Stuttgart, Germany pin grid array Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

AIT INLINE COMPUTATIONAL IMAGING: INDUSTRIAL USE CASES

Metal parts Electronic parts Product packaging

Coins

Printed circuit boards Measurement

Security print / OVD

1064μm 2381μm

5 Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

ICI SOFTWARE MODULES FOR 2D/3D TASKS

Acquisition Stereo matching Coarse refinement

. Light field data . High speed . Fast discrete acquisition multi-view / depth . Camera, light field denoising / illumination, stereo regularization and transport matching preserving control discontinuities Refined 2D/3D data ICI Sensor System . Fine depth model . Camera (depth map, point . Rectification Feature extraction Fine refinement Illumination cloud) . Transport stage . Depth measurement . Camera lens . Extraction of . Fast continuous confidence undistortion & robust image depth / . Refined 2D texture rectification features regularization images . Transport . Extraction of . Fusion with (all-in-focus, gloss / misalignment local surface additional depth shadow suppression) correction features cues . Fine 2.5D surface map

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 6 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

MODELLING SKEW TRANSPORT Linescan image n

(x3,y3)

Line 1 Sensor (x2,y2) v Line n u

(x1,y1) v u Linescan Lens time image 1 Input Image Stack

Corresponding points in different linescan images have different u-coordinate Goal: Rectified Image Stack (for easier matching) where transport appears aligned

Transport direction

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 7 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

GOAL: RECTIFIED IMAGE STACK (FOR EASIER MATCHING) WHERE TRANSPORT APPEARS ALIGNED

Corresponding points in different linescan images should have same u-coordinate

Rectification

. Camera lens undistortion & uncalibrated calibrated rectification . Transport misalignment correction

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 8 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

CALIBRATION PRINCIPAL Transport direction x‘ t=(t1,t2,t3)

v u

Focal length f

Sensor plane

Camera center

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 9 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

VIRTUAL IMAGE PLANE I‘

x‘ x‘‘ t=(t1,t2,t3)

v u H Coordinate system of new image plane I’

I‘ v‘

u‘

c‘ Parallel to transport direction

f Minimal rotation Same distance to camera center

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 10 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

VIRTUAL IMAGE PLANE Project sensor lines x‘ x‘‘ t=(t1,t2,t3)

v v‘ u u‘ H

I‘ Homography from I to I’ v‘

u‘

c‘

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 11 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

VIRTUAL IMAGE PLANE Resample in I’ x‘‘ x‘‘‘

Equal u’-coordinates H R

x‘‘‘ x‘‘

v‘ Equidistant spacing u‘

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 12 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

VIRTUAL IMAGE PLANE Pull back to I x‘‘‘ x‘‘‘‘ x‘ t=(t1,t2,t3) v u H R H-1 I

v I‘ v‘ u

u‘ c

c‘ f

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 13 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

VIRTUAL IMAGE PLANE Interpolate color values x‘‘‘ x‘‘‘‘ x‘

v u H R H-1

M(u,v) is the color value at position (u,v)

(u-1,v) (x‘‘‘‘,v) x=(u,v)

1 - s s

v u

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 14 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

CALIBRATION IMPROVES STEREO MATCHING

Acquisition Stereo matching Coarse refinement

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 15 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

IMPROVED CORRESPONDENCE ANALYSIS

Stereo matching

. High speed multi-view / light field stereo matching …

Refined 2D/3D data . Fine depth model Rectification Feature extraction (depth map, point cloud) . Depth measurement . Camera lens . Extraction of confidence undistortion & robust image . Refined 2D texture rectification features images . Transport . Extraction of (all-in-focus, gloss / misalignment local surface shadow suppression) correction features . Fine 2.5D surface map

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 16 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

SCENE COMPARISON – IMAGE GRADIENT

Uncalibrated Calibrated intensity intensity pixel pixel Maximal Maximal value value z z denoising denoising depth depth Reconstructed after Reconstructed after 17 Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

SCENE COMPARISON – 3D RECONSTRUCTION

Uncalibrated Calibrated

18 Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

EXAMPLE: STAIRCASE – INPUT LIGHTFIELD STACK

CAD model of the 3d printed model staircase (print tolerance approx. 0.0615mm)

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 19 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

EXAMPLE: STAIRCASE – 3D RECONSTRUCTION

Distance from camera center

Height difference of two steps

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 20 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

EXAMPLE: EURO CENT COINS – IMAGE STACK

10 Euro Cent coin 2 Euro Cent coin Diameter 19.75mm Diameter 18.75mm Thickness 1,93mm Thickness 1,67mm

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 21 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

EXAMPLE: EURO CENT COINS - UNCALIBRATED

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 22 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

EXAMPLE: EURO CENT COINS - CALIBRATED

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 23 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

EXAMPLE: EURO CENT COINS – ERROR MEASURE

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 24 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

TAKE-HOME MESSAGE

New calibration method turns AIT‘s Inline Computational Imaging system, which uses a single multi-line scan camera to generate 3d light field stacks into • a measurement device in µm-scale • at industrial inline production speed

For industrial applications, scientific work and our patents visit www.ait.ac.at/hpv

High speed transport

Scientific Vision Days 2018, 7-8 November | Stuttgart, Germany 25 Blaschitz et al. – Geometric calibration and image rectification of Inline Computational Imaging

THANK YOU FOR YOUR ATTENTION!

Bernhard Blaschitz [email protected]