Correcting Lens Distortions in Digital Photographs Wolfgang Hugemann c 2010 by EVU Abstract The wide-angle lenses (or rather zoom lenses when set to short focal length) typically produce a pronounced barrel distortion. This lens distortion affects damage mappings (i. e. the superposition of damage photographs) as well as perspective rectifications. Lens distortion can however be mostly corrected by applying suitable algorithmic transformations to the digital photograph. The paper presents the algorithms used for this correction, together with programs that perform either the entire task of correction or that allow one to determine the lens correction parameters. The paper concludes with some (rectified) example images and an estimation of the gains in accuracy achieved by applying lens correction algorithms. Introduction By the use of suitable software, lens distortions can be corrected in retrospect, eliminating much Ideally, a photograph should be a perfect pers- of the error produced by unsatisfactory shots. In pective mapping of the photographed scene. the following, we will present mathematical ap- This holds especially when the photograph is fur- proaches to describe lens distortion, present some ther processed into a to-scale representation of programs that will do most of the job for you, the pictured object, as often is the case in acci- and demonstrate the accuracy achieved. dent reconstruction, e. g. for shots of damaged vehicles or of the road surface – the latter often Modelling lens distortion taken from an elevated position and then recti- fied. When transforming picture coordinates into real- To keep lens distortions reasonably small, the world coordinates, we mostly use cartesian sys- general advice is to use a small-angle lens, if tems for both. This coordinate system seems to possible a telephoto lens. In practice, this is fit the problem most naturally, as photographs however often impossible, as the space around are rectangular and some (not very) special set- the photographed object is limited and the re- up situations (so-called nadir or coplanar photo- quired distance to the object cannot be achie- graphs) are simply to-scale mappings of the pho- ved. This holds especially for shots of the road tographed plane. surface, which are mostly taken by use of wide- When describing lens distortions, we should angle lenses in order to cover the desired space. however use a polar coordinate system, with the Furthermore, there are a lot of ‘external’ photo- lens’s main axis as its origin: obviously, the lens graphs, over which the reconstructionist has no is an axially symmetric object, so we should ex- influence on the camera settings. These are pic- pect all distortions to be rotation-symmetric. We tures taken by the people involved in the accident assume the lens’s main axis (the principal axis) or by the police, who – at least in Germany – of- to meet the image plane at an exact right angle, ten have to make do with inadequate equipment. at the principal point. 1 Wolfgang Hugemann In order to describe lens distortions, it is suffi- cient to investigate coplanar photographs, as any additional distortion created by the camera set- up will just be to the perspective and can be attended to in a later step. In coplanar pers- pective mapping, the real-world coordinates are just a fixed multiple of the image coordinates. It is common to denote real-world coordinates by capital letters X, Y and image coordinates by small letters x, y. So for a coplanar perspective a) barrel b) pincushion mapping we arrive at Fig. 1: Common lens distortions [2] X − X0 = c1 (ˆx − x0) (1) Y − Y0 = c2 (ˆy − y0) (2) This offset of the principal point xˆp, yˆc is camera- specific, as it results from manufacturing tole- These equations allow one to define the origins of rances in regard to the mutual mounting of the both coordinate systems freely. Common choices camera sensor and the lens. The distortion func- for the coordinate origin in digital photographs tions f(r), g(r) are however specific for a certain are the upper left corner and the principal point, make and model of camera, if the (zoom) lens which should ideally coincide with the middle is non-interchangeable, as is the case for consu- point of the image. Furthermore, the equations mer and mobile phone cameras. For interchan- consider different scale factors for the x- and y- geable lenses, the distortion is lens-specific, pos- directions, c1 and c2. For digital photographs, sibly needing some correction in regard to the these scale factors are identical, but video came- exact sensor size of the camera it is mounted on. ras might (virtually) use non-square pixels, as is It is common to model the distortion by di- the case in DV cameras. mensionless functions f(r), g(r, α), like in the In the equations above we used x,ˆ yˆ rather than above equations. Moreover, the radius r is of- x, y to denote the image coordinates of the ideal ten normalised to the dimensions of the sensor, perspective mapping: this is the common way to such that f(r) and r are both of magnitude one. denote estimates, and estimates they are, having to be derived from the physical image coordi- Experience shows that the effects of radial lens nates x, y. distortion f(r) exceed those of torsional distor- In order to describe lens distortion, it suffices tion g(r, α) at least by an order of magnitude. to establish the relationship between the physi- Consequently, most software only models radial cal coordinates of a pixel x, y and the coordi- distortion, i. e. tries to determine the functional nates x,ˆ yˆ of the ideal perspective mapping which relationship f(r) for a certain make and model of should be used in the above equations. In the fol- camera, for example, an SLR camera combined lowing, we assume x,ˆ yˆ to refer to the principal with a specific lens. For zoom lenses, the functio- point and x, y to refer to the physical centre of nal relationship f(r) is specific for a certain focal the image, the offset of the principal point being length, i. e. the functional relationship has to be denoted by xˆp, yˆp. We can split lens distortion modelled for different settings of the focal length. into a radial and a tangential (torsional) part Basically, we have to distinguish between, figure 1 [2]: rˆ2 =x ˆ2 +y ˆ2 (3) barrel distortion f(r) < 1 r2 = (x − xˆ )2 + (y − yˆ )2 (4) p p The apparent effect is that of an image rˆ = f(r) r (5) which has been mapped around a sphere or rαˆ = g(r, α) rα (6) barrel. 2 Correcting Lens Distortions in Digital Photographs pincushion distortion f(r) > 1 The visible effect is that lines that do not go through the centre of the image are bowed inwards, towards the centre of the image, like a pincushion. Most consumer cameras show pronounced barrel distortion when set to short focal length. Modelling radial distortion The most common functional approach for f(r) is a polynomial 2 n f(r) = 1 + a1r + a2r + ... + anr (7) Fig. 2: PTlens’s program window (stand-alone version) Optical theory shows that this polynomial should only feature even powers points P0,P1, ..., Pn is known to lie on a straight 2 4 2n line, their images p0, p1, ..., pn should also fall f(r) = 1 + a2r + a4r + ... + a2nr (8) onto a straight line, i. e. satisfy the equation In practice, the number of parameters is often ~pi = ~p0 + %i~e (12) limited to about three, i. e. either Any deviation from this rule has to be attributed 2 3 to lens distortion f(r) = 1 + a1r + a2r + a3r (9) ~pi = f~(~pi) (13) or We need two points to determine the two para- 2 4 6 f(r) = 1 + a1r + a2r + a3r (10) meters defining a straight line. Each additional point will then provide one more equation to de- with some of the coefficients possibly being zero. termine the parameters used in f~(~r). So if our functional approach is Tangential distortion 2 rˆ = (1 + a1r + a2r ) r (14) The most common model incorporating tangen- we would have to provide at least four points on tial distortion is the Brown-Conrady model [6, 5], one straight line in order to determine the two which adds a tangential component to the radial sought parameters a and a . In practice, cali- distortion 1 2 bration programs mostly use a rectangular grid ! ! xˆ x of straight lines, often a chequerboard, to gene- = (1 + a r2 + a r4 + a r6) yˆ 1 2 3 y rate a set of equations and then calculate the mapping parameters by a nonlinear least-squares 2 2 ! 2a4xy + a5(r + 2x ) fit. Some programs generate the set of control + 2 2 (11) a4(r + 2y ) + 2a5xy points on their own, often using pre-defined tem- plates; other programs require the user to select Determining the lens parameters the control points from the image. When determining the lens parameters, all pro- Ready-made solutions grams rely on the same paradigm: the ideal pers- pective mapping should map real world straight Fortunately, there are quite a few programs that lines to straight lines in the image. So if a set of will correct lens distortion in arbitrary photo- 3 Wolfgang Hugemann graphs automatically. These programs rely on the EXIF information embedded in the digital photograph, providing the make, model and fo- cal length. They are shipped with an extensive camera/lens database, providing the correction parameters for a variety of camera/lens combi- nations.