Fast and Optimal Laplacian Solver for Gradient-Domain Image Editing using Green Function Convolution Dominique Beaini, Sofiane Achiche, Fabrice Nonez, Olivier Brochu Dufour, Cédric Leblond-Ménard, Mahdis Asaadi, Maxime Raison Abstract In computer vision, the gradient and Laplacian of an image are used in different applications, such as edge detection, feature extraction, and seamless image cloning. Computing the gradient of an image is straightforward since numerical derivatives are available in most computer vision toolboxes. However, the reverse problem is more difficult, since computing an image from its gradient requires to solve the Laplacian equation (also called Poisson equation). Current discrete methods are either slow or require heavy parallel computing. The objective of this paper is to present a novel fast and robust method of solving the image gradient or Laplacian with minimal error, which can be used for gradient-domain editing. By using a single convolution based on a numerical Green’s function, the whole process is faster and straightforward to implement with different computer vision libraries. It can also be optimized on a GPU using fast Fourier transforms and can easily be generalized for an n-dimension image. The tests show that, for images of resolution 801x1200, the proposed GFC can solve 100 Laplacian in parallel in around 1.0 milliseconds (ms). This is orders of magnitude faster than our nearest competitor which requires 294ms for a single image. Furthermore, we prove mathematically and demonstrate empirically that the proposed method is the least-error solver for gradient domain editing. The developed method is also validated with examples of Poisson blending, gradient removal, and the proposed gradient domain merging (GDM). Finally, we present how the GDM can be leveraged in future works for convolutional neural networks (CNN). Keywords Computer vision · Poisson image editing · seamless cloning · Green function convolution · Gradient Laplacian Solver · gradient domain editing Although computing the gradient or Laplacian of an image is straightforward, the reverse problem of 1 Introduction computing the image from its gradient is a non-trivial task. In fact, this problem requires to solve a In computer vision and signal processing, the images differential equation [1–3] without knowing if an can be interpreted as numerical potentials, especially exact solution exists. When the gradient is computed when there is an interest in their gradient (). For from an image, it always generates a conservative example, early computer vision system algorithms field, meaning that the field can be integrated to relied a lot on numerical gradients and Laplacian [1– obtain a potential (the original image). For gradient 6] to extract important information about edges and domain image editing (GDIE), a non-conservative image boundaries. They are computed using simple perturbation is voluntarily introduced to the gradient, convolution kernel such as Sobel [7–9]. More meaning that the resulting field cannot be integrated recently, there have been growing interest in gradient into an exact solution. domain image editing (GDIE) applications, which Nevertheless, it is still interesting to solve the non- aim at editing or creating images from its gradient conservative gradient since it leads to many gradient- [1–3, 10, 11]. domain editing applications, such as gradient 2 erasing, seamless cloning and vectorization with Simple, fast and optimal gradient/Laplacian diffusion curves [1–3, 10, 11]. Furthermore, Bhat et solver. The implementation that we propose is al. presented a whole framework of gradient-domain simple, requiring only a few lines of code using any image editing with unique and useful applications library that implements the 2D fast Fourier transform such as color filtering and edge sharpening [12]. (FFT). The implementation is also significantly The first method to solve the image Laplacian faster than competing methods since we showed in (also called Poisson’s equation) was presented in Fig. 2 a 170x improvement compared to Tanaka’s 2003 by Perez et al. [2], which proposed to solve the method [3], thanks to our graphics processing units differential equation by iteratively minimizing the (GPU) implementation. We also showed that our variational problem. Other research followed by GFC solver can process 100 images in 1ms using optimizing the computation speed and error [2, 10], Pytorch, making it the fastest method available for others used the Jacobi method [1], and McCann discrete images. Finally, we demonstrated proposes a multi-grid solver [11]. These approaches mathematically in section 2.2 and empirically in converge to the approximate solution, but they are section 3.1.4 that GFC is optimal in the sense that it harder to implement since they are iterative, which is the least-error solver for gradient domain editing. also makes them slower to compute. An alternate Gradient domain merging applications. way of solving the Poisson equation is proposed by Inspired by edge saliency sharpening techniques [12] Tanaka [3] by modifying the Poisson problem into a and recent edge detection methods [16, 17], we closed-form problem using cosine transforms. develop a novel method of reducing texture More recent methods solve this problem by using information and enhancing boundaries contrast. Our methods based on multipoles and Green’s function work proposes the first method to use machine [13–15]. However, they are only implemented for learning edge detectors for this purpose. With our diffusion curves in vector graphics, and not for GFC solver that relies mainly on FFT, we show that gradient domain image editing. Furthermore, the the solver can be implemented in deep learning Green’s function methods [13–15] and the Tanaka libraries such as Tensorflow and Pytorch and can be method [3] propose analytical solutions for leveraged in future works for machine learning continuous space, but we propose a numerical applications. solution developed for a discrete space. Hence, our method is more suitable for discrete images and is demonstrated to have a lower error than Tanaka in 2 Computing the image from its section 3.1.4. gradient or Laplacian The objective of this paper is to present a novel fast and robust method of solving the image gradient To understand how to compute the image from its or Laplacian with minimal error, which can be used gradient field or Laplacian, we first focus on the for gradient-domain editing. mathematical understanding of the Green’s function In the research work presented here, a novel and its ability to solve any Laplacian [18]. We will method is proposed called Green Function show how to find the appropriate Green’s function Convolution (GFC), which allows solving any and how to solve either the gradient or the Laplacian. modified gradient. In the case of a non-conservative Then, we will demonstrate mathematically that using field, the proposed GFC method is proven to find the Green’s function is the optimal tool when there is a best possible approximation in terms of gradient non-conservative perturbation that is added to the error. In fact, we mathematically prove in section 2.2 gradient field. and empirically in section 3.1.4 that GFC is the optimal possible solver for any perturbation added to 2.1 Green’s function to solve the Laplacian the gradient, meaning that gradient domain editing can be done with minimal error. This subsection explains how the Green’s function Our contributions are summarized below: can be used to theoretically solve a Laplacian 3 (Poisson equation) on any signal. First, we define the gradient field of a signal Other GDIE methods proposed using multipoles (image) in equation (1), where ∇ is the nth and Green’s function based solvers [13, 15]. dimension gradient operator. In many applications However, we differentiate ourselves from their work such as computer vision, computing the gradient is [13, 15] by focusing on a purely numerical solution, very simple to do using numerical derivatives such instead of solving the Green’s function analytically. as the Sobel method [7, 19]. However, the reverse The Green’s function mono is given in equation problem of finding the signal (or image) from the (5), with the constant given in equation (6) field defined in (2) is not trivial, since the where Γ is the gamma function and is the curvilinear integral is not always defined. In fact, Euclidean distance [4, 13, 15, 18]. For the other ∫ the integral (2) is only defined in the case of a methods based on the Green’s function [13–15], conservative field. In the case of gradient domain is modified to account for the rectangular image editing (GDIE), the field is modified via a boundary around the image, which is not required for non-conservative perturbation, which renders us since we compute numerically. equation (2) unsolvable. −1 () mono = = (1) () , = 2 (5) −1 ∗ = − ⋅ mono = ⋅ , ∈ ℕ (2) , ≠ 2 − 2 2⁄ Instead of solving the gradient, most approaches = , = 2 (6) focus on solving the Laplacian (also known as (⁄2) Poisson equation) defined in (3). In our previous work [4, 5], we used a physics- = ⋅ = (3) inspired method, which convolved electromagnetic dipoles in the direction of the gradient for partial contour analysis. Those dipole potentials are in Since the Laplacian is a differential equation, we fact the Green’s function of the gradient , meaning propose to solve it using a Green’s function, which that they directly solve the gradient without first is defined as a function that solves a differential computing the Laplacian. The gradient solver using equation via convolution [18]. This definition is is presented in equation (7), where is the expressed in (4), where is the Green’s function number of dimensions ( = 2 for an image) and of the Laplacian ∇, ∇ ⋅ is the divergence operator is the axis of each dimension. Hence, each dipole is and ∗ is the convolution operator. The notation convolved with each component of the gradient.