Closed Form for Some Gaussian Convolutions Hossein Mobahi Computer Science & Artificial Intelligence Lab. Massachusetts Institute of Technology Cambridge, MA, USA [email protected] Abstract The convolution of a function with an isotropic Gaussian appears in many contexts such as differential equations, computer vision, signal pro- cessing, and numerical optimization. Although this convolution does not always have a closed form expression, there are important family of func- tions for which closed form exists. This article investigates some of such cases. 1 Introduction Motivation. The convolution of a function with an isotropic Gaussian, aka Weierstrass transform, appears in many contexts. In differential equations, it appears as the solution of diffusion equations [Widder, 1975]. In computer vision and image processing, it is used to blur images for generating multi- scale or scale-space representations [Koenderink, 1984]. In addition, deriva- tives of the blurred images are used for defining visual operations and fea- tures such as in SIFT [Lowe, 1999]. In signal processing, it defines filters for noise reduction [Blinchikoff and Zverev, 1976] and preventing aliasing in sam- pled data [Foley et al., 1990]. Finally, in optimization theory, it is used to simplify a nonconvex objective function by suppressing spurious critical points [Mobahi and Fisher, 2015]. arXiv:1602.05610v2 [math.CA] 7 Mar 2016 Numerical Approximation. In general, the Weierstrass transform does not have a closed form expression. When that is the case, the transformation needs to be computed or approximated numerically. Recent efforts, mainly moti- vated from optimization applications, have resulted in efficient algorithms for such numerical approximations, e.g. by adaptive importance sampling [Maggiar et al., 2015]. Exact Closed Form. Putting the general arguments aside, there are impor- tant family of functions for which the Weierstrass transform does have a closed form. This article investigates two such families, namely, multivariate polyno- mials and Gaussian radial basis functions. These two classes are rich, as they 1 are able to approximate any arbitrary function (under some mild conditions), up to a desired accuracy. In addition, we present a result which provides the closed form Weierstrass transform of a multivariate function, given the closed form transform of its univariate cases. Notation. We use x for scalars, x for vectors, . for the ℓ2 norm, and i for √ 1. We also use , for equality by definition. Wek define the isotropic Gaussian − kernel kσ(x) as below. x 2 dim(x) k k kσ(x) , (√2πσ)− e− 2σ2 . 2 Closed Form Expressions 2.1 Polynomials Given a multivariate polynomial function f : Rn R. We show that the → convolution of f with kσ has a closed form expression. Suppose f consists of m monomials, m f(x) , aj fj (x) , Xj=1 where fj is the j’th term and aj is its coefficient. Each monomial has the , n pd,j form fj (x) Πd=1xd with xd being the d’th component of the vector x, and pd,j being a nonnegative integer exponent of xd in fj . Then we have, m m m n pd,j n [f⋆kσ](x)= aj [fj⋆kσ](x) = ajΠd=1 [ d ⋆kσ](xd) = aj Πd=1u(xd,pd,j, σ) . Xj=1 Xj=1 Xj=1 The function u has the following form (see the Appendix for the proof), x u(x, p, σ) , [p ⋆ k ](x) = (iσ)p h ( ) , σ p iσ where hp is the p’th Hermite polynomial, i.e., 2 p 2 p x d x hp(x) , ( 1) e 2 e− 2 . − dxp The following table lists u(x, p, σ) for 0 p 10. ≤ ≤ 2 p u(x, p, σ) 0 1 1 x 2 σ2 + x2 3 3σ2x + x3 4 3σ4 +6σ2x2 + x4 5 15σ4x + 10σ2x3 + x5 6 15σ6 + 45σ4x2 + 15σ2x4 + x6 7 105σ6x + 105σ4x3 + 21σ2x5 + x7 8 105σ8 + 420σ6x2 + 210σ4x4 + 28σ2x6 + x8 9 945σ8x + 1260σ6x3 + 378σ4x5 + 36σ2x7 + x9 10 945σ10 + 4725σ8x2 + 3150σ6x4 + 630σ4x6 + 45σ2x8 + x10 2 3 4 4 Example Let f(x, y) , x y +λ(x +y ). Then, using the table, [f ⋆kσ](x, y) has the following form, 2 2 2 3 4 2 2 4 4 2 2 4 [f ⋆ kσ] = (σ + x )(3σ y + y )+ λ(3σ +6σ x + x +3σ +6σ y + y ) . 2.2 Gaussian Radial Basis Functions Isotropic Gaussian Radial-Basis-Functions (RBFs) are general function approx- imators [Hartman et al., 1990]. That is, any function can be approximated to a desired accuracy by the following form, 2 m kx−xj k − 2δ2 f(x) , aj e j , Xj=1 n where xj R is the center and δj R+ is the width of the j’th basis function, for j∈ = 1,...,m. We have shown∈ in [Mobahi et al., 2012] that the convolution of f with kσ has the following form, 2 m kx−xj k δj n − 2(δ2 +σ2) [f ⋆ kσ](x)= aj ( ) e j . 2 2 Xj=1 δj + σ q In order to be self-contained, we include our proof as presented in [Mobahi et al., 2012] in the appendix of this article. 2.3 Trigonometric Functions σ2 sin(x) e− 2 sin(x) σ2 cos(x) e− 2 cos(x) 2 1 2σ2 sin (x) 2 1 e− cos(2x) 2 1 − 2σ2 cos (x) 2 1+ e− cos(2x) 2σ2 sin(x)cos(x) e− sin(x)cos(x) 3 Example Let f(x, y) , 2 cos(x) 3 sin(y)+4cos(x) sin(y) 8 sin(x)cos(y)+ 2 − − 4 . Compute [f ⋆ kσ](x, y). We expand the quadratic form and then use the table, as shown below. 2 2 cos(x) 3 sin(y)+4cos(x) sin(y) 8 sin(x)cos(y)+4 (1) − − = 4cos 2(x) + 9 sin2(y)+16cos2(x) sin2(y) + 64 sin2(x)cos2(y)+16 (2) 12 cos(x) sin(y)+16cos2(x) sin(y) 32 cos(x) sin(x)cos(y) (3) − − +16 cos(x) 24 sin(y) cos(x) + 48 sin(y) sin(x)cos(y) 24 sin(y) (4) − − 64 cos(x) sin(y) sin(x) + 32cos(x) sin(y) 64 sin(x)cos(y) (5) − − 2.4 Functions with Linear Argument T This result tells us about the form of [f x ⋆ kσ()](w) given [f ⋆ kσ](y). Lemma 1 Let f : R R have well-defined Gaussian convolution f˜(y ; σ) , → T [f ⋆kσ](y). Let x and w be any real vectors of the same length. Then [f x ⋆ T kσ()](w)= f˜(w x ; σ x ). We now use this resultk ink the following examples. These examples are related to the earlier Table about the smoothed response functions. Example Suppose f(y) , sign(y). It is easy to check that f˜(y; σ) = erf( y ). √2σ T Applying the lemma to this implies that [sign(T x)⋆k ()](w) = erf( w x ). σ √2σ x k k Example Suppose f(y) , max(0,y). It is easy to check that f˜(y; σ) = y2 σ 1 y T e− 2σ2 + y 1+erf( ) . Applying the lemma to this implies that [max(0, x)⋆ √2π 2 √2σ T 2 (w x) T σ x 2 2 1 T w x k ()](w)= k k e− 2σ kxk + w x 1+erf( ) . σ √2π 2 √2σ x k k Example Suppose f(y) , sin(y). From trigonometric table in 2.3 we know 2 σ T f˜(y; σ) = e− 2 sin(y). Applying the lemma to this implies that [sin( x) ⋆ 2 2 σ kxk T kσ()](w)= e− 2 sin(w x). In particular, when x , (x, 1), then smoothed σ2(1+x2) version of sin(w1x + w2) w.r.t. w is e− 2 sin(w1x + w2). References [Blinchikoff and Zverev, 1976] Blinchikoff, H. and Zverev, A. (1976). Filtering in the time and frequency domains. R.E. Krieger Pub. Co. [Foley et al., 1990] Foley, J. D., van Dam, A., Feiner, S. K., and Hughes, J. F. (1990). Computer Graphics: Principles and Practice (2Nd Ed.). Addison- Wesley Longman Publishing Co., Inc., Boston, MA, USA. 4 [Hartman et al., 1990] Hartman, E., Keeler, J. D., and Kowalski, J. M. (1990). Layered neural networks with gaussian hidden units as universal approxima- tions. Neural Comput., 2(2):210–215. [Koenderink, 1984] Koenderink, J. J. (1984). The structure of images. Biological Cybernetics, 50(5):363–370. [Lowe, 1999] Lowe, D. G. (1999). Object recognition from local scale-invariant features. In Proceedings of the International Conference on Computer Vision, ICCV ’99. [Maggiar et al., 2015] Maggiar, A., Waechter, A., Dolinskaya, I. S., and Staum, J. (2015). A derivative-free trust-region algorithm for the optimization of functions smoothed via gaussian convolution using adaptive multiple impor- tance sampling. Technical Report IEMS Department, Northwestern Univer- sity. [Mobahi and Fisher, 2015] Mobahi, H. and Fisher, III, J. W. (2015). On the link between gaussian homotopy continuation and convex envelopes. In Lecture Notes in Computer Science (EMMCVPR 2015). Springer. [Mobahi et al., 2012] Mobahi, H., Zitnick, C. L., and Ma, Y. (2012). Seeing through the blur. In 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2012), Rhode Island, USA. IEEE Computer Society. [Widder, 1975] Widder, D. V. (1975). The Heat Equation. Academic Press. 5 6 Appendices A Polynomials p p x , Here we prove that [ ⋆ kσ](x) = (iσ) hp( iσ ), where i √ 1 and hp is the 2 2 − x dp x , p 2 2 p’th Hermite polynomial hp(x) ( 1) e dxp e− . Using Fourier Transform F we proceed as below, − p [ ⋆ kσ](x) (6) 1 p = F − F [ ⋆ kσ](x) (7) 1{ { p }} = F − F x F kσ(x) (8) { { } { }} 1 p (p) 1 1 σ2ω2 = F − √2π( i) δ (ω) e− 2 (9) { − √2π } p 1 (p) 1 1 σ2ω2 = √2π( i) F − δ (ω) e− 2 (10) − { √2π } p (p) 1 1 σ2ω2 iωx = √2π( i) δ (ω) e− 2 e dω (11) − ZR √2π p p d 1 1 σ2ω2+iωx √ 2 = 2π( i) p e− (12) − dω √2π ω=0 p p d 1 1 σ2ω2+iωx √ 2 = 2π( i) p e− (13) − dω √2π ω=0 p d 1 2 2 p 2 σ ω +iωx = ( i) p e− (14) − dω ω=0 (ω− 1 ix)2 p x2 σ2 p d 2σ2 2 − − σ2 = ( i) p e (15) − dω ω=0 ω 1 ix 2 ( − 2 ) 2 p σ p x d − 2 = ( i) e− 2σ2 e σ2 (16) − dωp ω=0 2 1 2 2 σ (ω− ix) p x 1 p σ2 = ( i) e− 2σ2 hp(σ(ω ix))( σ) e− 2 (17) − − σ2 − ω=0 x = (iσ)ph ( ) .