Arxiv:2101.01162V1 [Cs.IT] 4 Jan 2021 to Measure Diﬀerences Between Probability Densities [32]

TRANSPORT INFORMATION BREGMAN DIVERGENCES

WUCHEN LI

Abstract. We study Bregman divergences in probability density space embedded with the L2–Wasserstein metric. Several properties and dualities of transport Bregman divergences are provided. In particular, we derive the transport Kullback–Leibler (KL) divergence by a Bregman divergence of negative Boltzmann–Shannon entropy in L2– Wasserstein space. We also derive analytical formulas and generalizations of transport KL divergence for one-dimensional probability densities and Gaussian families.

1. Introduction

Bregman divergences between probability densities are crucial in statistical inference, optimization, and image/signal processing with vast applications in AI inference problems and optimizations [8, 27, 36]. They measure differences between two densities by general- izing L2 (Euclidean) distances. In general, the Bregman divergence is not symmetric and satisfies several duality properties, which are useful in estimation and optimization algorithms. One typical example is the Kullback–Leibler (KL) divergence, which is a Bregman divergence of (negative) Boltzmann-Shannon entropy in L2 space. Information geometry [1, 4, 5] studies properties of Bregman divergences. It focuses on the Fisher-Rao information metric, a.k.a. the Hessian metric of (negative) Boltzmann– Shannon entropy in L2 space. A known fact is that the Fisher-Rao information metric can be used to construct the KL divergence and its generalizations [4, 12] with desirable duality properties. Recently, optimal transport, a.k.a. Wasserstein distance, introduces the other type of distance functions in probability density space. It uses the pushforward mapping functions arXiv:2101.01162v1 [cs.IT] 4 Jan 2021 to measure differences between probability densities [32]. A particular example is the L2– Wasserstein distance, which forms an analog of L2 distance between mapping functions. It also introduces a metric space for probability densities, namely the L2–Wasserstein space [3, 30]. In this space, the L2–Wasserstein distance shows a particular convexity property towards mapping functions [3, 24]. This convexity property nowadays has vast applications in fluid dynamics [11, 16, 17], inverse problems [35], and AI inference problems [2, 13, 31]. Natural questions arise. What are Bregman divergences in L2–Wasserstein space? In particular, what is the “KL divergence” in L2–Wasserstein space?

Key words and phrases. Transport Bregman divergence; Transport KL divergence; Transport Jensen- Shannon divergence. W. Li is supported by the start up funding in University of South Carolina. 1 2 LI

In this paper, we formulate Bregman divergences in L2–Wasserstein space, namely transport Bregman divergences. We study several properties of transport Bregman divergences. In particular, we derive the transport Bregman divergence of (negative) Boltzmann– Shanon entropy. It can be viewed as the KL divergence in L2–Wasserstein space, whose properties, examples, and symmetric generalizations are provided. d We briefly present the main result. Denote a compact smooth set by Ω ⊂ R , and let P(Ω) be the probability density space supported on Ω. Given a “convex” functional F : P(Ω) → R, define Z δ DT,F (pkq) = F(p) − F(q) − ∇x F(q), ∇xΦp(x) − x q(x)dx, Ω δq(x) δ 2 where p, q ∈ P(Ω), δq(x) is the L first variation w.r.t. q(x), and ∇xΦp is the optimal transport map function that pushforwards q to p, such that

(∇xΦp)#q = p.

Here we call DT,F the transport Bregman divergence. If F is a second moment functional, R 2 2 i.e. F(p) = kxk p(x)dx, then DT,F forms the L –Wasserstein distance. If F is the Ω R negative Boltzmann–Shanon entropy, i.e. F(p) = Ω p(x) log p(x)dx, then the transport Bregman divergence satisﬁes Z 2 DTKL(pkq) = DT,F (pkq) = ∆xΦp(x) − log det(∇xΦp(x)) − d q(x)dx. Ω

We name DTKL the transport KL divergence. We notice that DTKL has a closed form formula in one dimensional sample space. Z 1 −1 −1 ∇xFp (x) ∇xFp (x) DTKL(pkq) := −1 − log −1 − 1 dx, 0 ∇xFq (x) ∇xFq (x) −1 where Fp, Fq are cumulative distribution functions (CDFs) of p, q, respectively, and Fp , −1 Fq are their inverse CDFs. We remark that DTKL is an Itakura–Saito type divergence in term of mapping functions. There are joint works in the literature between optimal transport and information geometry to study Bregman divergences [2, 7, 13, 22, 23, 25, 33, 34]. In particular, [13, 33, 34] apply linear programming formulations of optimal transport. They use divergence functions on sample space as ground costs to construct the ones in probability density space. Compared to the above approaches, we define Bregman divergences by Jacobi operators of mapping functions. And our divergence functionals are built from both gradient and Hessian operators of information entropies in L2–Wasserstein space. Besides, our definition inherits ideas from the Wasserstein subdifferential calculus defined in [3]. Here we focus on formulations and generalizations towards transport Bregman divergences. This paper is organized as follows. In section 2, we review the definition of Bregman divergence in Euclidean space. In section 3 and 4, we construct transport Bregman divergences and study their properties. In section 5, we study the transport KL divergence and its symmetric generalizations. Several analytical formulas of transport KL divergences for one-dimensional probability densities and Gaussian families are provided. TRANSPORT INFORMATION BREGMAN DIVERGENCES 3

2. Bregman divergences in Euclidean space

In this section, we briefly recall the definition of Bregman divergences in Euclidean space. Bregman divergences generalize Euclidean distances as follows. d Definition 1 (Bregman divergence). Denote a closed convex set Ω ⊂ R . Let (·, ·) be a Euclidean inner product and denote the Euclidean norm by k · k. Let ψ :Ω → R be a smooth strictly convex function. The Bregman divergence Dψ :Ω × Ω → R is defined by

Dψ(ykx) = ψ(y) − ψ(x) − (∇ψ(x), y − x), for any x, y ∈ Ω.

We present several examples of Bregman divergences.

1 2 (i) Let Ω = R and ψ(z) = z , z ∈ Ω. Then 2 2 2 Dψ(ykx) = y − x − 2x(y − x) = (y − x) .

Here Dψ forms the Euclidean distance. (ii) Let Ω = R+ and ψ(z) = z log z, z ∈ Ω. Then y D (ykx) = y log − (y − x). ψ x Here Dψ leads to the KL divergence in R+. (iii) Let Ω = R+ and ψ(z) = − log z, z ∈ Ω. Then y y D (ykx) = − log − 1. (1) ψ x x Here Dψ is known as the Itakura–Saito divergence in R+. There are several properties of Bregman divergences.

• Nonnegativity: Dψ(ykx) ≥ 0; d • Hessian metric: Consider a Taylor expansion as follows. Denote ∆x ∈ R , then 1 D (x + ∆xkx) = ∆xT∇2ψ(x)∆x + o(k∆xk2), ψ 2 where ∇2ψ is the Hessian operator of ψ w.r.t. the Euclidean metric. If ψ(z) = 2 1 z log z, then ∇ ψ = z is known as the Fisher-Rao information metric; • Asymmetry: In general, Dψ is not necessary symmetric w.r.t. x and y, i.e. Dψ(ykx) 6= Dψ(xky). For this reason, we call Dψ the “divergence” function instead of a distance function; • Convexity in the ﬁrst variable: Dψ(ykx) is always convex w.r.t. y, not necessary w.r.t. x. The Itakura–Saito divergence (1) is an example. ∗ ∗ ∗ • Duality: Denote the conjugate (dual) function of ψ by ψ (x ) = supx∈Ω (x, x ) − ψ(x). Then ∗ ∗ Dψ∗ (x ky ) = Dψ(ykx). Here x∗ = ∇ψ(x), y∗ = ∇ψ(y) are dual points corresponding to x and y. In practice, Bregman divergences have been extensively studied in probability density space embedded with the L2 metric, which have vast applications in statistics, optimization and AI. In this paper, instead of using the L2 metric, we formulate and study Bregman divergences w.r.t. the L2–Wasserstein metric. 4 LI

3. Bregman divergences in L2–Wasserstein space

In this section, we deﬁne the Bregman divergence in L2–Wasserstein space. Several concrete examples are provided.

3.1. Review of L2–Wasserstein space. We briefly recall some facts in L2–Wasserstein d space [3, 32]. Denote Ω by a d–dimensional compact convex set. E.g., Ω = T , which is a d-dimensional torus. Denote the smooth probability density space by n Z o P(Ω) = p ∈ C∞(Ω): p(x)dx = 1, p(x) ≥ 0 . Ω Given p, q ∈ P(Ω), the L2–Wasserstein distance is defined by Z 2 n 2 o DistT(p, q) = inf kT (x) − xk q(x)dx: T#q(x) = p(x) , T :Ω→Ω Ω where the infimum is among all differemorphisms T that pushforward q to p. Here, # represents a pushforward operator, such that

p(T (x))det(∇xT (x)) = q(x). The optimality condition for the map function T can be formulated below. Deﬁnition 2 (Transport coordinates). Given p, q ∈ P(Ω), suppose that there exists ∞ strictly convex functions Φp, Φq ∈ C (Ω), such that kxk2 T (x) = ∇ Φ (x) and Φ (x) = . (2) x p q 2 In this case, the L2–Wasserstein distance can be formulated by sZ 2 DistT(p, q) = k∇xΦp(x) − ∇xΦq(x)k q(x)dx. Ω

From now on, we always use ∇xΦp, ∇xΦq to represent functions T , x, respectively. And we call Φp, Φq the transport coordinates.

There is also a linear programming reformulation of L2–Wasserstein distance. Denote the optimal joint probability density function for densities p and q by

π = (∇Φq × ∇Φp)#q = (id × ∇Φp)#q, (3) where Z Z π(x, y)dy = q(x), π(x, y)dx = p(y). Ω Ω In this sense, the L2–Wasserstein distance in term of the optimal joint density function satisﬁes Z Z 2 2 DistT(p, q) = ky − xk π(x, y)dxdy. Ω Ω In addition, the L2–Wasserstein distance also introduces a metric in P(Ω). Denote the tangent space at q ∈ P(Ω) by Z ∞ TqP(Ω) = {σ ∈ C (Ω): σ(x)dx = 0}. Ω TRANSPORT INFORMATION BREGMAN DIVERGENCES 5

Deﬁne an inner product gT(q): TqP(Ω) × TqP(Ω) → R, such that Z gT(q)(σ, σ) = (∇xΦ(x), ∇xΦ(x))q(x)dx, Ω where Φ, σ ∈ C∞(Ω) satisfy

σ(x) = −∇x · (q(x)∇xΦ(x)). 2 In literature, (P(Ω), gT) is often called the L –Wasserstein space. In this paper, we intro- duce Bregman divergences in (P(Ω), gT). To do so, we review several useful facts . Proposition 1 (Facts in L2–Wasserstein space [3, 32]). The following facts hold.

2 2 (i) The L gradient of functional DistT(p, q) w.r.t. q(x) satisﬁes 1 δ Dist (p, q)2 = Φ (x) − Φ (x), 2 δq(x) T p q

δ 2 where Φp, Φq are defined in (2). Here δq(x) is the L first variation operator w.r.t density q at x ∈ Ω. (ii) Denote a smooth functional F : P(Ω) → R. The gradient operator of functional F in (P(Ω), gT) satisfies δ grad F(q)(x) = −∇ · (q(x)∇ F(q)) ∈ T P(Ω). T x x δq(x) q

(iii) The Hessian operator of functional F in (P(Ω), gT) satisﬁes Z Z 2 2 δ HessTF(q)(σ, σ) = ∇xy F(q)(∇Φ(x), ∇Φ(y))q(x)q(y)dxdy Ω Ω δq(x)δq(y) Z (4) 2 δ + ∇x F(q)(∇Φ(x), ∇Φ(x))q(x)dx, Ω δq(x)

∞ δ2 2 where Φ, σ ∈ C (Ω) satisfy σ(x) = −∇x ·(q(x)∇xΦ(x)), δq(x)δq(y) is the L second 2 variation operator w.r.t. q(x), q(y) respectively, ∇xy is the second diﬀerential 2 operator in Ω w.r.t. x, y respectively, and ∇x is the Hessian operator in Ω w.r.t. x; see details in [17]; (iv) We say that functional F is λ–geodesic convex in (P(Ω), gT), if for any σ ∈ TqP(Ω) and q ∈ P(Ω), there exists a constant λ, such that

HessTF(q)(σ, σ) ≥ λgT(q)(σ, σ).

In other words, for any p, q ∈ P(Ω) with Φp, Φq deﬁned in (2), then for any t ∈ [0, 1], t(1 − t) F((t∇Φ + (1 − t)∇Φ ) q) ≤ tF(p) + (1 − t)F(q) − λ Dist (p, q)2. p q # 2 T

In literature [24] and [32, Deﬁnition 16.5], the geodesic convexity in (P(Ω), gT) is named the displacement convexity. 6 LI

3.2. Transport Bregman divergence. We are now ready to state the main result of this paper. We deﬁne Bregman divergences in L2–Wasserstein space.

Deﬁnition 3 (Transport Bregman divergence). Let F : P(Ω) → R be a smooth strictly displacement convex functional. Deﬁne DT,F :Ω × Ω → R by Z δ DT,F (pkq) = F(p) − F(q) − ∇x F(q),T (x) − x q(x)dx, (5) Ω δq(x) where T (x) is the optimal transport map function (2) from q to p, such that

T#q = p and T (x) = ∇xΦp(x). In other words, Z δ DT,F (pkq) = F(p) − F(q) − ∇x F(q), ∇xΦp(x) − ∇xΦq(x) q(x)dx, Ω δq(x) where Φp, Φq are deﬁned in (2). We call DT,F the transport Bregman divergence.

The following proposition describes that functional (5) is a generalization of Euclidean Bregman divergence in Deﬁnition 1.

Proposition 2. Functional DT,F satisﬁes the following equality Z 1 δ 2 DT,F (pkq) =F(p) − F(q) − gradTF(q)(x) · DistT(p, q) dx. 2 Ω δq(x)

Proof. The proof follows from the properties of the L2–Wasserstein space. From Proposi- tion 1 (i), (ii), we have Z 1 δ 2 gradTF(q)(x) · DistT(p, q) dx 2 Ω δq(x) Z δ = − ∇ · (q(x)∇ F(q)) · (Φ (x) − Φ (x))dx x x δq(x) p q Ω (6) Z δ = ∇x F(q), ∇xΦp(x) − ∇xΦq(x) q(x)dx Ω δq(x) Z δ = ∇x F(q),T (x) − x q(x)dx, Ω δq(x) where the second equality holds by the integration by parts formula. And we also apply the fact that T (x) = ∇xΦp(x) is the optimal mapping function.

We also formulate transport Bregman divergences by the optimal joint density function. Proposition 3. Denote π by (3). Then Z δ DT,F (pkq) = F(p) − F(q) − ∇x F(q), y − x π(x, y)dxdy. Ω δq(x)

Proof. We represent the optimal mapping function by Z T (x) = ∇xΦp(x) = yπ(x, y)dy. Ω TRANSPORT INFORMATION BREGMAN DIVERGENCES 7

Hence Z δ Z Z δ ∇x F(q),T (x) − x q(x)dx = ∇x F(q), y − x π(x, y)dxdy. Ω δq(x) Ω Ω δq(x) Remark 1 (Connections with Euclidean Bregman divergences). As shown in proposition 2, we replace the gradient operators of both functional and distance in Euclidean space to the corresponding ones in L2–Wasserstein space. Remark 2. Our formulations of Bregman divergences connects with the sub-diﬀerential calculus proposed in [3, Chapter 10]. One can also view the transport Bregman divergence in Proposition 3 as a deﬁnition for transport Bregman divergences. Here the optimal map function T = ∇xΦp is not necessary required to be smooth. In this paper, we focus on formulations of transport Bregman divergences, and leave their analytical properties in future works.

3.3. Formulations. We ﬁrst demonstrate several examples of transport Bregman divergences. Proposition 4 (Mapping formulations). Transport Bregman divergences have the following formulations in term of mapping functions Φp, Φq deﬁned in (2). (i) Consider a linear energy by Z V(p) = V (x)p(x)dx, Ω ∞ d where the linear potential function V ∈ C (Ω) is strictly convex in R . Then Z DT,V (pkq) = DV (∇xΦp(x)k∇xΦq(x))q(x)dx, (7) Ω

where DV :Ω × Ω → R is a Euclidean Bregman divergence of V deﬁned by

DV (z1kz2) = V (z1) − V (z2) − ∇V (z2) · (z1 − z2), for any z1, z2 ∈ Ω. (ii) Consider an interaction energy by 1 Z Z W(p) = W (x, x˜)p(x)p(˜x)dxdx,˜ 2 Ω Ω where the interaction kernel potential function is W (x, x˜) = W (˜x, x) ∈ C∞(Ω×Ω). Assume W (x, x˜) = W˜ (x − x˜), where W˜ (z) is a convex function of z in Ω with W˜ (z) = W˜ (−z). Then Z Z 1 DT,W (pkq) = DW˜ ∇xΦp(x) − ∇Φp(˜x)k∇Φq(x) − ∇Φq(˜x) q(x)q(˜x)dxdx,˜ (8) 2 Ω Ω ˜ where DW˜ :Ω × Ω → R is a Euclidean Bregman divergence of W deﬁned by ˜ ˜ ˜ DW˜ (z1kz2) = W (z1) − W (z2) − ∇W (z2) · (z1 − z2), for any z1, z2 ∈ Ω. 8 LI

(iii) Consider a negative entropy by Z U(p) = U(p(x))dx, Ω where the entropy potential U :Ω → R is second differentiable and convex. Then Z 2 2 DT,U (pkq) = DUˆ ∇xΦp(x)k∇xΦq(x) q(x)dx, Ω d×d d×d where DUˆ : R × R → R is a matrix Bregman divergence function. Denote ˆ d×d function U : R+ × R → R by q det(A) Uˆ(q, A) = U( ) , det(A) q where q ∈ R+ is the given reference density, and ˆ ˆ ˆ DUˆ (AkB) = U(q, A) − U(q, B) − tr ∇BU(q, B) · (A − B) , d×d for any A, B ∈ R and ∇B is the Frechet derivative of a symmetric matrix B. In details, Z n q(x) 2 DT,U (pkq) = U( 2 )det(∇xΦp(x)) − U(q(x)) Ω det(∇xΦp(x)) (9) 2 0 o + tr(∇xΦp(x) − I) U (q(x))q(x) − U(q(x)) dx, d×d where I ∈ R is an identity matrix. Proposition 5 (Joint density formulations). Transport Bregman divergences have the following formulations by the joint density function π defined in (3). (i) Formula (7) leads to Z Z DT,V (pkq) = DV (ykx)π(x, y)dxdy. Ω Ω (ii) Formula (8) leads to 1 Z Z Z Z DT,W (pkq) = DW˜ (y − y˜kx − x˜)π(x, y)π(˜x, y˜)dxdydxd˜ y.˜ 2 Ω Ω Ω Ω (iii) Formula (9) leads to Z Z Z Z DT,U (pkq) = U( π(x, y)dx)dy − U( π(x, y)dy)dx Ω Ω Ω Ω Z Z Z + (y · ∇xπ(x, y) − d)U¯( π(x, z)dz)dxdy, Ω Ω Ω where we denote U¯(z) := zU 0(z) − U(z). Remark 3 (Comparisons with Wasserstein-Bregman divergence). We notice that formulation (i) is similar but different with divergence functionals defined in [13, 33]. In our formulation, joint density π is the optimal one from the L2–Wasserstein space; while in [13] or [33], joint density π solves the related linear programming problem w.r.t. divergence type ground costs. TRANSPORT INFORMATION BREGMAN DIVERGENCES 9

Remark 4. We remark that formulation (ii) leads to a quadratic programming problem with interaction potential divergence functions DW˜ .

Remark 5. In information geometry [1] and information theory [9], the entropy functional U(p) is often named information entropy. Because of this reason, we call divergence functional (9) the transport information Bregman divergence.

Besides, we derive several closed form solutions of transport Bregman divergences in one dimensional space.

1 Proposition 6 (One dimensional closed form solutions). Let Ω = [0, 1] or T . Denote −1 −1 Fp, Fq as the cumulative distribution of densities p, q, respectively, and let Fp , Fq be their inverse functions. We have the following closed formulas of transport Bregman divergences.

(i) The transport Bregman divergence of linear energy V (7) satisﬁes

Z 1 −1 −1 DT,V (pkq) = DV (Fp (x)kFq (x))dx. 0

(ii) The transport Bregman divergence of interaction energy W (8) satisﬁes

Z 1 Z 1 1 −1 −1 −1 −1 DT,W (pkq) = DW˜ (Fp (x) − Fp (˜x)kFq (x) − Fq (˜x))dxdx.˜ 2 0 0

(iii) The transport Bregman divergence of negative entropy U (9) satisﬁes

Z 1 −1 −1 DT,U (pkq) = DU˜ (∇xFp (x)k∇xFq (x))dx, (10) 0

where we denote 1 U˜(z) = zU( ), z ˜ and DU˜ is the Euclidean Bregman divergence of U deﬁned by

˜ ˜ ˜ DU˜ (z1kz2) = U(z1) − U(z2) − ∇zU(z2) · (z1 − z2).

3.4. Examples. In this subsection, we present several examples of transport Bregman divergences.

Example 1 (Linear energy). We remark that the transport Bregman divergence of linear energy (7) is the expectation of a “linear Bregman divergence potential”. Several examples are given below. 10 LI

R 2 (i) If V(ρ) = Ω kxk p(x)dx, then Z 2 2 DT,V (pkq) = kT (x)k − kxk − 2(T (x) − x, x) q(x)dx Ω Z = kT (x) − xk2q(x)dx Ω Z 2 = k∇xΦp(x) − ∇xΦq(x)k q(x)dx Ω Z Z = ky − xk2π(x, y)dxdy Ω Ω 2 =DistT(p, q) . R 2 In this case, we observe that when V(p) = Ω kxk p(x)dx is the second moment functional, the transport Bregman divergence forms the L2–Wasserstein distance. In other words, we can view L2–Wasserstein distance as the transport Bregman divergence of a second moment functional. If d = 1, then Z 1 −1 −1 2 DT,V (pkq) = kFp (x) − Fq (x)k dx. 0 This is a known closed form solution for L2–Wasserstein distance [3]. R (ii) If Ω = [0, 1] and V(p) = Ω x log xp(x)dx, then Z ∇xΦp(x) DT,V (pkq) = ∇xΦp(x) log − ∇xΦp(x) − ∇xΦq(x) q(x)dx Ω ∇xΦq(x) Z Z y = y log − (y − x) π(x, y)dxdy Ω Ω x Z 1 −1 −1 Fp (x) −1 −1 = Fp (x) log −1 − Fp (x) − Fq (x) dx. 0 Fq (x) R (iii) If Ω = [0, 1] and V(p) = − Ω log xp(x)dx, then Z ∇xΦp(x) ∇xΦp(x) DT,V (pkq) = − log − 1 q(x)dx Ω ∇xΦq(x) ∇xΦq(x) Z Z y y = − log − 1 π(x, y)dxdy Ω Ω x x Z 1 −1 −1 Fp (x) Fp (x) = −1 − log −1 − 1 dx. 0 Fq (x) Fq (x) Example 2 (Interaction energy). We remark that the transport Bregman divergence of interaction energy (8) is the expectation of an “interaction Bregman divergence potential”. Two examples are given below.

R R 2 (i) If W(p) = Ω Ω kx − x˜k p(x)p(˜x)dxdx˜, then Z Z 2 DT,W (pkq) = k(∇Φp(x) − ∇Φp(˜x)) − (∇Φq(x) − ∇Φq(˜x))k q(x)q(˜x)dxdx˜ Ω Ω Z Z = k(y − y˜) − (x − x˜)k2π(x, y)π(˜x, y˜)dxdydxd˜ y.˜ Ω Ω TRANSPORT INFORMATION BREGMAN DIVERGENCES 11

If d = 1, then Z 1 Z 1 −1 −1 −1 −1 2 DT,W (pkq) = k(Fp (x) − Fp (˜x)) − (Fq (x) − Fq (˜x))k dxdx.˜ 0 0 R R 1 (ii) If W(p) = Ω Ω log kx−x˜k p(x)p(˜x)dxdx˜, then Z Z k∇xΦp(x) − ∇Φp(˜x)k k∇xΦp(x) − ∇Φp(˜x)k DT,W (pkq) = − log − 1 q(x)q(˜x)dxdx˜ Ω Ω k∇Φq(x) − ∇Φq(˜x)k k∇Φq(x) − ∇Φq(˜x)k Z Z Z Z ky − y˜k ky − y˜k = − log − 1 π(x, y)π(˜x, y˜)dxdydxd˜ y.˜ Ω Ω Ω Ω kx − x˜k kx − x˜k If d = 1, then Z 1 Z 1 −1 −1 −1 −1 kFp (x) − Fp (˜x)k kFp (x) − Fp (˜x)k DT,W (pkq) = −1 −1 − log −1 −1 − 1 dxdx.˜ 0 0 kFq (x) − Fq (˜x)k kFq (x) − Fq (˜x)k Example 3 (Negative Entropy). We remark that the transport Bregman divergence of entropy (9) is the expectation of a matrix Bregman divergence for mapping Jacobi operators. Two examples are given below. R (i) Let U(p) = Ω p(x) log p(x)dx. Then the transport Bregman divergence forms Z 2 DT,U (pkq) = ∆xΦp(x) − log det(∇xΦp(x)) − d q(x)dx Ω Z Z Z = p(x) log p(x) − q(x) log q(x) dx + (y · ∇xπ(x, y) − d)q(x)dxdy. Ω Ω Ω If d = 1, then Z 1 −1 −1 ∇xFp (x) ∇xFp (x) DT,U (pkq) = −1 − log −1 − 1 dx. (11) 0 ∇xFq (x) ∇xFq (x) R p(x)2 (ii) Let U(p) = Ω 2 dx. Then the transport Bregman divergence forms Z 1 1 2 DT,U (pkq) = ∆xΦp(x) + 2 − 1 − d q(x) dx 2 Ω det(∇xΦp(x)) Z Z Z 1 2 2 1 2 = p(x) − q(x) dx + (y · ∇xπ(x, y) − d)q(x) dxdy. 2 Ω 2 Ω Ω If d = 1, then

Z 1 2 1 1 1 −1 DT,U (pkq) = −1 − −1 · ∇xFp (x)dx. 2 0 ∇xFp (x) ∇xFq (x)

4. Properties

In this section, we study several properties of transport Bregman divergences.

Proposition 7 (Properties). Transport Bregman divergence DT,F has the following properties. 12 LI

(i) Non-negativity: Suppose F is displacement convex, then

DT,F (pkq) ≥ 0. Suppose F is strictly displacement convex, then

DT,F (pkq) = 0 if and only if DistT(p, q) = 0. (ii) Transport Hessian metric: Consider a Taylor expansion as follows. Denote σ = −∇ · (q∇Φ) ∈ TqP(Ω) and ∈ R, then 2 D ((id + ∇Φ) qkq) = Hess F(q)(σ, σ) + o(2), T,F # 2 T where id: Ω → Ω is the identical map, id(x) = x, and HessTF(q) is deﬁned in (4), which is the Hessian operator of functional F at q ∈ P(Ω) w.r.t. L2–Wasserstein metric. (iii) Linearity: Consider two functionals F, G : P → R and a > 0. Then

DT,F+aG(pkq) = DT,F (pkq) + aDT,G(pkq).

(iv) Asymmetry: In general, DT,F (pkq) 6= DT,F (qkp). One typical example is formula (11), which is an Itakura–Saito divergence in transport coordinates.

Proof. The proof follows from the definition of transport Bregman divergence. (i) Since F is λ–geodesic convex in (P, gT), then Z δ 2 λ 2 F(p) ≥ F(q) + gradTF(q)(x) · DistT(p, q) dx + DistT(p, q) , Ω δq(x) 2 i.e. λ D (pkq) ≥ Dist (p, q)2. T,F 2 T Hence if λ ≥ 0, we have DT,F (pkq) ≥ 0. In particular, if λ > 0, then DT,F (pkq) = 0 iff 2 DistT(p, q) = 0. (ii) Here the Hessian metric property follows from the definition. Consider a Taylor expansion in (P, gT) by Z 2 2 F((id + ∇Φ)#q) = F(q) + gradTF(q)(x) · Φ(x)dx + HessTF(q)(σ, σ) + o( ). Ω 2 From the definition of transport Bregman divergences, we have 2 D ((id + ∇Φ) qkq) = Hess F(q)(σ, σ) + o(2). T,F # 2 T (iii) Here, the linearity follows from the linearity of gradient operator in L2–Wasserstein metric. Notice that for any a > 0, we have δ δ grad F(q) + agrad G(q) = − ∇ · q(x)∇ F(q) − a∇ · q(x)∇ G(q) T T x δq(x) x δq(x) δ = − ∇ · q(x)∇ (F(q) + aG(q)) x δq(x)

=gradT(F + aG)(q). Following Proposition 2, we ﬁnish the proof. TRANSPORT INFORMATION BREGMAN DIVERGENCES 13

We next formulate a duality theorem for transport Bregman divergences. Denote ∞ C (Ω)/R be a smooth functional space, whose element is uniquely determined up to ∞ a constant shrift. Consider a functional F : P(Ω) → R. Denote F˜ : C (Ω)/R → R, such that F˜(Φ) = F(∇Φ#q). (12) Theorem 1 (Transport duality). Assume that functional F˜ is convex w.r.t. Φ in L2 ∗ ∞ space. Denote F : C (Ω)/R → R, such that Z F˜∗(Φ∗) = sup Φ(x)Φ∗(x)dx − F˜(Φ). (13) Φ∈C∞(Ω)/R Ω Then the following relations hold. (i) δ δ ∇ F˜∗( F˜(Φ)) = ∇ Φ(x), x δΦ∗(x) δΦ x ∞ for any Φ ∈ C (Ω)/R. (ii)

DT,F (pkq) = DF˜(ΦpkΦq), 2 where DF˜ is a Bregman divergence in L space satisfying Z ˜ ˜ δ ˜ DF˜(pkq) = F(Φp) − F(Φq) − (Φp(x) − Φq(x)) · F(Φq)dx. Ω δΦq(x) (iii) ∗ ∗ DF˜(ΦpkΦq) = DF˜∗ (ΦqkΦp) = DT,F (pkq), ∗ ∗ ∞ where Φp, Φq ∈ C (Ω) satisfy

∗ δ ˜ ∗ δ ˜ Φp(x) = F(Φp), Φq(x) = F(Φq), δΦp(x) δΦq(x) 2 and DF˜∗ is a Bregman divergence in L space, such that Z ∗ ∗ ˜∗ ˜∗ ∗ ∗ δ ˜∗ ∗ DF˜∗ (ΦqkΦp) = F (Φq) − F (Φp) − Φq(x) − Φp(x) · ∗ F (Φp)dx. Ω δΦp(x)

Proof. We notice that using transport coordinates (2), the L2–Wasserstein space is ﬂat. And the proof here follows from the duality of transport coordinates in L2 space. (i) Consider Z Φ = arg sup Φ(x)Φ∗(x)dx − F˜(Φ). Φ∈C∞(Ω)/R Ω Then δ Φ∗ = F˜(Φ). δΦ Also, consider Z F˜(Φ) = F˜∗∗(Φ) = sup Φ(x)Φ∗(x)dx − F˜∗(Φ∗). Φ∈C∞(Ω)/R Ω 14 LI

Hence δ δ δ Φ = F˜∗(Φ∗) = F˜∗( F˜(Φ)), δΦ∗ δΦ∗ δΦ where the equality in above formula holds up to a constant shrift. ˜ (ii) We next prove DT,F (pkq) = DF˜∗ (ΦpkΦq). From the deﬁnition of functional F, it suﬃces to prove that δ δ F˜(Φq) = −∇x · (q(x)∇x F(q)). (14) δΦq(x) δq(x) Here the proof follows [3, Lemma 10.4.1]. For the completeness of paper, we derive (14) ∞ here. Consider a small perturbation h ∈ C (Ω) with ∈ R, then

F˜(Φq + h) = F((∇Φq + ∇h)#q) = F((id + ∇h)#q), where id(x) = x is an identity map. Standard calculations show that d Z δ F˜(Φq + h)|=0 = (∇xh(x), ∇x F(q))q(x)dx d Ω δq(x) Z δ = − h(x)∇x · (q(x)∇x F(q))dx, Ω δq(x) which finishes the proof. (iii) Given p, q ∈ Ω, denote Φ∗ = δ F˜(Φ ), Φ∗ = δ F˜(Φ ). We only need to prove the p δΦp p q δΦq q 2 duality of Bregman divergence DF˜∗ (ΦpkΦq) in L space. On the one hand, Z ˜ ˜ δ ˜ DF˜(ΦpkΦq) =F(Φp) − F(Φq) − (Φp(x) − Φq(x)) · F(Φq)dx Ω δΦq(x) Z δ Z δ =F˜(Φp) + Φq(x) · F˜(Φq)dx − F˜(Φq) − Φp(x) · F˜(Φq)dx Ω δΦq(x) Ω δΦq Z ˜ ˜∗ ∗ ∗ =F(Φp) + F (Φq) − Φp(x)Φq(x)dx, Ω where we apply the definition of conjugate functional F˜∗ in L2 space. On the other hand, Z ∗ ∗ ˜∗ ∗ ˜∗ ∗ ∗ δ ˜∗ ∗ DF˜∗ (ΦqkΦp) =F (Φq) − F (Φp) − (Φq(x) − Φp(x)) · ∗ F (Φp)dx Ω δΦp(x) Z Z ˜∗ ∗ ∗ δ ˜∗ ∗ ˜ ∗ ∗ δ ˜∗ ∗ =F (Φq) + Φp(x) · ∗ F (Φp)dx − F(Φp) − Φq(x) · ∗ F (Φp)dx Ω δΦp(x) Ω δΦp Z ˜∗ ˜ ∗ =F (Φq) + F(Φp) − Φp(x)Φq(x)dx, Ω where we apply the definition of conjugate of conjugate functional F˜ = F˜∗∗ in L2 space. This finishes the proof. Remark 6. We emphasize the fact that the Legendre duality in Theorem 1 represents the one in probability density space P(Ω). It is different from the Legendre duality of the ground cost defined in sample space Ω. TRANSPORT INFORMATION BREGMAN DIVERGENCES 15

5. Transport KL divergence

In this section, we study the transport Bregman divergence of negative Boltzmann– Shannon entropy, namely transport KL divergence.

5.1. Review of KL divergence. We first review some facts about KL divergence, which is defined by Z p(x) DKL(pkq) = p(x) log dx. Ω q(x) It is a Bregman divergence in L2 space of negative H, where H represents the Boltzmann– Shannon entropy defined by Z H(p) = − p(x) log p(x)dx. Ω There are several properties of KL divergence, which are useful in Bayesian sampling and AI inference problems. Firstly, define the cross entropy by Z Hq(p) = − p(x) log q(x)dx. Ω Hence KL divergence can be formulated by Z Z DKL(pkq) = − p(x) log q(x)dx + p(x) log p(x)dx Ω Ω =Hq(p) − H(p). Secondly, KL divergence has many desired properties, such as nonnegativity, separability etc. Lastly, KL divergence can be used to generate other divergences functionals. Notice that KL divergence is non-symmetric w.r.t. p, q. In practice, one can define a symmetrized divergence. One typical example is the Jenson-Shannon divergence defined by 1 1 D (pkq) = D (pkr) + D (qkr), JS 2 KL 2 KL p+q 2 where r = 2 is the geodesic midpoint (Barycenter) in L space.

5.2. Transport KL divergence. We are now ready to derive an analog of KL divergence in L2–Wasserstein space. We consider a transport Bregman divergence of (negative) Boltzmann-Shannon entropy as in Example 3 (i).

Definition 4 (Transport KL divergence). Define DTKL : P(Ω) × P(Ω) → R by Z 2 DTKL(pkq) = ∆xΦp(x) − log det(∇xΦp(x)) − d q(x)dx, Ω where ∇xΦp is the differemorphism map from q to p, such that

(∇xΦp)#q = p.

Denote π = (id × ∇Φp)#q, then Z Z DTKL(pkq) = p(x) log p(x) − q(x) log q(x) dx + y · ∇xπ(x, y) − d q(x)dx. Ω Ω 16 LI

If d = 1, then Z 1 −1 −1 ∇xFp (x) ∇xFp (x) DTKL(pkq) = −1 − log −1 − 1 dx. 0 ∇xFq (x) ∇xFq (x)

We call DTKL the transport KL divergence.

We study several properties of transport KL divergence. We ﬁrst derive an analog of cross entropy in L2–Wasserstein space below.

Deﬁnition 5 (Transport cross entropy). Deﬁne HT,q : P(Ω) → R by Z Z HT,q(p) = ∆xΦp(x)q(x)dx − q(x) log q(x)dx − d, Ω Ω where ∇xΦp#q = p. We call HT the transport cross entropy. If d = 1, then Z 1 −1 Z ∇xFp (x) HT,q(p) = −1 dx − q(x) log q(x)dx − 1. 0 ∇xFq (x) Ω

We next demonstrate an equality for transport KL divergence. Proposition 8. DTKL(pkq) = HT,q(p) − H(p).

Proof. The proof follows from Deﬁnition 4. As in Example 3 (i), from ∇xΦp#q = p, i.e., 2 p(∇xΦp(x))det(∇xΦp(x)) = q(x), (15) we have Z H(q) = q(x) log q(x)dx Ω Z 2 2 = p(∇xΦp(x))det(∇xΦp(x)) log p(∇xΦp(x))det(∇xΦp(x)) dx Ω Z 2 2 = p(∇xΦp(x))det(∇xΦp(x)) log p(∇xΦp(x)) + log det(∇xΦp(x)) dx Ω Z Z 2 = q(x) log det(∇xΦp(x))dx + p(y) log p(y)dy Ω Ω Z 2 = q(x) log det(∇xΦp(x))dx − H(p), Ω where we use the fact that y = ∇xΦp(x) in the last equality. Hence Z 2 DTKL(pkq) = ∆xΦp(x) − log det(∇xΦp(x)) − d q(x)dx Ω Z Z Z = p(x) log p(x)dx − q(x) log q(x)dx + ∆xΦp(x)q(x)dx − d Ω Ω Ω = − H(p) + HT,q(p), which ﬁnishes the proof.

We last derive several properties of transport KL divergence. TRANSPORT INFORMATION BREGMAN DIVERGENCES 17

Theorem 2. The transport KL divergence has the following properties. (i) Nonnegativity: For any p, q ∈ P(Ω), then

DTKL(pkq) ≥ 0. And DTKL(pkq) = 0 iﬀ p(x + c) = q(x), for any x ∈ Ω, d where c ∈ R is a constant vector. (ii) Separability: The transport KL divergence is additive for independent distributions. Suppose (p1, p2), (q1, q2) ∈ P(Ω) × P(Ω) are independent distributions with joint distributions

p(x, y) = p1(x)p2(y), q(x, y) = q1(x)q2(y). Then DTKL(pkq) = DTKL(p1kq1) + DTKL(p2kq2). (iii) Transport Hessian information metric: Denote σ = −∇ · (q∇Φ) ∈ TqP(Ω) and ∈ R. Consider a Taylor expansion by 2 D ((id + ∇Φ) qkq) = Hess H(q)(σ, σ) + o(2). TKL # 2 T Here HessTH is the Hessian operator of (negative) Boltzmann–Shanon entropy H(q) on L2–Wasserstein space [32], known as the transport Hessian information metric [19]. In details, Z 2 2 HessTH(q)(σ, σ) = tr ∇xΦ(x), ∇xΦ(x) q(x)dx, Ω where Φ, σ ∈ C∞(Ω) satisfy the elliptical equation σ(x) = −∇x · q(x)∇xΦ(x) .

(iv) Transport convexity: Denote p1 = (∇xΦp1 )#q and p2 = (∇xΦp2 )#q with pλ = λ∇xΦp1 + (1 − λ)∇xΦp2 q. # Then for any λ ∈ [0, 1], we have

DTKL(pλkq) ≤ λDTKL(p1kq) + (1 − λ)DTKL(p2kq).

Proof. Here (iii) follows from Proposition 7. We only need to prove (i), (ii) and (iv). (i) Denote the eigendecomposition of a symmetric matrix function by 2 T ∇ Φp(x) = V ΛV, where Λ = diag(λi(x))1≤i≤d is a positive eigenvalue matrix and V is an orthogonal eigen- vector matrix. Then d X Z DTKL(pkq) = (λi(x) − log λi(x) − 1)q(x)dx ≥ 0. i=1 Ω

In the above proof, we use the fact that λi − log λi − 1 ≥ 0, where the equality holds when λi = 1. Hence DTKL(pkq) = 0 implies the fact that λi(x) = 1 for i = 1, ··· , d, on the 18 LI

2 support of q. Thus ∇xΦp(x) = I. In other words, ∇xΦp(x) = x + c, where c is a constant d 2 vector in R . From ∇xΦp#q = p, i.e. p(∇xΦp)det(∇ Φp) = q, we prove the result.

(ii) Consider pushforward operators ∇xΦp1 , ∇xΦp2 from q1, q2 to p1, p2, respectively. In other words,

∇xΦp1 (x)#q1(x) = p1(x), ∇yΦp2 (y)#q2(y) = p2(y). Notice

(∇xΦp1 (x), ∇yΦp2 (y))#(q1(x)q2(y)) = p1(x)p2(y).

Denote Φp(x, y) = Φp1 (x) + Φp2 (y). Then ∇Φ(x, y) = (∇xΦp1 (x), ∇yΦp2 (y)). Thus

∇Φ(x, y)#q(x, y) = p(x, y). We are now ready to check the separability property. Notice dim(Ω × Ω) = 2d, then

∆Φp(x, y) = ∆xΦp1 (x) + ∆yΦp2 (y), and 2 2 ∇xxΦp1 (x) 0 det(∇ Φp(x, y)) =det 2 0 ∇yyΦp2 (y) 2 2 =det(∇xxΦp1 (x)) · det(∇yyΦp2 (y)). Hence Z Z 2 DTKL(pkq) = ∆Φp(x, y) − log det(∇ Φp(x, y)) − 2d q(x, y)dxdy Ω Ω Z Z = ∆xΦp1 (x) + ∆yΦp2 (y) Ω Ω 2 2 − log det(∇xΦp1 (x)) − log det(∇yΦp2 (y)) − 2d q1(x)q2(y)dxdy Z Z 2 = ∆xΦp1 (x) − log det(∇xΦp1 (x)) − d Ω Ω 2 + ∆yΦp2 (y) − log det(∇yΦp2 (y)) − d) q1(x)q2(y)dxdy

=DTKL(p1kq1) + DTKL(p2kq2).

d×d (iv) Denote a matrix function f : R → R, such that f(A) = tr(A) − log det(A) − d. Then f is convex w.r.t. the matrix variable A. In this notation, we have Z 2 DTKL(pkq) = f(∇xΦp(x))q(x)dx. Ω Thus Z 2 2 DTKL(pλkq) = f λ∇xΦp1 (x) + (1 − λ)∇xΦp2 (x) q(x)dx Ω Z 2 2 ≤ λf(∇xΦp1 (x)) + (1 − λ)f(∇xΦp2 (x)) q(x)dx Ω =λDTKL(p1kq) + (1 − λ)DTKL(p2kq). TRANSPORT INFORMATION BREGMAN DIVERGENCES 19

Remark 7. We recall the fact that the convexity for classical KL divergence is on both terms of p, q w.r.t. the L2 metric. However, the classical KL divergence may not be convex in term of pushforward mapping function ∇xΦp. A known fact is that the KL divergence in term of mapping depends on the convexity of negative log target distribution on sample space. This is different from the ones in transport KL divergence. Here the convexity w.r.t pushforward map ∇xΦp holds based on the definition of transport KL divergence. See detailed comparisons in appendix subsection 6.3. Remark 8. We notice that the transport convexity in (iv) can be different from the displacement convexity. We only show the transport convexity of transport KL divergence for a fixed reference density q. This fact is similar to the generalized convexity defined in [3, Definition 9.2.4]. Remark 9. We remark that the Hessian operator of (negative) Boltzmann–Shannon entropy in L2–Wasserstein space connects Fisher-Rao metric, Gamma calculus and Ricci curvature on a sample space; see [17, 19, 32]. This paper proposes to construct Bregman divergences by using this transport information Hessian metric.

5.3. Examples in Gaussian distributions. Suppose that pX , pY are two Gaussian d distributions with zero means in R , such that 1 − 1 xTΣ−1x 1 − 1 xTΣ−1x pX (x) = e 2 X , pY (x) = e 2 Y , (16) p d p d (2π) det(ΣX ) (2π) det(ΣY ) d×d where ΣX ,ΣY ∈ R are symmetric positive deﬁnite matrices. Proposition 9. The transport KL divergence in Gaussian family (16) forms

1 1 1 − 1 1 1 det(ΣY ) 2 2 2 2 2 DTKL(pX kpY ) = log + tr ΣX ΣX ΣY ΣX ΣX − d. (17) 2 det(ΣX )

When ΣX , ΣY commute, i.e. ΣX ΣY = ΣY ΣX , (17) can be simpliﬁed by

1 1 1 det(ΣY ) 2 − 2 DTKL(pX kpY ) = log + tr ΣX ΣY − d. 2 det(ΣX )

Proof. The proof follows from the fact [29] that the transport map ∇ΦpX , with the property ∇ΦpX #pY = pX , satisﬁes

1 1 1 − 1 1 2 2 2 2 2 ∇xΦpX (x) = ΣX ΣX ΣY ΣX ΣX x.

1 1 1 − 1 1 2 2 2 2 2 Then ∆xΦpX (x) = tr ΣX ΣX ΣY ΣX ΣX . Combining these facts into Deﬁnition 4, we derive formula (17). Example 4. We remark that the transport KL divergence (17) is diﬀerent from the classical KL divergence, where Z pX (x) DKL(pX kpY ) = pX (x) log dx Ω pY (x) 1 det(ΣY ) 1 −1 d = log + tr ΣX ΣY − . 2 det(ΣX ) 2 2 20 LI

We compare both KL divergence and transport KL divergence in Figure 1.

50 8

40 6 30 4 20 2 KL divergence 10

0 1 Transport KL divergence 0 1 1 0.8 1 0.8 0.8 0.8 0.6 0.6 0.6 0.6 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.2 X X Y 0 0 Y 0 0

Figure 1. A comparison between KL divergence and transport KL divergence for one dimensional Gaussian distributions. Left represents the KL divergence. Right represents the transport KL divergence.

5.4. Transport Jenson–Shannon divergence. We also deﬁne a symmetrized KL divergence in L2–Wasserstein space. We call it the transport Jenson–Shannon divergence.

Deﬁnition 6 (Transport Jensen–Shannon divergence). Deﬁne DTJS : P(Ω) × P(Ω) → R by 1 1 D (pkq) = D (pkr) + D (qkr), TJS 2 TKL 2 TKL where r ∈ P(Ω) is the geodesic midpoint (Barycenter) between p and q in L2–Wasserstein space, i.e. 1 r = ∇xΦp + ∇xΦq q. 2 #

We present several closed form solutions for transport Jenson–Shanon divergence. Proposition 10. The transport Jenson–Shanon divergence in one dimensional sample space satisﬁes 1 −1 −1 1 Z ∇xF (x) · ∇xF (x) D (pkq) = − log p q dx. TJS 2 1 −1 −1 2 0 4 (∇xFp (x) + ∇xFq (x)) Proposition 11. The transport Jenson–Shanon divergence in Gaussian family (16) sat- isﬁes

DTJS(pX kpY ) 1 1 2 2 1 1 1 − 1 1 1 1 1 − 1 1 1 det(ΣX )det(ΣY ) 1 2 2 2 2 2 2 2 2 2 2 = − log + tr ΣX ΣX ΣZ ΣX ΣX + ΣY ΣY ΣZ ΣY ΣY − d, 2 det(ΣZ ) 2 (18) TRANSPORT INFORMATION BREGMAN DIVERGENCES 21

0.6

0.4

0.2

0 1 1 0.8

Transport Jenson-Shanon divergence 0.8 0.6 0.6 0.4 0.4 0.2 0.2 X Y 0 0

Figure 2. Illustration of transport Jensen–Shannon divergences for one dimensional Gaussian distributions. where 1 1 1 1 − 1 1 Σ = ( + )Σ ( + ), with = Σ 2 Σ 2 Σ Σ 2 2 Σ 2 . Z 4 I T Y I T T X X Y X X In particular, if ΣX , ΣY commute, then (18) can be simpliﬁed by

1 1 1 det(Σ 2 )det(Σ 2 ) D (p kp ) = − log X Y . TJS X Y 1 1 2 1 2 2 2 det 4 (ΣX + ΣY )

Proof of Proposition 10 and 11. If d = 1, from [3, Theorem 6.0.2], the geodesic pt(x) ∈ P(Ω), t ∈ [0, 1], in L2–Wasserstein space connecting q to p satisﬁes −1 −1 −1 ∇xFpt (x) = t∇xFp (x) + (1 − t)∇xFq (x).

The geodesic midpoint, i.e. r = p 1 , satisﬁes 2

−1 −1 1 −1 −1 ∇xFr (x) = ∇xFp 1 (x) = ∇xFp (x) + ∇xFq (x) . 2 2 From example 3 (i), we prove the result. Similarly, in Gaussian family, the results follow 2 the proof in Proposition 9. Here the geodesic midpoint p 1 = N (0, ΣZ ) in L –Wasserstein 2 space satisﬁes 1 1 1 Σ = ( ( + ))TΣ ( ( + )) = ( + )Σ ( + ). Z 2 I T Y 2 I T 4 I T Y I T

In particular, if ΣX ,ΣY commute, then 1 1 1 1 1 Σ 2 = Σ 2 + Σ 2 . Z 2 X 2 Y From Proposition 9 and formula (18), we derive the result. 22 LI

Remark 10. A known fact is that the classical Jensen–Shannon divergence between Gauss- ian distributions does not have a closed-form solution [28]. Interestingly, the transport Jensen–Shannon entropy has a closed form solution. And the geodesic midpoint in L2– Wasserstein space provides the other way to construct symmetric divergence functionals.

6. Discussion

In this paper, we formulate Bregman divergences in L2–Wasserstein space, namely transport Bregman divergences. They are generalizations of the L2–Wasserstein distance. We also derive the transport KL divergence by a transport Bregman divergence of negative Boltzmann–Shannon entropy. We remark that the transport KL divergence is an Itakura– Saito type divergence in one-dimensional sample space. We also propose a symmetrized generalization of transport KL divergence. These transport divergences have shown convexity properties in terms of pushforward mapping functions. We expect that transport Bregman divergences will be useful in AI inference and optimization problems.

References [1] S. Amari. Information Geometry and Its Applications, 2016. [2] S. Amari, R. Karakida, and M. Oizumi. Information geometry connecting Wasserstein distance and Kullback–Leibler divergence via the entropy-relaxed transportation problem. Information Geometry, 1, 13–37, 2018. [3] L. Ambrosio, N. Gigli and G. Savare. Gradient Flows in Metric Spaces and in the Space of Probability Measures, 2008. [4] N. Ay, and S. Amari. A Novel Approach to Canonical Divergences within Information Geometry. Entropy, 17, 8111–8129, 2015. [5] N. Ay, J. Jost, H. V. Lê,and L. Schwachhöfer. Information geometry, 2017. [6] D. Bakry and M. Emery.´ Diffusions hypercontractives. Séminaire de probabilitésde Strasbourg, 19:177– 206, 1985. [7] M. Bauer and K. Modin. Semi-invariant Riemannian metrics in hydrodynamics. Calculus of Variations and Partial Differential Equations, volume 59, 2020. [8] A. Cichocki, and S. Amari. Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities. Entropy, 12, 1532-1568, 2010. [9] T. M. Cover and J. A. Thomas. Elements of Information Theory. Wiley Series in Telecommunications. Wiley, New York, 1991. [10] I. Csiszar. Information-type measures of difference of probability distributions and indirect observa- tions. Studia Sci. Math, 2, 299–318, 1967. [11] L. C. Evans, O. Savin, and W. Gangbo. Diffeomorphisms and Nonlinear Heat Flows. SIAM J. Math. Anal., 37(3), 737751, 2005. [12] D. Felice and N. Ay. Towards a canonical divergence within information geometry. arXiv:1806.11363, 2018. [13] X. Guo, J. Hong, and N. Yang. Ambiguity set and learning via Bregman and Wasserstein. arXiv:1705.08056, 2017. [14] R. Jordan, D. Kinderlehrer, and F. Otto. The variational formulation of the Fokker-Planck equation. SIAM J. Math. Anal., 29(1):1–17, 1998. [15] B. Khesin, G. Misiolek, and K. Modin. Geometric Hydrodynamics of Compressible Fluids. arXiv:2001.01143, 2020. [16] J. D. Lafferty. The density manifold and configuration space quantization. Transactions of the Amer- ican Mathematical Society, 305(2):699–741, 1988. [17] W. Li. Transport information geometry: Riemannian calculus in probability simplex. arXiv:1803.06360 [math], 2018. TRANSPORT INFORMATION BREGMAN DIVERGENCES 23

[18] W. Li. Diffusion Hypercontractivity via Generalized Density Manifold. arXiv:1907.12546, 2019. [19] W. Li. Hessian metric via transport information geometry. arXiv:2003.10526, 2020. [20] W. Li and G. Montufar. Natural gradient via optimal transport. Information Geometry, 1, 181–214, 2018. [21] W. Li and G. Montufar. Ricci curvature for parametric statistics via optimal transport. Information Geometry, 3, 89–117, 2020. [22] L. Malago and G. Pistone. Natural Gradient Flow in the Mixture Geometry of a Discrete Exponential Family. Entropy, 17, 4215–4254, 2015. [23] L. Malago and G. Pistone. Combinatorial Optimization with Information Geometry: The Newton Method. Entropy, 16, 4260–4289, 2014. [24] R.J. McCann. A Convexity Principle for Interacting Gases. Advances in mathematics, 128, 153–179, 1997. [25] K. Modin. Geometry of matrix decompositions seen through optimal transport and information geometry. Journal of Geometric Mechanics, 9 (3) : 335-390, 2017. [26] E. Nelson. Derivation of the SchrödingerEquation from Newtonian Mechanics. Physical Review, 150(4):1079–1085, 1966. [27] F. Nielsen. Emerging trends in visual computing, Lecture Notes in Computer Science 6, CD-ROM, 2009. [28] F. Nielsen. On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means. Entropy, 21(5), 485, 2019. [29] A. Takatsu. Wasserstein geometry of Gaussian measures. Osaka J. Math. 48, No. 4, 1005–1026, 2011. [30] F. Otto. The geometry of dissipative evolution equations: the porous medium equation. Communica- tions in Partial Differential Equations, 26(1-2):101–174, 2001. [31] G. Peyréand M. Cuturi. Computational Optimal Transport. Foundations and Trends in Machine Learning, vol. 11, no. 5-6, pp. 355-607, 2019. [32] C. Villani. Optimal Transport: Old and New, 2009. [33] T.L. Wong. Logarithmic divergences from optimal transport and Renyi geometry. Information Geom- etry, 1, 3978, 2018. [34] T.L. Wong and J.W. Yang. Optimal transport and information geometry. arXiv:1906.00030v3, 2019. [35] Y. Yang, B. Engquist, J. Sun, and B. D. Froese. Application of Optimal Transport and the Quadratic Wasserstein Metric to Full-Waveform Inversion. Geophysics, 2017. [36] W. Yin, S. Osher, J. Darbon, and D. Goldfarb. Bregman Iterative Algorithms for Compressed Sensing and Related Problems. SIAM Journal on Imaging Sciences, 1(1):143-168, 2008. 24 LI

Appendix

In this appendix, we ﬁrst present all derivation proofs in this paper. We next summarize several formulations of transport Bregman divergences. We last compare the convexity diﬀerence between KL divergence and transport KL divergence.

6.1. Proofs in section 3. We present all derivations of transport information Bregman divergences.

Proof of Proposition 4. Our proof follows from Deﬁnition 3. Notice that T (x) = ∇xΦp(x), x = ∇xΦq(x). δ (i) Since δq(x) V(q) = V (x), then

Z Z Z DT,V (pkq) = V (x)p(x)dx − V (x)q(x)dx − (T (x) − x, ∇xV (x))q(x)dx Ω Ω Ω Z = V (T (x)) − V (x) − (T (x) − x, ∇xV (x)) q(x)dx Ω Z = DV (T (x)kx)q(x)dx. Ω

δ R (ii) Since δq(x) W(q) = Ω W (x, x˜)q(˜x)dx˜, then

1 Z Z 1 Z Z DT,W (pkq) = W (x, x˜)p(x)p(˜x)dxdx˜ − W (x, x˜)q(x)q(˜x)dxdx˜ 2 Ω Ω 2 Ω Ω Z Z − (T (x) − x, ∇xW (x, x˜))q(x)q(˜x)dxdx˜ Ω Ω 1 Z Z 1 Z Z = W (T (x),T (˜x))q(x)q(˜x)dxdx˜ − W (x, x˜)q(x)q(˜x)dxdx˜ 2 Ω Ω 2 Ω Ω 1 Z Z − (T (x) − T (˜x) − (x − x˜), ∇xW (x, x˜))q(x)q(˜x)dxdx˜ 2 Ω Ω 1 Z Z = W˜ (T (x) − T (˜x)) − W˜ (x − x˜) 2 Ω Ω − (T (x) − T (˜x) − (x − x˜), ∇W˜ (x − x˜)) q(x)q(˜x)dxdx˜ 1 Z Z = DW˜ (T (x) − T (˜x)kx − x˜)q(x)q(˜x)dxdx.˜ 2 Ω Ω

In above derivations, the second equality uses the pushforward relation Z Z Z Z W (x, x˜)p(x)p(˜x)dxdx˜ = W (T (x),T (˜x))q(x)q(˜x)dxdx,˜ Ω Ω Ω Ω TRANSPORT INFORMATION BREGMAN DIVERGENCES 25 and applies the equality that Z Z (T (x) − x, ∇xW (x, x˜))q(x)q(˜x)dxdx˜ Ω Ω 1 Z Z 1 Z Z = (T (x) − x, ∇xW (x, x˜))q(x)q(˜x)dxdx˜ + (T (˜x) − x,˜ ∇W (˜x, x))q(x)q(˜x)dxdx˜ 2 Ω Ω 2 Ω Ω 1 Z Z = (T (x) − x − (T (˜x) − x˜), ∇xW (x, x˜))q(x)q(˜x)dxdx,˜ 2 Ω Ω where ∇xW (x, x˜) = −∇x˜W (x, x˜). δ 0 (iii) In this case, δq(x) U(q) = U (q(x)). Then Z 0 DT,U (pkq) = U(p) − U(q) − T (x) − x, ∇xU (q(x)) q(x)dx. (19) Ω Denote y = T (x), then Z U(p) = U(p(y))dy Ω Z = U(p(T (x)))det(∇xT (x))dx (20) Ω Z q(x) = U( )det(∇xT (x))dx, Ω det(∇xT (x)) where we use the fact p(T (x))det(∇xT (x)) = q(x). In addition, Z 0 − (T (x) − x, ∇xU (q(x)))q(x)dx Ω Z 0 = ∇x · q(x)(T (x) − x) U (q(x))dx Ω Z 0 0 = ∇xq(x),T (x) − x U (q(x)) + ∇x · (T (x) − x)U (q(x))q(x) dx (21) Ω Z Z 0 = ∇xU(q(x)),T (x) − x dx + ∇x · (T (x) − x)U (q(x))q(x) dx Ω Ω Z 0 = ∇x · (T (x) − x) U (q(x))q(x) − U(q(x)) dx. Ω Substituting (20) and (21) into (19), and formulating the results in term of a matrix divergence form, we derive the result.

Proof of Proposition 5. The proof follows the relation between the joint density and the mapping. (i) For the linear energy, we have Z DT,V (pkq) = DV (T (x)kx)q(x)dx Ω Z Z = DV (ykx)π(x, y)dxdy. Ω Ω 26 LI

(ii) For the interaction energy, we have 1 Z Z DT,W (pkq) = DW˜ (T (x) − T (˜x)kx − x˜)q(x)q(˜x)dxdx˜ 2 Ω Ω 1 Z Z Z Z = DW˜ (y − y˜kx − x˜)π(x, y)π(˜x, y˜)dxdydxd˜ y.˜ 2 Ω Ω Ω Ω (iii) For the entropy, we have Z DT,U (pkq) = U(p(x)) − U(q(x)) + (∇x · T (x) − d)U¯(q(x)) dx Ω Z Z Z Z = U( π(x, y)dx)dy − U( π(x, y)dy)dx Ω Ω Ω Ω Z Z Z + (∇x · yπ(x, y)dy − d)U¯( π(x, z)dz)dx Ω Ω Ω Z Z Z Z = U( π(x, y)dx)dy − U( π(x, y)dy)dx Ω Ω Ω Ω Z Z Z + ( y · ∇xπ(x, y)dy − d)U¯( π(x, z)dz)dx, Ω Ω Ω which ﬁnishes the proof.

Proof of Proposition 6. The proof applies the pushforward relation:

p(T (x))|∇xT (x)| = q(x). Notice that the optimal map T (x) in one dimensional spatial domain is monotonically increasing, i.e. ∇xT (x) ≥ 0. Then

p(T (x))∇xT (x) = q(x), i.e. dFp(T (x)) = dFq(x), where Fp, Fq are cumulative functions of p, q, respectively. One can solve the above equation by −1 T (x) = Fp (Fq(x)). −1 Denote y = Fq(x). Then x = Fq (y) and dy 1 1 q(x) = = = . dx dx −1 dy ∇yFq (y) We are now ready to derive closed form solutions for (5). (i) For (7), we have Z DT,V (pkq) = DV (T (x)kx)q(x)dx Ω Z −1 = DV (Fp (Fq(x))kx)dFq(x) Ω Z 1 −1 −1 = DV (Fp (y)kFq (y))dy. 0 TRANSPORT INFORMATION BREGMAN DIVERGENCES 27

(ii) For (8), we have 1 Z Z DT,W (pkq) = DW˜ (T (x) − T (˜x)kx − x˜)q(x)q(˜x)dxdx˜ 2 Ω Ω Z Z 1 −1 −1 = DW˜ (Fp (Fq(x)) − Fp (Fq(˜x))kx − x˜)dFq(x)dFq(˜x) 2 Ω Ω Z 1 Z 1 1 −1 −1 −1 −1 = DW˜ (Fp (y) − Fp (˜y)kFq (y) − Fq (˜y))dydy.˜ 2 0 0 (iii) For (9), we have Z n q(x) 0 o DT,U (pkq) = U( )∇xT (x) − U(q(x)) + ∇x · (T (x) − x) q(x)U (q(x)) − U(q(x)) dx Ω ∇xT (x) Z n q(x) −1 = U( −1 )∇xFp (Fq(x)) − U(q(x)) Ω ∇xFp (Fq(x)) −1 0 o + (∇xFp (Fq(x)) − 1) q(x)U (q(x)) − U(q(x)) dx Z 1 n 1 −1 1 = U( −1 −1 )∇xFp (y) − U( −1 ) 0 ∇yFq (y)∇xFp (y) ∇yFq (y) −1 1 0 1 1 o −1 + (∇xFp (y) − 1) −1 U ( −1 ) − U( −1 ) ∇yFq (y)dy ∇yFq (y) ∇yFq (y) ∇yFq (y) Z 1 n 1 −1 1 −1 = U( −1 )∇yFp (y) − U( −1 )∇yFq (y) 0 ∇yFp (y) ∇yFq (y) −1 −1 1 0 1 1 o + (∇yFp (y) − ∇yFq (y)) −1 U ( −1 ) − U( −1 ) dy ∇yFq (y) ∇yFq (y) ∇yFq (y) Z 1 n ˜ −1 ˜ −1 −1 −1 ˜ 0 −1 o = U(∇yFp (y)) − U(∇yFq (y)) − ∇yFp (y) − ∇yFq (y) U (∇yFq (y)) dy 0 Z 1 −1 −1 = DU˜ (∇yFp (y)k∇yFq (y))dy, 0 where the fourth equality applies the chain rule that −1 −1 −1 dy ∇yFp (y) y = Fq(x), ∇xFp (y) = ∇yFp (y) · = −1 , dx ∇yFq (y) and the last equality applies the fact that 1 1 1 1 U˜(z) = zU( ), U˜ 0(z) = U( ) − U 0( ). z z z z This ﬁnishes the proof.

6.2. Closed form formulas. In this subsection, we summarize the derived transport divergence functions as follows. Given p, q ∈ P(Ω), denote the convex function Φp :Ω → R, such that (∇xΦp)#q = p, i.e. 2 p(∇xΦp(x))det(∇xΦp(x)) = q(x). 28 LI

Deﬁne the transport Bregman divergence of functional F(p) by Z δ DT,F (pkq) = F(p) − F(q) − ∇x F(q), ∇xΦp(x) − x q(x)dx. Ω δq(x) Several examples of transport Bregman divergences are given below. R (i) Linear energy: If V(p) = Ω V (x)p(x)dx, then Z DT,V (pkq) = DV (∇xΦp(x)kx)q(x)dx, Ω

where DV is a Euclidean Bregman divergence of V deﬁned by

DV (z1kz2) = V (z1) − V (z2) − ∇V (z2) · (z1 − z2), for any z1, z2 ∈ Ω. 1 R R ˜ (ii) Interaction energy: If W(p) = 2 Ω Ω W (x − x˜)p(x)p(˜x)dxdx˜, then Z Z 1 DT,W (pkq) = DW˜ ∇xΦp(x) − ∇x˜Φp(˜x)kx − x˜ q(x)q(˜x)dxdx,˜ 2 Ω Ω ˜ where DW˜ is a Euclidean Bregman divergence of W deﬁned by ˜ ˜ ˜ DW˜ (z1kz2) = W (z1) − W (z2) − ∇W (z2) · (z1 − z2), for any z1, z2 ∈ Ω. R (iii) Negative entropy: If U(p) = Ω U(p(x))dx, then Z n q(x) 2 DT,U (pkq) = U( 2 )det(∇xΦp(x)) − U(q(x)) Ω det(∇xΦp(x)) 0 o + (∆xΦp(x) − d) q(x)U (q(x)) − U(q(x)) dx. R (iv) Transport KL divergence: If U(p) = Ω p(x) log p(x)dx, then Z 2 DTKL(pkq) = ∆xΦp(x) − log det(∇xΦp(x)) − d q(x)dx. Ω Given one dimensional sample space, several closed formulas of transport divergences are given below.

(i) Linear energy: Z 1 −1 −1 DT,V (pkq) = DV (Fp (x)kFq (x))dx. 0 (ii) Interaction energy: Z 1 Z 1 1 −1 −1 −1 −1 DT,W (pkq) = DW˜ (Fp (x) − Fp (˜x)kFq (x) − Fq (˜x))dxdx.˜ 2 0 0 (iii) Negative entropy: Z 1 −1 −1 DT,U (pkq) = DU˜ (∇xFp (x)k∇xFq (x))dx, 0 where 1 U˜(z) = zU( ), z TRANSPORT INFORMATION BREGMAN DIVERGENCES 29 ˜ and DU˜ is a Euclidean Bregman divergence of U deﬁned by ˜ ˜ ˜ DU˜ (z1kz2) = U(z1) − U(z2) − ∇zU(z2) · (z1 − z2).

p2 If U(p) = 2 , then Z 1 1 1 1 2 −1 DT,U (pkq) = ( −1 − −1 ) · ∇xFp (x)dx. 2 0 ∇xFp (x) ∇xFq (x) (iv) Transport KL divergence: Z 1 −1 −1 ∇xFp (x) ∇xFp (x) DTKL(pkq) = −1 − log −1 − 1 dx. 0 ∇xFq (x) ∇xFq (x) (v) Transport Jenson–Shannon divergence:

1 −1 −1 1 Z ∇xF (x) · ∇xF (x) D (pkq) = − log p q dx. TJS 2 1 −1 −1 2 0 4 (∇xFp (x) + ∇xFq (x)) Given Gaussian distributions, several analytical formulas for transport information Breg- man divergences are provided. Denote

1 − 1 xTΣ−1x 1 − 1 xTΣ−1x pX (x) = e 2 X , pY (x) = e 2 Y . p d p d (2π) det(ΣX ) (2π) det(ΣY ) (i) Transport KL divergence:

1 1 1 − 1 1 1 det(ΣY ) 2 2 2 2 2 DTKL(pX kpY ) = log + tr ΣX ΣX ΣY ΣX ΣX − d. 2 det(ΣX )

If ΣX ,ΣY commute, i.e. ΣX ΣY = ΣY ΣX , then

1 1 1 det(ΣY ) 2 − 2 DTKL(pX kpY ) = log + tr ΣX ΣY − d. 2 det(ΣX ) (ii) Transport Jenson-Shannon divergence:

1 det(ΣX )det(ΣY ) DTJS(pX kpY ) = − log 2 4 det(ΣZ ) 1 1 1 1 − 1 1 1 1 1 − 1 1 + tr Σ 2 Σ 2 Σ Σ 2 2 Σ 2 + Σ 2 Σ 2 Σ Σ 2 2 Σ 2 − d, 2 X X Z X X Y Y Z Y Y where 1 1 1 1 − 1 1 Σ = ( + )Σ ( + ), and = Σ 2 Σ 2 Σ Σ 2 2 Σ 2 . Z 4 I T Y I T T X X Y X X

If ΣX ,ΣY commute, then

1 1 1 det(Σ 2 )det(Σ 2 ) D (p kp ) = − log X Y . TJS X Y 1 1 2 1 2 2 2 det 4 (ΣX + ΣY ) 30 LI

6.3. Comparisons between KL divergence and transport KL divergence. In this subsection, we compare KL divergence with transport KL divergence. This is an example to compare Bergman divergences in L2 space and L2–Wasserstein space. To do so, we ﬁrst formulate the classical KL divergence in optimal transport coordinates. Proposition 12 (KL divergence in transport coordinates [3]). KL divergence has the following formulations: Z Z Z 2 DKL(pkq) = q(x) log q(x)dx − log det(∇xΦp(x))q(x)dx − log q(∇xΦp(x))q(x)dx, Ω Ω Ω where ∇xΦp#q = p.

Proof. Denote y = ∇xΦp(x). From the Monge-Amper´eequation, we have Z p(y) DKL(pkq) = p(y) log dy Ω q(y) Z p(∇xΦp(x)) 2 = p(∇xΦp(x)) log det(∇xΦp(x))dx Ω q(∇xΦp(x)) Z q(x) = q(x) log 2 dx Ω det(∇xΦp(x))q(∇xΦp(x)) Z Z Z 2 = q(x) log q(x)dx − q(x) log det(∇xΦp(x))dx − q(x) log q(∇xΦp(x))dx. Ω Ω Ω

We notice that the negative Boltzmann entropy term is convex w.r.t both p and ∇xΦp(x), i.e. Z Z 2 H(p) = − p(x) log p(x)dx = log det(∇xΦp(x))q(x)dx. Ω Ω This is true because t log t is convex w.r.t. t ∈ R+, and − log det(A) is convex w.r.t matrix d×d A ∈ R . Hence we know that the major diﬀerence between KL divergence and transport KL divergence is the corresponding cross entropy term. Here the cross entropy in L2 space leads to Z Z Hq(p) = − log q(x)p(x)dx = − log q(∇xΦp(x))q(x)dx. Ω Ω While the transport cross entropy in L2–Wasserstein space forms Z HT,q(p) = ∆xΦp(x)q(x)dx − C, Ω R where C = d + Ω q(x) log q(x)dx. To summarize, KL divergence DKL(pkq) = −H(p) + Hq(p) is convex w.r.t. mapping function ∇xΦp(x) if − log q(x) is convex in Ω. While transport KL divergence DTKL(pkq) = −H(p) + HT,q(p) is always convex w.r.t. mapping function ∇xΦp(x). Email address: [email protected]

Department of Mathematics, University of South Carolina.