Arxiv:1808.06148V5 [Math.ST]

arXiv:1808.06148v5 [math.ST] 20 Sep 2018 the the and t divergence a is man [12] divergence(KL-divergence) Kullback-Leibler the example, t and divergence Bregman the 16]. [8, between divergence relation the shown have and KL-divergence h esnSanndvrec J-iegne 1]i yeof type a is [13] divergence. (JS-divergence) divergence Jensen-Shannon The iegne[,10] [1, divergence J 2. 1. properties. following the e oei h edo ahn erig inlpoesn n oon. so and processing signal learning, machine of field the S in role key a and sn titycne ucin where function, convex strictly a using F,α and h rga iegneadtefdvreeicuewl-nw dive well-known include f-divergece the and divergence Bregman The o rbblt distributions probability For h rga iegne[7] divergence Bregman The iegne r ucin esr h iceac ewe two between discrepancy the measure functions are Divergences D D χ EEAIE RGA N ESNDIVERGENCES JENSEN AND BREGMAN GENERALIZED ( f ( ( ,q p, sur iegneadthe and divergence -square ,q p, ,q p, 1 .Bsdsta ile aeitoue h kwJne diver Jensen skew the introduced have Nielsen that Besides 0. = (1) ,q p, Abstract. itne h-qaedvrec,Apadvrec,Centr divergence, inequality. Alpha Lin’s divergence, d Chi-square Jensen-Shannon distance, divergence, Jeffreys divergence, Leibler Keywords: generaliz a is inequality this show equality. diverge g-Bregman and the divergence d between Kullback-Leibler g-Jensen inequality an the chi-sq derive to the we over, addition distance, in Hellinger alpha-divergence (the divergences the f-divergence include of or g-divergences class Bregman these a the show to We ske similar and vergence. properties divergence some (g-Bregman Jen satisfy skew classes vergence) the divergence and divergence new Bregman These the of definitions the ing ) 0 = ) ) def ≥ ∈ (1 = S o all for 0 HC NLD OEF-DIVERGENCES SOME INCLUDE WHICH iegnei enda function a as defined is divergence a , ⇐⇒ − D α nti ae,w nrdc e lse fdvrecsb ext by divergences of classes new introduce we paper, this In rga iegne -iegne esndvrec,Kul divergence, Jensen f-divergence, divergence, Bregman f ) p ( F ,q p, p = k ( p q + ) f q ) ∈ dvrec.O h te ad h elne distance, Hellinger the hand, other the On -divergence. def = S KL αF OOIONISHIYAMA TOMOHIRO P p 1. ( B i ( α q p q and F ) i dvrec 5 ]aeatp fthe of type a are 9] [5, -divergence k Introduction f ( − q ,q p, ( ) p q F q def i = i ahdvrec sdfie sfollows. as defined is divergence each , ) r ersnaiedvrecsdfie by defined divergences representative are ) ((1 ,f F, 1 def X = i − : F p α S i ( ) ln( p p → ) + − p q R i i αq F D (1) ) r titycne functions convex strictly are ( o paramter a for ) q : ) S aedvrec and divergence uare vrec,Hellinger ivergence, h − i,Parallelogram, oid, × vrec) More- ivergence). to fLnsin- Lin’s of ation c n h skew the and nce hc eogto belong which S kwJne di- Jensen skew p e divergence. sen -esndi- g-Jensen w − → q, R h kwJensen skew the p fteBreg- the of ype onsadplay and points ∇ ese Jensen skew he hc satisfies which lback- F f gne.For rgences. end- -divergence. ( ie set a Given q α ) i ∈ and gence (0 , f 1) - 2 TOMOHIRO NISHIYAMA

Hellinger distance

2 def 2 H (p, q) = (√pi √qi) (2) − Xi Pearson χ-square divergence

2 def (pi qi) D 2 (p q) = − (3) χ ,P k q Xi i Neyman χ-square divergence

2 def (pi qi) D 2 (p q) = − (4) χ ,N k p Xi i α-divergence

def 1 α 1−α Dα(p q) = (p q 1) (5) k α(α 1) i i − − Xi JS-divergence 1 p + q 1 p + q JS(p, q) def= KL p + KL q (6) 2 k 2 2 k 2 The main goal of this paper is to introduce new classes of divergences (”g- divergence”) and study properties of the g-divergences. First, we introduce the g-Bregman divergence, the symmetric g-Bregman divergence and the skew g-Jensen divergence by extending the deﬁnitions of the Breg- man divergence and the skew Jensen divergence respectively. This idea is based on Zhang’s papers [21, 22]. For the g-Bregman divergence, we show some geometrical properties (for triangle, parallelogram, centroids) and derive an inequality between the symmetric g-Bregman divergence and the skew g-Jensen divergence. Then, we show that the g-Bregman divergence includes the Hellinger distance, the Pearson and Neyman χ-square divergence and the α-divergence. Furthermore, we show that the skew g-Jensen divergence include the Hellinger distance and the α-divergence. These divergences have not only the properties of f-divergence but also some properties similar to the Bregman or the skew Jensen divergence. Finally we derive many inequalities by using an inequality between the g-Bregman divergence and skew g-Jensen divergence.

2. The g-Bregman divergence and the skew g-Jensen divergence 2.1. Definition of the g-Bregman divergence and the skew g-Jensen divergence. By using a injective function g, we define the g-Bregman divergence and the skew g-Jensen divergence. Definition 1. Let Ω Rd be a convex set and let p, q be points in Ω. Let F (x) be a strictly convex function⊂ F :Ω R. The Bregman divergence and the→ symmetric Bregman divergence are defined as def BF (p, q) = F (p) F (q) p q, F (q) (7) − − h − ∇ i def BF, (p, q) = BF (p, q)+ BF (q,p)= p q, F (p) F (q) . (8) sym h − ∇ − ∇ i The symbol , indicates an inner product. For the parameterh i α R 0, 1 , the scaled skew Jensen divergence [15,16] is defined as ∈ \{ } def 1 sJF,α(p, q) = (1 α)F (p)+ αF (q) F ((1 α)p + αq) . (9) α(1 α) − − − − GENERALIZED BREGMAN AND JENSEN DIVERGENCES WHICH INCLUDE SOME F-DIVERGENCES3

Nielsen has shown the relation between the Bregman divergence and the Jensen divergence as follows [16].

BF (q,p) = lim sJF,α(p, q) (10) α→0 BF (p, q) = lim sJF,α(p, q) (11) α→1 1 sJF,α(p, q)= (1 α)BF (p, (1 α)p+αq)+αBF (q, (1 α)p+αq) (12) α(1 α) − − − − Definition 2. (Definition of the g-convex set)Let S be a vector space over the real numbers and g be a injective function g : S S. If S satisfies the following condition, we call the set S ”g-convex set”. → For all p and q in S and all α in the interval (0, 1), the point g−1 (1 α)g(p)+ − αg(q) also belongs to S. Definition 3. (Definition of the g-divergences) Let Ω be a g-convex set and let g :Ω Ω be a function which satisfies g(p)= g(q) p = q. Let F :Ω R be a strictly→ convex function. Let α be a parameter in (0⇐⇒, 1). → We define the g-Bregman divergence, the g-symmetric Bregman divergence and the (scaled) skew g-Jensen divergence as follows.

g def BF (p, q) = BF (g(p),g(q)) (13) g def BF,sym(p, q) = BF,sym(g(p),g(q)) (14) g def sJF,α(p, q) = sJF,α(g(p),g(q)) (15) From the deﬁnition of function g, these functions satisfy divergence properties Dg(p, q)=0 g(p) = g(q) p = q and Dg(p, q) 0, where Dg denotes the g-divergences.⇐⇒ When g(p) is⇐⇒ equal to p for all p Ω,≥ the g-divergences are consistent with original divergences. ∈ In the following, we assume that g is a function having an inverse function.

2.2. Geometrical properties of the g-Bregman divergence. In this subsection, we show the g-Bregman divergence and the symmetric g-Bregman divergence satisfy some geometrical properties as well as the Bregman and the symmetric Bregman divergence. First, we show the g-Bregman divergence and the symmetric g-Bregman divergence satisfy the linearity, the generalized law of cosines and the generalized parallelogram law. Then, we show the point which minimize the weighted average of g-Bregman divergence (g-Bregman centroids) is equal to the quasi-arithmetic mean [2, 19]. This property can be used for clustering algorithms such as k-means algorithm [14]. Linearity For positive constant c1 and c2, g g g Bc1F1+c2F2 (p, q)= c1BF1 (p, q)+ c2BF2 (p, q) (16) holds. Proposition 1. (Generalized law of cosines) Let Ω be a g-convex set. For points p,q,r Ω, the following equations hold. ∈ g g g BF (p, q)= BF (p, r)+ BF (r, q) g(p) g(r), F (g(q)) F (g(r)) (17) g − h − g ∇ − ∇g i BF,sym(p, q)= BF,sym(p, r)+ BF,sym(r, q) (18) g(p) g(r), F (g(q)) F (g(r)) + g(q) g(r), F (g(p)) F (g(r)) −h − ∇ − ∇ i h − ∇ − ∇ i 4 TOMOHIRO NISHIYAMA

It is easily proved in the same way as the Bregman divergence by putting p′ = g(p), q′ = g(q) and r′ = g(r). Theorem 1. (Generalized parallelogram law) Let Ω bea g-convex set. When points p,q,r,s Ω satisfy g(p)+g(r)= g(q)+g(s), the following equations holds. ∈ g g g g g g BF,sym(p, q)+BF,sym(q, r)+BF,sym(r, s)+BF,sym(s,p)= BF,sym(p, r)+BF,sym(q,s) (19) The left hand side is the sum of four sides of rectangle pqrs and the right hand side is the sum of diagonal lines of rectangle pqrs. Proof. It has been shown that the Bregman divergence satisﬁes the four point identity(see equation (2.3) in [20]).

F (r) F (s),p q = BF (q, r)+ BF (p,s) BF (p, r) BF (q,s) (20) h∇ − ∇ − i − − By putting p′ = g(p), q′ = g(q), r′ = g(r) and s′ = g(s), we can prove the g- Bregman divergence satisfies the similar four point identity. F (g(r)) F (g(s)),g(p) g(q) = Bg (q, r)+Bg (p,s) Bg (p, r) Bg (q,s) (21) h∇ −∇ − i F F − F − F By combining the assumption and the definition of the symmetric g-Bregman divergence ((8) and (14)), we have Bg (r, s)= Bg (q, r)+ Bg (p,s) Bg (p, r) Bg (q,s). (22) − F,sym F F − F − F Exchanging p r and q s in (22) and taking the sum with (22), the result follows. This theorem↔ can↔ be also proved in the same way as mentioned in the paper [18]. Definition 4. (Definition of the multivariate skew g-Jensen divergence) Let Ω be a g-convex set and let g :Ω Ω be a function which satisfies g(p)= g(q) p = q. → ⇐⇒ Let F : Ω R be a strictly convex function. Let pν (ν = 1, 2, 3 ,N) are points → ··· N in Ω. Let αν 0(ν =1, 2, 3 ,N) be parameters which satisfy αν = 1. ≥ ··· ν=1 We define the multivariate skew g-Jensen divergence as follows.P N N g def J α(p ,p , ,pN ) = αν F (g(pν )) F αν g(pν ) , (23) F, 1 2 ··· − νX=1 Xν=1 where α denotes a vector (α , α , , αN ). 1 2 ··· Theorem 2. (The centroids of the g-Bregman divergence) Let Ω be a g-convex set. Let pν (ν = 1, 2, 3 ,N) and q are points in Ω. Let ··· N αν 0(ν =1, 2, 3 ,N) be parameters which satisfy αν = 1. ≥ ··· ν=1 Then, the following inequality holds. P N g g αν B (pν , q) J α(p ,p , ,pN ) (24) F ≥ F, 1 2 ··· Xν=1 Equality holds if and only if N −1 q = g αν g(pν ) (25) Xν=1 Because the function g is injective by the definition of the g-Bregman divergence, this value is the quasi-arithmetic mean. For example, p (q: the arithmetic mean) (26a)  ln p (q: the geometric mean) (26b) g(p)=  −1  p (q: the harmonic mean) (26c) pr (q: the power mean). (26d)   GENERALIZED BREGMAN AND JENSEN DIVERGENCES WHICH INCLUDE SOME F-DIVERGENCES5

Proof. We can prove this theorem in the same way as Theorem 3.1 in [17]. By the deﬁnition of the g-Bregman divergence, we have

N N g αν B (pν , q)= αν F (g(pν )) F (g(q)) g(pν ) g(q), F (g(q)) (27) F − − h − ∇ i νX=1 νX=1

′ def N Let g(p ) = ν=1 αν g(pν ). The RHS of (27) yields N P N ′ ′ αν F (g(pν )) F (g(p )) + F (g(p )) F (g(q)) αν g(pν ) g(q), F (g(q)) − − − h − ∇ i Xν=1 νX=1 (28) N N ′ ′ = αν F (g(pν )) F (g(p )) + F (g(p )) F (g(q)) αν g(pν ) g(q), F (g(q)) − − − h − ∇ i Xν=1 νX=1 N N ′ g ′ ′ = αν F (g(pν )) F (g(p )) + B (p , q) αν F (g(pν )) F (g(p )) − F ≥ − νX=1 Xν=1 N N g = αν F (g(pν )) F αν g(pν ) = J α(p ,p , ,pN ). − F, 1 2 ··· νX=1 Xν=1

g ′ ′ −1 N Because B (p , q) equal to zero if and only if q = p = g αν g(pν ) , we F ν=1 prove the theorem. P Proposition 2. For the g-Bregman divergence, the following equation holds.

g gˆ BF (q,p)= BF ∗ (p, q), (29) where F ∗(x∗) def= sup x∗, x F (x) denotes the Legendre convex conjugate and x{h i− } gˆ def= F g. ∇ ◦ ′ ′ ∗ ′ ′ Proof. For the Bregman divergence, the equation BF (q ,p )= BF ( F (p ), F (q )) holds [16, 17]. By putting p′ = g(p) and q′ = g(q), the result follows.∇ ∇

Corollary 1. Let Ω be a g-convex set. Let pν (ν =1, 2, 3 ,N) and q are points ··· N in Ω. Let αν 0(ν =1, 2, 3 ,N) be parameters which satisfy αν = 1. ≥ ··· ν=1 Then, the following inequality holds. P N g gˆ αν B (q,pν ) J ∗ α(p ,p , ,pN ) (30) F ≥ F , 1 2 ··· Xν=1 Equality holds if and only if

N −1 q =g ˆ αν gˆ(pν ) (31) Xν=1 Proof. From Proposition 2, we obtain

N N g gˆ αν BF (q,pν )= αν BF ∗ (pν , q) (32) νX=1 νX=1 Applying Theorem 2 to this equation, the result follows. From Theorem 2 and Corollary 1, we conclude the point which minimize the weighted average of the g-Bregman divergence is unique. 6 TOMOHIRO NISHIYAMA

2.3. Relation between the g-Bregman and skew g-Jensen divergence. In this subsection, we derive inequality between the g-Bregman and the skew g-Jensen divergence. The following equations also hold for the g-Bregman and the skew g- Jensen divergence.

g g BF (q,p) = lim sJF,α(p, q) (33) α↓0 g g BF (p, q) = lim sJF,α(p, q) (34) α↑1 It is easily proved by putting p′ = g(p) and q′ = g(q) in (10) and (11). Then, we show some lemmas. Lemma 1. Let Ω be a g-convex set and let p, q be points p, q Ω. For a parameter α (0, 1) and a point r Ω which satisfies g(r) = (1 ∈α)g(p) + αg(q), the g-Bregman∈ divergence can be∈ expressed as follows. − 1 sJ g (p, q) def= (1 α)Bg (p, r)+ αBg (q, r) (35) F,α α(1 α) − F F − It is easily proved in the same way as equation (12) by putting p′ = g(p) and q′ = g(q). Lemma 2. Let Ω be a g-convex set and let p, q be points p, q Ω. For a parameter α (0, 1) and a point r Ω which satisfies g(r)=(1 α)g(p)+∈αg(q), the following equations∈ hold. ∈ − α Bg (p, q)= Bg (p, r)+ Bg (r, q)+ Bg (r, q) (36) F F F 1 α F,sym 1 − α Bg (q,p)= Bg (r, p)+ Bg (q, r)+ − Bg (r, p) (37) F F F α F,sym Proof. We first prove (36). By the generalized law of cosines (17), we have Bg (p, q)= Bg (p, r)+ Bg (r, q)+ g(r) g(p), F (g(q)) F (g(r)). (38) F F F h − ∇ − ∇ From the assumption, the equation α g(r) g(p)= (g(q) g(r)) (39) − 1 α − − holds. By substituting (39) to (38) and using (14), the result follows. By exchanging p and q in (38), we can prove (37) in the same way.

Lemma 3. Let Ω be a g-convex set and let p, q be points p, q Ω. For a parameter α (0, 1) and a point r Ω which satisﬁes g(r)=(1 α)g(p)+∈αg(q), the following equation∈ holds. ∈ − 1 1 Bg (p, q)= Bg (p, r)+ Bg (r, q) (40) F,sym α F,sym 1 α F,sym − Proof. Taking the sum of (36) and (37), the result follows.

Theorem 3. (Bregman-Jensen inequality) Let Ω be a g-convex set and let p, q be points in Ω. For a parameter α (0, 1), the skew g-Jensen divergence and the symmetric g-Bregman divergence, the∈ following inequality holds. Bg (p, q) sJ g (p, q) (41) F,sym ≥ F,α GENERALIZED BREGMAN AND JENSEN DIVERGENCES WHICH INCLUDE SOME F-DIVERGENCES7

Proof. Let r Ω be a point which satisﬁes g(r) = (1 α)g(p)+ αg(q). From (35) and using B∈g (p, q) Bg (p, q), we have − F ≤ F,sym 1 1 1 1 sJ g (p, q)= Bg (p, r)+ Bg (q, r) Bg (p, r)+ Bg (q, r). F,α α F 1 α F ≤ α F,sym 1 α F,sym − − (42) Using Lemma 3, we have 1 1 sJ g (p, q) Bg (p, r)+ Bg (r, q)= Bg (p, q) (43) F,α ≤ α F,sym 1 α F,sym F,sym − Corollary 2. (Bregman-Jensen inequality for parallelograms) Let Ω be a g-convex set and let p,q,r,s Ω be points which satisfy g(p)+ g(r)= g(q)+ g(s) (The points p,q,r,s are vertices∈ of ”parallelogram”). α 1 1 1 1 Let = ( 4 , 4 , 4 , 4 ). Then, the following inequality holds.

1 g g g g g B (p, q)+ B (q, r)+ B (r, s)+ B (s,p) J α(p,q,r,s) 8 F,sym F,sym F,sym F,sym ≥ F, (44) Proof. Let c Ω be a point which satisﬁes g(c)= 1 g(p)+ g(r) = 1 g(q)+ g(s) . ∈ 2 2 From Theorem 3, we have

g g 1 1 BF,sym(p, r) sJ 1 (p, r)=4 F (g(p)) + F (g(r)) F (g(c)) (45) ≥ F, 2 2 2 −

g g 1 1 BF,sym(q,s) sJ 1 (q,s)=4 F (g(q)) + F (g(s)) F (g(c)) (46) ≥ F, 2 2 2 − Taking the sum of (45) and (46) and applying Theorem 1, we have g g g g BF,sym(p, q)+ BF,sym(q, r)+ BF,sym(r, s)+ BF,sym(s,p) (47)

2 F (g(p)) + F (g(q)) + F (g(r)) + F (g(s)) 8F (g(c)) ≥ − 1 By the assumption, we get g(c) = 4 g(p)+ g(q)+ g(r)+ g(s) . By the deﬁnition of the multivariate skew g-Jensen divergence (Deﬁnition 4), the result follows.

3. Examples We show some examples of the g-divergences. We denote by ”GBD” the g-Bregman divergence, by ”SGBD” the symmetric g-Bregman divergence, ”SGJD” by the skew g-Jensen divergence and by ”BJ- inequality” the Bregman-Jensen inequality (41). A parameter α is in (0, 1) except for example 3) and we use a parameter β (0, 1) ∈ instead of α in example 3). The variables pi and qi are non-negative real numbers for all i. From example 1) to 6), we show the case that GBD belong to a class of f- divergence and example 7) is the case of information geometry [3–5] and statistical mechanics.

1) KL-divergence, Jeﬀreys divergence, JS-divergence

Generating functions: F (p)= pi ln pi, g(p)= p • i P def pi GBD: generalized KL-divergence KL(p q) = ( pi + qi + pi ln( )) • k i − qi P 8 TOMOHIRO NISHIYAMA

def pi SGBD: Jeﬀreys divergence (J-divergence) [11] J(p, q) = (pi qi) ln( ) • i − qi P SGJD: scaled skew JS-divergence • 1 def 1 JSα(p q) = (1 α)KL(p (1 α)p + αq)+ αKL(q (1 α(1−α) k α(1−α) − k − k − α)p + αq)

BJ-inequality: generalized Lin’s inequality α(1 α)J(p, q) JSα(p, q). • 1 − ≥ When α is equall to 2 , this inequality is consistent with Lin’s inequality [13].

2) KL-divergence, Jeﬀreys divergence, α-divergence

Generating functions: F (p)= exp(pi), g(p) = ln p • i P def qi GBD: generalized reverse KL-divergence KL(q p) = (pi qi +qi ln( )) • k i − pi P def pi SGBD: J-divergence J(p, q) = (pi qi) ln( ) • i − qi P SGJD: generalized α-divergence D1−α(p q), • def 1 α 1−α k where Dα(p q) = (p q αpi (1 α)qi) k i α(α−1) i i − − − P BJ-inequality: J(p, q) Dα(p q) • ≥ k

3) α-divergence

1 Generating functions: F (p)= 1 p α , g(p)= pα for α R 0, 1 • 1−α i i ∈ \{ } P GBD: generalized α-divergence Dα(p q) • k def SGBD: symmetric α-divergence Dα, (p, q) = Dα(p q)+ Dα(q p) • sym k k 1 1 α α α SGJD: (1 β)pi + βqi (1 β)p + βq • β(1−β)(1−α) i − − − i i P 1 1 α α α BJ-inequality: Dα, (p, q) (1 β)pi+βqi (1 β)p + βq • sym ≥ β(1−β)(1−α) i − − − i i P 1 2 By using the equations D 1 (p q) = H (p, q), 2D (p q) = Dχ2,P (p q) and 2 ( 2 ) k (2) k k 2D (p q)= D 2 (p q) [9], we obtain example 4) to 6). (−1) k χ ,N k 4) Hellinger distance

Generating functions: F (p)= p2, g(p)= √p • i i P 2 def 2 GBD: Hellinger distance H (p, q) = (√pi √qi) • i − P SGBD: Hellinger distance 2H2(p, q) • SGJD: Hellinger distance H2(p, q) • GENERALIZED BREGMAN AND JENSEN DIVERGENCES WHICH INCLUDE SOME F-DIVERGENCES9

BJ-inequality: trivial •

5) Pearson χ-square divergence

2 Generating functions: F (p)= 2 √pi, g(p)= p • − i P 2 def (pi−qi) GBD: Pearson χ-square divergence D 2 (q p) = • χ ,P k i qi P def SGBD: symmetric χ-square divergence D 2 (p, q) = D 2 (p q)+D 2 (q p) • χ ,sym χ k χ k 2 2 2 SGJD: ( (1 α)pi αqi + (1 α)p + αq ) • α(1−α) i − − − − i i P p 2 2 2 BJ-inequality: D 2 (p, q) ( (1 α)pi αqi+ (1 α)p + αq ) • χ ,sym ≥ α(1−α) i − − − − i i P p 6) Neyman χ-square divergence

−1 −1 Generating functions: F (p)= pi , g(p)= p • i P 2 def (pi−qi) GBD: Neyman χ-square divergence D 2 (q p) = • χ ,N k i pi P SGBD: symmetric χ-square divergence D 2 (p, q) • χ ,sym 1 piqi SGJD: ((1 α)pi + αqi ) • α(1−α) i − − (1−α)qi+αpi P 1 piqi BJ-inequality: D 2 (p, q) ((1 α)pi + αqi ) • χ ,sym ≥ α(1−α) i − − (1−α)qi+αpi P 7) Information geometry and statistical mechanics i In the dually ﬂat space M, there exist two aﬃne coordinates θ , ηi and two convex i functions called potential ψ(θ), φ(η). For the points P,Q M, we put pi = θ (P ), i ∈ qi = θ (Q) or pi = ηi(P ), qi = ηi(Q).

Generating functions: F (p)= ψ(θ) or F (p)= φ(η), g(p)= p • def i GBD: canonical divergence D(P Q) = φ(η(Q))+ψ(θ(P )) ηi(Q)θ (P ) • k − i P def i i SGBD: DA(P,Q) = (ηi(Q) ηi(P ))(θ (Q) θ (P )) • i − − P SGJD: sJψ,α(θ(P ),θ(Q)) or sJφ,α(η(P ), η(Q)) •

BJ-inequality: DA(P,Q) sJψ,α(θ(P ),θ(Q)) or DA(P,Q) sJφ,α(η(P ), η(Q)) • ≥ ≥

For the exponential or mixture families, GBD is equal to the KL-divergence and SGBD is equal to the J-divergence. For the exponential families, SGJD sJψ,α(θ(P ),θ(Q)) is equal to the skew Bhattacharyya distance [6, 16], and for the mixture families , SGJD sJφ,α(η(P ), η(Q)) is equal to the skew-JS divergence(see [18] in detail). In the case of canonical ensemble in statistical mechanics, the probability distribution function belongs to exponential families. Because the manifold of the exponential family is a dually ﬂat space, the same equations and inequality hold 10 TOMOHIRO NISHIYAMA for 1 θ = β (48) − ≡−kBT η = U (49) ψ = βF (50) − 1 φ = S, (51) −kB where kB is the Boltzmann constant, T is temperature, U is the internal energy, S is the entropy and F is the Helmholtz free energy. Deﬁning a parameter α as a index of the state which satisfy 1 = (1 α) 1 +α 1 Tα − T0 T1 or Uα = (1 α)U + αU , BJ-inequality can be written as follows. − 0 1 (T T )(U U ) Fα F F α(1 α) 1 − 0 1 − 0 (1 α) 0 α 1 (52) − T0T1 ≥ Tα − − T0 − T1 (T1 T0)(U1 U0) α(1 α) − − Sα (1 α)S0 αS1 (53) − T0T1 ≥ − − −

4. Conclusion We have introduced the g-Bregman divergence and the skew g-Jensen divergence by extending the deﬁnitions of the Bregman and the skew Jensen divergence respectively. First, we have shown the geometrical properties of the g-Bregman divergence and the symmetric g-Bregman divergence. Then, we have derived an inequality between the symmetric g-Bregman divergence and the skew g-Jensen divergence. Finally, we have shown they include divergences which belong to a class of f- divergence . If the relationship between these three classes of divergence are studied in more detail, it is expected to proceed applications to various ﬁelds including machine learning.

References

[1] Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society. Series B (Methodological), pages 131–142, 1966. [2] Shun-ichi Amari. Information geometry and its applications: Convex function and dually flat manifold. In Emerging Trends in Visual Computing, pages 75–102. Springer, 2009. [3] Shun-ichi Amari. Information geometry and its applications. Springer, 2016. [4] Shun-ichi Amari and Andrzej Cichocki. Information geometry of divergence functions. Bul- letin of the Polish Academy of Sciences: Technical Sciences, 58(1):183–195, 2010. [5] Shun-ichi Amari and Hiroshi Nagaoka. Methods of information geometry, volume 191. Amer- ican Mathematical Soc., 2007. [6] Anil Bhattacharyya. On a measure of divergence between two statistical populations defined by their probability distributions. Bull. Calcutta Math. Soc., 35:99–109, 1943. [7] Lev M Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7(3):200–217, 1967. [8] Jacob Burbea and C Radhakrishna Rao. On the convexity of some divergence measures based on entropy functions. Technical report, PITTSBURGH UNIV PA INST FOR STATISTICS AND APPLICATIONS, 1980. [9] Andrzej Cichocki and Shun-ichi Amari. Families of alpha-beta-and gamma-divergences: Flex- ible and robust measures of similarities. Entropy, 12(6):1532–1568, 2010. [10] Imre Csiszár. Information-type measures of difference of probability distributions and indirect observation. studia scientiarum Mathematicarum Hungarica, 2:229–318, 1967. GENERALIZED BREGMAN AND JENSEN DIVERGENCES WHICH INCLUDE SOME F-DIVERGENCES11

[11] Harold Jeffreys. An invariant form for the prior probability in estimation problems. Proc. R. Soc. Lond. A, 186(1007):453–461, 1946. [12] Solomon Kullback. Information theory and statistics. Courier Corporation, 1997. [13] Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145–151, 1991. [14] Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 28(2):129–137, 1982. [15] Frank Nielsen. A family of statistical symmetric divergences based on jensen’s inequality. arXiv preprint arXiv:1009.4004, 2010. [16] Frank Nielsen and Sylvain Boltz. The burbea-rao and bhattacharyya centroids. IEEE Trans- actions on Information Theory, 57(8):5455–5466, 2011. [17] Frank Nielsen and Richard Nock. Sided and symmetrized bregman centroids. IEEE transactions on Information Theory, 55(6):2882–2904, 2009. [18] Tomohiro Nishiyama. Divergence functions in dually flat spaces and their properties. arXiv preprint arXiv:1808.06482, 2018. [19] Emilio Porcu, Jorge Mateu, and George Christakos. Quasi-arithmetic means of covariance functions with potential applications to space–time data. Journal of Multivariate Analysis, 100(8):1830–1844, 2009. [20] Simeon Reich and Shoham Sabach. Two strong convergence theorems for bregman strongly nonexpansive operators in reflexive banach spaces. Nonlinear Analysis: Theory, Methods & Applications, 73(1):122–135, 2010. [21] Jun Zhang. Divergence function, duality, and convex analysis. Neural Computation, 16(1):159–195, 2004. [22] Jun Zhang. Nonparametric information geometry: From divergence function to referential- representational biduality on statistical manifolds. Entropy, 15(12):5384–5418, 2013.