Bounding the Entropic Region via Information Geometry Yunshu Liu John MacLaren Walsh Dept. of ECE, Drexel University, Philadelphia, PA 19104, USA [email protected] [email protected]

P Abstract—This paper suggests that information geometry may marginal distribution pX (x ) = x pX (x) via form a natural framework to deal with the unknown part of A A N \A N the boundary of entropic region. An application of information X H(X ) = pX (x ) log2 pX (x ) (1) geometry shows that distributions associated with Shannon facets A A A A A x − can be associated, in the right coordinates, with affine collections A of distributions. This observation allows an information geomet- One can stack these entropies of different non-empty subsets ric reinterpretation of the Shannon-type inequalities as arising into a 2N 1 dimensional vector h = (H(X ) ), from a Pythagorean style relationship. The set of distributions which can be− clearly viewed as h , a functionA of|A the ⊆ joint N which violate Ingleton’s inequality, and hence are linked with the (pX ) 2N 1 distribution pX . A vector h? R − is said to be entropic part of the entropic region which is yet undetermined, is shown ∈ also to have a surprising affine information geometric structure if there is some joint distribution pX such that h? = h(pX ). in a special case involving four random variables and a certain The region of entropic vectors is then the image of the set = support. These facts provide strong evidence for the link between P D pX pX (x) 0, pX (x) = 1 of valid joint probability information geometry and characterizing the boundary of the x mass{ | functions≥ under the function }h( ), and is denoted by entropic region. · 2N 1 Γ∗ = h( ) (R − (2) N D I.INTRODUCTION ¯ It is known that the closure of this set ΓN∗ is a convex cone The region of entropic vectors is a convex cone that is [1], but surprisingly little else is known about this cone for known to be at the core of many yet undetermined fundamental N 4. Understanding the “shape” and boundaries of the set ¯ ≥ limits for problems in data compression, network coding, and ΓN∗ is the subject of this paper. The fundamental importance of ¯ multimedia transmission. This set has recently shown to be ΓN∗ lies in several contexts in signal processing, compression, non-polyhedral, but its boundaries remain unknown. In II, network coding and information theory [1]. we give some background on the region of entropic vectors,§ A. Outer Bounds and Inner Bounds which include discussion about inner and outer bounds for Viewed as a function h = H(X ) of the selected subset, it, and the gap between Shannon outer bound and Ingleton A A inner bound for 4 random variables. In III we point out with the convention that h = 0, entropy is sub-modular [1], ∅ that existing inner and outer bounds, which§ based on linear [2], meaning that constructions, are unable to parameterize the boundary of h + h h + h , , (3) the region of entropic vectors because non-linear dependence A B ≥ A∩B A∪B ∀A B ⊆ N structures are necessary to parameterize the unknown part of and is also non-decreasing and non-negative, meaning that it. We argue that information geometry may be a tool that h h 0 . (4) could parameterize this unknown part of the boundary where A ≥ B ≥ ∀B ⊆ A ⊆ N the best known bounds do not meet. This claim is partially Viewed as requirements for arbitrary set functions (not neces- substantiated by providing in III-B an information geometric sarily entropy) the inequalities (3) and (4) are known as the § interpretation of Shannon facets of the region of entropic polymatroidal axioms [1], [2], and a function obeying them is vectors and the Shannon type inequalities, as well as by called the rank function of a polymatroid. If a set function r providing in III-C an information geometric characterization that obeys the polymatroidal axioms (3) and (4) additionally § of a special type of distributions at four random variables that obeys violate Ingleton Inequalities. The paper concludes in IV with r , r Z (5) a number of interesting directions for future work. § A ≤ |A| A ∈ ∀A ⊆ N then it is the rank function of a matroid on the ground set . N II.BOUNDINGTHE REGIONOF ENTROPIC VECTORS Since entropy must obey the polymatroidal axioms, the set of all rank functions of polymatroids forms an outer bound Consider N discrete random variables X = X = N for the region of entropic vectors which is often denoted by (X1,...,XN ), = 1,...,N with joint probability mass N { }  2N 1  function pX (x). To every non-empty subset of these random h R −  ∈  variables X := (Xn n ), 1,...,N , there ΓN = h h + h h + h , (6) A | ∈ A A ⊂ { } A B ≥ A∩B A∪B ∀A B ⊆ N is associated a Shannon entropy H(X ) calculated from the  h h 0  A P ≥ Q ≥ ∀Q ⊆ P ⊆ N  is a polyhedron, this polyhedral set is often known as 3 if K ik, jk, il, jl, kl ΓN ΓN fij(K) = ∈ { } ¯ min 4, 2 K otherwise the Shannon outer bound for ΓN∗ [1], [2]. { | |} While in the low dimensional cases we have Γ = Γ∗ and 2 2 ij Γ = Γ¯∗, for N 4, however, Γ = Γ¯∗ . Zhang and Yeung Lemma 1: (Matu´s)[ˇ 14] The cone G = h 3 3 ≥ N 6 N 4 { ∈ first showed this in [2] by proving a new inequality among 4 Γ4 Ingletonij 0 , i, j N distinct is the convex hull variables of| 15 extreme rays.≤ } They∈ are generated by the 15 linearly ijk ijl ikl jkl i 2I(C; D) ≤ I(A; B) + I(A; C,D) + 3I(C; D|A) + I(C; D|B) independent functions fij, r1 , r1 , r1 , r1 , r1∅, r3∅, r1, rj, rik, rjk, ril, rjl, rk, rl , where kl = N ij. which held for entropies and was not implied by the poly- 1 1 1 1 1 2 2 − ¯ matroidal axioms, which they dubbed a non-Shannon type As identified early, the region of entropic vector Γ4∗ is known ij inequality to distinguish it from inequalities implied by ΓN . as long as we know the structure of six cones G4 . Because For roughly the next decade a few authors produced other new of symmetry, we only need focus on one of the six cones, in 34 non-Shannon inequalities [3], [4]. In 2007, Matu´sˇ [5] showed the rest of the paper, we will focus on understanding G4 . that Γ¯∗ is not a polyhedron for N 4. The proof of this N ≥ fact was carried out by constructing three sequences of non- III.PERSPECTIVESON HOWTO USE INFORMATION Shannon inequalities, including GEOMETRY TO UNDERSTAND ENTROPY GEOMETRY ¯ s[I(X1; X2|X3) + I(X1; X2|X4) + I(X3; X4) − I(X1; X2)] As identified in the previous sections, ΓN∗ is a convex s(s + 1) cone that is known to be non-polyhedral, which suggests that +I(X ; X |X ) + [I(X ; X |X ) + I(X ; X |X )] ≥ 0 2 3 1 2 1 3 2 1 2 3 differential geometric tools may be necessary to characterize ¯ Additionally, Matu´sˇ constructed a curve known to be in ΓN∗ it. Information geometry, on the other hand, is a discipline which, together with the infinite sequences of inequalities, was in statistics which endows the manifold of probability dis- ¯ geometrically arranged in a manner that prohibits ΓN∗ from tributions with a special “dually flat” differential geometric being a polyhedron. Nonetheless, despite this characterization, structure created by selecting a Riemannian metric based on ¯ the outer bound of even Γ4∗ is still far away from tight. the Fisher information and a family of affine connections Alternately, from the inside, the most common way to called the α-connections. Building on these observations, here generate inner bounds for the region of entropic vectors is to we wish to sketch out a couple of ideas in an attempt to show ¯ consider special families of distributions for which the entropy that characterizing the boundary of ΓN∗ could be thought of function is known to have certain properties. [6] talked about as a problem in information geometry, and that information generating inner bounds through representable matroid. [7], geometry may in fact be a convenient framework to utilize to ¯ [8], [9], [10], [11] focus on calculating inner bounds based on calculate the boundaries of ΓN∗ . special properties of binary random variables. A third way to A. Review of Some Relevant Ideas from Information Geometry generate inner bounds is based on inequalities for subspace Information geometry endows a manifold of probability dis- arrangements. Define N to be the conic hull of all subspace S tributions p(x; ξ), parameterized by a vector of real numbers ranks for N subspaces, it is known that N is an inner bound ¯ S ξ = [ξi], with a Riemannian metric, or inner product between for ΓN∗ [12]. At present, N is only known for N 5. We have ¯ ¯ S ¯ ≤ tangent vectors, given by the Fisher information: 2 = Γ2∗ = Γ2, 3 = Γ3∗ = Γ3 and 4 Γ4∗ Γ4, where 4 is givenS by the ShannonS type inequalitiesS ⊂ (i.e.⊂ the polymatroidalS ∂ log p(x; ξ) ∂ log p(x; ξ) gi,j(ξ) , Eξ[ ] (8) axioms) together with an additional form of inequality known ∂ξi ∂ξj as Ingleton’s inequality [12], [13], [14] which states that for This allows us to calculate an inner product between two N = 4 random variables tangent vectors c = P c ∂ and d = P d ∂ at a particular Ingleton = I(X ; X X ) + I(X ; X X ) (7) i i ξi i i ξi ij k l| i k l| j point ξ in the manifold of probability distributions, as +I(Xi; Xj) I(Xk; Xl) 0 X − ≥ < c, d > = c d g (ξ) (9) B. The Gap between and Γ ξ i j i,j S4 4 i,j Since the region of entropic vector Γ is unknown starting N∗ Selecting this Riemannian metric (indeed, we could have from N = 4, we focus on N = 4 in this paper. As we know, selected others), and some additional structure, allows the dif- Γ , where Γ is generated by 28 elemental Shannon type 4 ( 4 4 ferential geometric structure to be related to familiar properties informationS inequalities[1] and is generated by the previous 4 of exponential families and mixture families of probability 28 Shannon type informationS inequalities together with six distributions. While the resulting theory is very elegant, it Ingleton’s inequalities (7). Matu´sˇ in [14] pointed out that Γ is 4 is also somewhat complex, and hence we must omit the the disjoint union of and six sets h Γ Ingleton < 0 . 4 4 ij general details, referring the interested reader to [15], and only The six cones ijS { ∈ | are} G4 = h Γ4 Ingletonij 0 introduce a small subset of the concepts. symmetric due to the permutation{ ∈ of| inequalities Ingleton≤ } , ij In particular, let’s focus our attention on the manifold so it sufficient to study only one of the cones. Furthermore, (X ) of probability distributions for a random vector X = Matu´sˇ give the extreme rays of Gij in Lemma1 by using the 4 [PX ,...,X ] taking values on the Cartesian product X = following functions: for N = 1, 2, 3, 4 , with I N and 1 N { } ⊆ 1 2 N , where n is a finite set with values 0 t N I , define X × X × · · · × X X ≤ ≤ | \ | denoted by vi,n, i 1,..., n . To a particular probability I ∈ { |X |} QN r (J) = min t, J I with J N mass function pX we can associate a length 1 t { | \ |} ⊆ n=1 |Xn| − → vector, called the m-coordinates or η-coordinates, by listing m-geodesic connecting pX to Π (pX ) is orthogonal, in the E → the probabilities of all but one of the outcomes in X into sense of achieving Riemannian metric value 0, at Π (pX ) to E a vector. Such an η can be determined uniquely from the → the tangent vector of the e-geodesic connecting Π (pX ) and probability mass function pX , and owing to the fact that E any other point in . Additionally, for any other point qX , the probability mass function must sum to one, the omitted we have the PythagoreanE like relation ∈ E probability can be calculated, and hence the probability mass → → function can be determined from η. D(pX qX ) = D(pX Π (pX )) + D(Π (pX ) qX ). (14) Alternatively we could parameterize the probability mass || || E E || function for such a joint distribution with a vector θ, whose B. Information Geometric Structure of the Shannon Exposed QN n 1 elements take the form Faces of the Region of Entropic Vectors n=1 |X | −   ¯   ik 1, 2,..., k , As identified in II-A, Γ2 = Γ2∗ and Γ3 = Γ3∗, implying that pX(vi1,1, . . . , viN ,N ) ∈ { |X |} ¯ § θ = log k 1,...,N ,  Γ2∗ and Γ3∗ are fully characterized by Shannon type information pX(v1,1, . . . , v1,N ) ∈Q { } ¯ k ik = 1. inequalities. For N = 4, even though Γ4 = Γ4∗ and the region 6 is not a polyhedral cone, there are still many6 exposed faces These coordinates provide an alternate unique way of spec- of Γ¯ defined by attaining equality in a particular Shannon ifying the joint probability mass function p , called the e- N∗ X type information inequality of the form (3) or (4). Such coordinates or θ coordinates. exposed faces of Γ¯ could be referred to as the “Shannon A subfamily of these probability mass functions associated N∗ facets” of entropy, and in this section we aim to characterize with those η coordinates that take the form the distributions associated with these Shannon facets via η = Ap + b (10) information geometry. Let ⊥ be the submanifold of probability distributions for for some p for any particular fixed A and b, that is, that lie in EA which X and X c = X are independent an affine submanifold of the η coordinates, are said to form A A N \A a m-autoparallel submanifold of probability mass functions. ⊥ = pX ( ) pX (x) = pX (x )pX c (x c ) x (15) This is not a definition, but rather a consequence of a theorem EA { · | A A A A ∀ } , involving a great deal of additional structure which we must then we define ⊥ , ↔, ⊥ and as follows EA∪B EA B MA omit here [15].  ⊥ = pX ( ) pX = pX( ) pX( )c Similarly, a subfamily of these probability mass functions EA∪B n · A∪B A∪B o , associated with those θ coordinates that take the form ↔, ⊥ = pX ( ) pX = pX pX X pX( )c EA B · A B\A| A∩B A∪B θ = Aλ + b (11) = pX ( ) pX = pX δ c MA { · | A · A } for some λ for any particular fixed A and b, that is, that lie  c in an affine submanifold of the θ coordinates, are said to form 1 if Xi = v1,i i , where δ c = ∀ ∈ A . Note ↔, ⊥ is a a e-autoparallel submanifold of probability mass functions. A 0 otherwise EA B An e-autoparallel submanifold (resp. m-autoparallel sub- submanifold of ⊥ such that the random variables X , EA∪B A∪B manifold) that is one dimensional, in that its λ (resp. p) in addition to being independent from X( )c form the A∪B parameter vector is in fact a scalar, is called a e-geodesic (resp. Markov chain X X X . These sets of A\B ↔ A∩B ↔ B\A m-geodesic). distributions will be useful because I(X ; X( )c ) = A∪B A∪B On this manifold of probability mass functions for random h + h( )c h = 0 for every distribution in ⊥ A∪B , A∪B − N EA∪B variables taking values in the set X , we can also define the and ↔, ⊥, I(X ; X X ) = h + h h( ) Kullback Leibler divergence, or relative entropy, measured in EA B A\B B\A| A∩B , A B − A∩B − h( ) = 0 for every distribution in ↔, ⊥, and h h = 0 bits, according to forA∪B every distribution in . EA B N − A MA  x  X pX ( ) , , D(pX qX ) = pX (x) log2 (12) Proposition 1: ↔ ⊥ ⊥ (X ), ↔ ⊥ is an || qX (x) , , x X EA B ⊂ EA∪B ⊂ P EA B ∈ e-autoparallel submanifold of ⊥ and ⊥ is an e- EA∪B EA∪B Note that in this context D(p q) 0 with equality iff. p = q, autoparallel submanifold of (X ). P hence this function is a bit like|| ≥ a distance, however it does Proposition 2: Let , 1,...,N be given, where not in general satisfy symmetry or triangle inequality. , then A B ⊆ {(X ), }is a m-autoparallel Let be a particular e-autoparallel submanifold, and con- A ⊆ B MA ⊆ MB ⊆ P MA E submanifold of and is a m-autoparallel subman- sider a probability distribution p not necessarily in this B B X ifold of (X ). M M → submanifold. The problem of finding the point Π (pX ) in P E The proofs of Proposition1 and Proposition2 are based on closest in Kullback Leibler divergence to pX defined by E the properity of exponential family and conditional exponential → Π (pX ) , arg min D(pX qX ) (13) family, we omitted them due to space limit. These two E qX || ∈E Propositions have shown that Shannon facets are associated is well posed, and is characterized in the following two with affine subsets of the family of probability distribution, ways(here →Π with a right arrow means we are minimizing when it is regarded in an appropriate parameterization. In fact, E over the second arguments qX ). The tangent vector of the as we shall presently show, all Shannon Type information p → , and for , given by (18) for every ↔ ⊥ we Π ↔ ⊥ (pX ) qX , , ∈ EA B can reorganizeEA B the divergence as

→ D(pX qX ) = D(pX Π , (pX )) ↔, ⊥ || || EA B m-geodesic m-geodesic → , +D(Π ↔ ⊥ (pX ) qX X qX qX( )c ) , A\B A∩B B A∪B EA B || | The remainder of the theorem is proved by substituting (17) and (18) in the equation for the divergence.  e-geodesic C. Information Geometric Structure of Ingleton-Violating En- → Π , (p) ↔, ⊥ tropic Vectors & their Distributions EA B → Π ⊥ (p) EA∪B As identified in III-A, for a given random vector X = , § ↔, ⊥ [X1,...,XN ] and given cardinality X = 1 2 N , EA B X ×X X×· · ·×X e-autoparallel submanifold ⊥ the set of joint probability mass functions for is a K =

e-autoparallel submanifold EA∪B QN D n=1 n 1 dimensional manifold, with m-coordinates η. Now suppose|X | − a particular K M elements of η are all zero − entropy submodularity with M < K. This is a M dimensional face on the boundary of , and we will label a distribution in it as PMatom. Fig. 1. Submodularity of the entropy function is equivalent to the non- → → D negativity of a divergence D( π (p)|| π (p)) between two For X = [X4X3X2X1], it is easy to prove h(P2atom) 4 E E , ⊥ ↔, ⊥ ⊆ S A∪B → A B for any P and any cardinality X . Our numerical exper- information projections, one projection ( π (p)) is to a set that is 2atom E , ↔, ⊥ iments also suggest that h , however, this is → A B (P3atom) 4 a submanifold the other projection’s ( π (p)) set. A Pythagorean ⊆ S ij E⊥ not true for P4atom, that is to say, h(P4atom) G4 = A∪B → style relation shows that for such an arrangement D(p|| π (p)) = ∩ 6 ∅ E , for some P4atom. Here we focus on the the 4 atom sup- ↔, ⊥ → → → A B D(p|| π (p)) + D( π (p)|| π (p)). port (0000)(0110)(1010)(1111) that have distributions violate E E E , ⊥ ⊥ ↔, ⊥ A∪B A∪B A B Ingleton34. A distribution with this support has been used ¯ in [5] by Matu´sˇ to show that ΓN∗ is not a polyhedron for N 4, and it is also the 4 atom support utilized with another inequalities correspond to the positivity of divergences of distribution≥ in [16] associated with the 4 atom conjecture. Next projections to these submanifolds. The nested nature of these we will use Information Geometry to help understand the 4 submanifolds, and the associated Pythagorean relation, is one atom distributions that violate Ingleton’s inequalities. way to view the submodularity of entropy, as we shall now Denote the manifold of all probability mass explain with Figure1 and theorem1. 4atom functions forD four⊂ binary D random variables with the support Theorem 1: The submodularity (3) of the entropy function (0000)(0110)(1010)(1111). Parameterize these distributions can be viewed as a consequence of Pythagorean style with the parameters η1 = P(X = 0000) = α, η2 = information projection relationships depicted in Figure1. P(X = 0110) = β α, η3 = P(X = 1010) = γ α, − P3 − In particular, submodularity is equivalent to the inequality η4 = P(X = 1111) = 1 ηi = 1 + α β γ, yielding − i=1 − −   the m-coordinates of 4atom. The associated e-coordinates can → → D ηi D Π (pX ) Π , (pX ) 0 (16) be calculated as θi = log2 for i = 1, 2, 3. Now we consider ⊥ ↔, ⊥ η4 EA∪B || EA B ≥ the submanifold u1 = px 4atom I(X3; X4) = 0 , since by Proposition1, D is{ a e-autoparallel∈ D | submanifold of} Du1   4atom. In fact, an equivalent definition is u1 = px → → D D { ∈ D Π (pX ) Π , (pX ) = H(X ) + H(X ) 4atom θ1 +θ2 +θ3 = 0 . Numerical calculation illustrated ⊥ ↔, ⊥ EA∪B || EA B A B D |− } H(X ) H(X ) in the Figure2 lead to the following proposition, which we − A∩B − A∪B have verified numerically. Proof: The projections are Proposition 3: Let m1 = px 4atom Ingleton34 = 0 , → and (17) then is a e-autoparallelD { submanifold∈ D | of and} is Π ⊥ (pX ) = pX pX( )c , m1 4atom EA∪B A∪B A∪B D D parallel with u1 in e-coordinate, an equivalent definition → D , (18) of is = p θ + θ + θ = Π ↔ ⊥ (pX ) = pX X pX pX( )c m1 m1 x 4atom 1 2 3 E , A\B| A∩B B A∪B D0.5 α0 D2 { ∈ D | − A B log ( − ) , where α0 is the solution of α log α (1 2 α0 } − 2 − − since for → given by (17) for every α) log (1 α) = 1+2α in 0 < α < 1 . Π ⊥ (pX ) qX ⊥ 2 2 2 EA∪B ∈ EA∪B − we can reorganize the divergence as In fact, using this equivalent definition, we can also deter- → mine all the distributions in 4atom that violate Ingleton34 > D(pX qX ) = D(pX Π ⊥ (pX )) D || || EA∪B 0 as the submanifold V io = px 4atom θ1 + θ2 + θ3 < 0.5 α0 2 D { ∈ D | − → log ( − ) . Because we are dealing with 4atom, a 3 +D(Π ⊥ (pX ) qX( ) qX( )c ) 2 α0 A∪B A∪B } D EA∪B || dimensional manifold, we can use a plot to visualize our results in Fig.2. In Fig.2, besides and , we also Du1 Dm1 q1 D characterization, the mapping from 4atom to Γ4∗ is straight m1 D D forward: the curve hn1 = h( n1), the point hu = h(pu) and n1 D D the point hm = h(pm). Du1 IV. CONCLUSIONSAND FUTURE WORK Ds1 θ3 w1 In this paper, we aimed to suggest that information geometry D may provide a useful framework to parameterize this unknown part of the region of entropic vectors. Along these lines, it was first shown that Shannon facets possess certain information pu geometric properties and Shannon-type inequalities had a pm natural information geometric structure. Next, we character- θ1 θ2 ized a special type of 4 atom distributions via Information DVio Geometry, and showed there is a natural information geometric

Fig. 2. Manifold D4atom in θ coordinate explanation for it. The characterization also suggests it is possible to generate better inner bounds by growing in the f34 number of atoms. Future work includes study the distributions with arbitrarily large cardinality that set Ingleton to zero via hm V¯R Information Geometry and extend our work to more variables. hu ACKNOWLEDGEMENTS V¯ The authors gratefully acknowledge NSF support of this M r1∅ research through grants CCF-1016588 and CCF-1053702.

hm1 REFERENCES [1] Raymond W. Yeung, Information Theory and Network Coding. hn1 V¯ Springer, 2008. N [2] Zhen Zhang and Raymond W. Yeung, “On Characterization of Entropy Function via Information Inequalities,” IEEE Trans. on Information 34 Fig. 3. G4 : one of the six gaps between S4 and Γ4 Theory, vol. 44, no. 4, Jul. 1998. [3] K. Makarychev, Y. Makarychev, A. Romashchenko, and N. Vereshcha- gin, “A new class of non-Shannon-type inequalities for entropies,” Communication in Information and Systems, vol. 2, no. 2, pp. 147–166, plot the following submanifolds and points: December 2002. [4] R. Dougherty, C. Freiling, and K. Zeger, “Six new non-Shannon infor- q1 = px 4atom Ingleton34 = 0.1 mation inequalities,” in IEEE International Symposium on Information D { ∈ D | } = p Ingleton = 0.126 Theory (ISIT), July 2006, pp. 233–236. Ds1 { x ∈ D4atom | 34 − } [5] Frantisekˇ Matu´s,ˇ “Infinitely Many Information Inequalities,” in IEEE w1 = px 4atom Ingleton34 = 0.16 Int. Symp. Information Theory (ISIT), Jun. 2007, pp. 41–44. D { ∈ D | − } [6] Babak Hassibi, Sormeh Shadbakht, Matthew Thill, “On Optimal Design n1 = px 4atom β = γ = 0.5 of Network Codes,” in Information Theory and Applications, UCSD, D { ∈ D | } p = p α = 0.25, β = γ = 0.5 Feb. 2010, presentation. u { x ∈ D4atom | } [7] J. M. Walsh and S. Weber, “A Recursive Construction of the Set of Bi- pm = px 4atom α 0.33, β = γ = 0.5 nary Entropy Vectors,” in 2009 Allerton Conference on Communication, { ∈ D | ≈ } Control, and Computing. As we can see from Fig.2, m1 and u1 are e-autoparallel [8] ——, “A Recursive Construction of the Set of Binary Entropy Vectors D D and Related Inner Bounds for the Entropy Region,” IEEE Trans. Inf. and parallel to each other. As Ingleton34 goes from 0 to Theory, vol. 57, no. 10, Oct. 2011. negative values, the hyperplane becomes a non-symmetric [9] ——, “Relationships Among Bounds for the Region of Entropic Vectors 2010 Allerton Conference on Communication, ellipsoid, and as Ingleton34 becomes smaller and smaller, in Four Variables,” in Control, and Computing. the ellipsoid shrinks, finally shrinking to a single point pm [10] Congduan Li, J. M. Walsh, S. Weber, “A computational approach for at Ingleton34 0.1699, the point associated with the determining rate regions and codes using entropic vector bounds,” in four atom conjecture≈ − in [16]. Also for each e-autoparallel 2012 Allerton Conference on Communication, Control, and Computing. [11] Congduan Li, J. Apte, J. M. Walsh, S. Weber, “A new computational submanifold e V io that is parallel to m1 in e- D∀ ⊂ D D approach for determining rate regions and optimal codes for coded coordinate, the minimum Ingleton of e is achieved at point networks,” in IEEE International Symposium on Network Coding, 2013. D∀ e n1, where n1 is the e-geodesic in which marginal [12] D. Hammer, A. Romashschenko, A. Shen, N. Vereshchagin, “Inequal- D∀ ∩ D D ities for Shannon Entropy and Kolmogorov Complexity,” Journal of distribution of X3 and X4 are uniform, i.e. β = γ = 0.5. Computer and System Sciences, vol. 60, pp. 442–464, 2000. Now we will map some submanifolds of 4atom to the [13] A. W. Ingleton, “Representation of Matroids,” in Combinatorial Mathe- 34 D matics and its Applications, D. J. A. Welsh, Ed. San Diego: Academic entropic region. From Lemma1, G4 , one of the six gaps Press, 1971, pp. 149–167. between 4 and Γ4 is characterized by extreme rays V¯P = [14] F. Matu´sˇ and M. Studeny,´ “Conditional Independences among Four ¯ ¯ S ¯ 13 14 23 24 1 2 Random Variables I,” Combinatorics, Probability and Computing, no. 4, (VM , VR, r1∅, f34), where VM = (r1 , r1 , r1 , r1 , r2, r2) and ¯ 123 124 134 234 3 4 pp. 269–278, 1995. VR = (r1 , r1 , r1 , r1 , r1, r1, r3∅). In addition, we define [15] S. Amari and H. Nagaoka, Methods of Information Geometry. American ¯ 1 2 12 34 VN = (r1, r1, r1 ), and use Fig.3 to help visualize G4 . Mathematical Society Translations of Mathematical Monographs, 2004, In Fig.3, is one of the six Ingleton-violating extreme vol. 191. f34 [16] Randall Dougherty, Chris Freiling, Kenneth Zeger, “Non-Shannon ray of Γ , V¯ ,V¯ and r∅ are all extreme rays of that Information Inequalities in Four Random Variables,” Apr. 2011, 4 M R 1 S4 make Ingleton34 = 0. Based on the information geometric arXiv:1104.3602v1.