On Designing Probabilistic Supports to Map the Entropy Region

John MacLaren Walsh, Ph.D. & Alexander Erick Trofimoff Dept. of Electrical & Computer Engineering Drexel University Philadelphia, PA, USA 19104 [email protected] & [email protected]

Abstract—The boundary of the entropy region has been shown by time-sharing linear codes [2]. For N = 4 and N = 5 to determine fundamental inequalities and limits in key problems r.v.s, the region of entropic vectors (e.v.s) reachable by time- in network coding, streaming, distributed storage, and coded sharing linear codes has been fully determined ([5] and [15], caching. The unknown part of this boundary requires nonlinear constructions, which can, in turn, be parameterized by the [16], respectively), and is furthermore polyhedral, while the support of their underlying probability distributions. Recognizing full region of entropic vectors remains unknown for all N ≥ 4. that the terms in entropy are submodular enables the design of As such, for N ≥ 4, and especially for the case of N = 4 such supports to maximally push out towards this boundary. ¯∗ and N = 5 r.v.s, determining ΓN amounts to determining Index Terms—entropy region; Ingleton violation those e.v.s exclusively reachable with non-linear codes. Of I.INTRODUCTION the highest interest in such an endeavor is determining those From an abstract mathematical perspective, as each of the extreme rays generating points on the boundary of Γ¯∗ re- key quantities in information theory, entropy, mutual informa- N quiring such non-linear code based constructions. Once these tion, and conditional mutual information, can be expressed as extremal e.v.s from non-linear codes have been determined, linear combinations of subset entropies, every linear informa- all of the points in Γ¯∗ can be generated through time-sharing tion inequality, or inequality among different instances of these N their constructions and the extremal linear code constructions. quantities, specifies a half-space in which the entropy region For N = 4 r.v.s, determining whether a entropic vector Γ¯∗ [1], [2] must live, and vice-versa. This in turn implies that N can be achieved with the time-sharing of linear codes is characterizing the boundary of Γ¯∗ is equivalent to determining N equivalent to checking whether it obeys Ingleton’s inequality all fundamental inequalities in information theory[1], [3], [17], [5], and for N ≥ 5 violation of Ingleton’s inequality is a [2]. Other abstract equivalences have shown that determining sufficient condition for an entropic point to require nonlinear it amounts to determining all inequalities in the sizes of constructions. These sorts of conditions identify the e.v.s of in- N subgroups and their intersections[4], and all inequalities terest directly through conditions expressed in their entropies, among Kolmogorov complexities[5], and that the faces of but a method for clearly working these conditions back to Γ¯∗ encode information about implications among conditional N structure in the joint probability mass function is yet unknown. independences [6] which are key to the language of graphical Connections with inequalities for the sizes of subgroups of a models in machine learning. From a more applied perspective, common finite group [4] has provided one direction for con- it has been shown the determining the boundary of Γ¯∗ is N structions probably sufficient in the limit [18], but the groups equivalent to determining the capacity regions of all networks must grow arbitrary large in proofs for these constructions under network coding[7], [8], which in turn have been shown to suffice, and extremality of scores in various directions is to be key ingredients in building optimal coding protocols for difficult to couple with group properties other than being non- streaming information with low delay over multipath routed abelian. On the other hand, in practice numerical optimization networks[9], [10], limits for secret sharing systems[11], as well [19], [20] to map Γ¯∗ consistently exhibits limited support of as fundamental tradeoffs between the amount of information N a non-quasi-uniform nature, motivating a study of arbitrary a large distributed information storage system can store and supports for non-uniform distributions [21]. A particularly the amount of network traffic it must consume to repair failed simple way of encoding nonlinear dependence between r.v.s disks[12], [13], [14]. in a joint probability mass function is through its probabilistic The entropy region can be broadly broken into two parts, support, or the collection of N-dimensional outcomes for the the part that can be achieved by time-sharing linear codes, and N discrete r.v.s which have strictly positive probabilities. Main the part that cannot. Here, linear codes construct the discrete Contribution: This paper advances a nascent structural theory random variables (r.v.s) as vectors created by multiplying vec- identifying and organizing which probabilistic supports can tors of r.v.s uniformly distributed over a finite field by matrices yield e.v.s in the part of Γ¯∗ reachable exclusively with non- with elements drawn from the same field. For collections of N linear codes. Additionally, in order to push the constructions as exclusively N ≤ 3 r.v.s, all entropy vectors are achievable ¯∗ close to the boundary of ΓN as possible, structural theory and This material is based upon work support by the National Science Foun- methods for ranking supports relative to one another in terms dation under Grant No. 1812965 & 1421828. of Ingleton score and other measures of code nonlinearity are investigated. The new theorems provide guidance and where we have defined H(ξ; p), and, explanation for observations that were made exclusively with ( ) ⋀ numerical experiments in [21]. H(XA) = H ξn; p , (7) II.ENCODING SUPPORTS WITH SETSOF SET PARTITIONS n∈A A collection of N discrete random variables X := so we can formally express the entropic vector as a function (X1,...,XN ) taking values in the set X := X1 ×· · ·×XN , is [ ( ) ] often specified by defining the joint probability mass function ⋀ h(ξ; p) = H ξn; p |A ⊆ {1,...,N} , (8) (PMF) pX : X → [0, 1], which assigns to each vector n∈A x = (x1, . . . , xN ) ∈ X its probability pX (x) := P[X = x]. Perhaps the most familiar representation of a probabilistic and the set of entropy vectors reachable with a given equiva- support is the set of outcomes mapped to strictly positive lence class of probabilistic supports as probabilities under the PMF HN (X >0) = HN (ξ) := {π(h(ξ; Sk)) |π ∈ SN } , (9) { ⏐ } X >0 = {x ∈ X |pX (x) > 0} . (1) ⏐ ∑k where Sk = p ⏐p ≥ 0, i=1 pi = 1 . Two different sup- A k-atom probabilistic support, or k-atom support for short, ports X >0 can generate identical collections of N set par- is one with |X >0| = k. titions ξ, and thus via (9) exactly the same set of entropy The key object under study is the closure of the set of vectors. For this reason, we choose to represent a N-variable entropy vectors reachable with a given k-atom support k-atom probabilistic support not as a set of vectors of out- ⎧ ⏐ ∑ ⎫ pX (x) = 1, comes X >0 but instead as the collection of N set partitions ⎨ ⏐ x∈X >0 ⎬ o ⏐ ξ. Additionally, as we are adding all permutations of the N HN (X >0) = h(pX ) ⏐ pX (x) ≥ 0 ∀x ∈ X , , (2) ⏐ random variable labels back in (9), which also form the index ⎩ ⏐ pX (x) = 0 ∀x ∈/ X >0 ⎭ ordering ξ, we can take ξ to be a multi-set, as two different with the explicit goal of providing a structural characterization orderings of the indexing of this multiset will yield identical of those k-atom supports which can generate e.v.s in the un- H (ξ)s. We further restrict consideration to ξ being a set known part of Γ¯∗ requiring non-linear codes. In this respect, N N of set partitions, as repetition of the same partition is more because Γ¯∗ is invariant to the identification of the labels of the N properly thought of as an N ′ variable k-atom support with N-r.v.s, but identification of a particular k-atom support can N ′ < N together with a repetition of identical r.v.s. Finally, break this symmetry, we consider the set of entropy vectors since the entropies will be independent of the ordering of the to be reachable from X to be not only Ho (X ) but >0 N >0 labels {1, ..., k} of the elements in (1), from the orbit of the o HN (X >0) = {π(h) |h ∈ HN (ξ), π ∈ SN } (3) set of permutations of these labels Sk (the symmetric group of order k), we select the ξ which is minimum under the natural where π ∈ , a permutation in the symmetric group of order SN lexicographic ordering. N, acts on the entropy vector h by permuting the subsets of {1, ..., N} that index its dimensions. From the viewpoint of Definition 1 (Canonical Minimal k-atom support): A canon- ¯∗ the goal of mapping ΓN , those supports X >0 generating the ical minimal k-atom support is a set of N set partitions ξ same sets HN (X >0) are functionally equivalent. It is thus of of the set {1, . . . , k} whose meet is the singletons interest to group probabilistic supports into equivalence classes ⋀ based on those entropy vectors HN (X >0) that they can reach. ξn = {{i}|i ∈ {1, . . . , k}} (10) In this regard, an important observation is that a k-atom n∈{1,...,N} N support induces a collection of set partitions which directly and whom is minimal under lexicographic ordering in its determine the entropy vectors it can reach. Indeed, labeling orbit under the action of the group Sk. X >0 via X >0 = {x1,..., xk} with xi = [xi,1, . . . , xi,N ], we can associate each random variable Xn with a set partition ξn In addition to grouping together different supports that are whose blocks are the indices i of those outcomes xi ∈ X >0 equivalent for the purposes of mapping entropy, one of the with xi,n equal to some particular outcome from Xn, benefits or representing supports with set partitions isthe

ξn = {{i ∈ {1, . . . , k} |xi,n = xn } |xn ∈ Xn } . (4) ordering among entropies can be determined via refinement.

The meet of these N set partitions ξ1, . . . , ξN is the set of Lemma 1 (Entropy is ordered under refinement.): Let ξ1 and singletons, i.e. ξ2 be two set partitions of the set {1, . . . , k}, and suppose ⋀ the partition ξ1 refines the partition ξ2, denoted by ξ1 ⪯ ξ2. ξn = {{i} |i ∈ {1, . . . , k}} . (5) p = [p , . . . , p ]T p ≥ n∈{1,...,N} Then, for all probability vectors 1 k , i 0, i ∈ {1, . . . , k}, ∑k p = 1, Together with the vector of non-zero probabilities p = i=1 i

[p1, . . . , pk], pi = pX1,...,XN (xi), ξ determines H(ξ1; p) ≥ H(ξ2; p) (11) ( ) ( ) ∑ ∑ ∑ Proof: Let X1 be a random variable generating the set H(Xn) = H(ξn; p) := − pi log pi , (6) partition ξ1 and X2 be a random variable generating the set B∈ξn i∈B i∈B partition ξ2. Since ξ1 refines ξ2, their meet ξ1 ∧ ξ2 = ξ1. Proof: Define a ternary random variable X1 whose set par- c Thus, H(X1,X2) = H(ξ1 ∧ ξ2; p) = H(ξ1; p) = H(X1). tition representation is ξ1 = {A, B\A, (A ∪ B) } and Substituting H(X1,X2) = H(ξ1; p) and H(X2) = H(ξ2; p) a second ternary random variable X2 whose set partition c into the basic entropy inequality H(X1,X2) ≥ H(X2) then representation is ξ2 = {B, A\B, (A∪B) }. The join partition c shows (11). ■ is ξ1 ∨ ξ2 = {A ∪ B, (A ∪ B) } and the meet partition c An even more powerful type of inequality linking the is ξ1 ∧ ξ2 = {A ∩ B, A\B, B\A, (A ∪ B) }. Plugging entropies achievable with set partitions invokes entropy sub- these partitions into (12), the partition-based submodularity modularity via the meet and join operators as shown in the of entropy, and canceling repeated identical terms f((A ∪ following lemma. B)c), f(A\B), f(B\A) between positive and negative terms, one obtains the inequality (14). ■ Lemma 2 (Partition Based Entropy Submodularity): Let ξ1 This submodularity of these block components of entropy and ξ2 be two set partitions of the set {1, . . . , k}. For all provide a very powerful tool for determining whether the col- T probability vectors p = [p1, . . . , pk] , pi ≥ 0, i ∈ {1, . . . , k}, lection of entropy vectors reachable with a given probabilistic ∑k i=1 pi = 1, support live in a particular half-space. H(ξ ; p) + H(ξ ; p) ≥ H(ξ ∧ ξ ; p) + H(ξ ∨ ξ ; p) (12) 1 2 1 2 1 2 Theorem 1: The entropy vectors reached by the probabilistic support ξ will all live in the halfspace Proof: This can in fact be viewed as another instance of the { ⏐ T } T submodularity of entropy as a set function of a collection h ⏐c h ≥ 0 , so that c HN (ξ) ≥ 0, if for all π ∈ SN , of r.v.s via a clever identification of r.v.s. Indeed, let X1 ∑ dB,πH(Y B) ≥ 0 (15) be a random variable inducing the set partition ξ1, X2 be B⊆{1,...,k} a random variable inducing the set partition ξ2, and Z be a random variable inducing the set partition ξ1 ∨ ξ2. Be- is a balanced Shannon type information inequality in k- cause ξ1 ∧ (ξ1 ∨ ξ2) = ξ1, we can think of H(ξ1; p) as variables Y1,...,Yk, where H(X1,Z), and similarly because ξ2 ∧ (ξ1 ∨ ξ2) = ξ2 we ⎧ ∑ can think of H(ξ2; p) as H(X2,Z), substituting these and ⎨ cA B ∈ F(ξ) d := (16) H(X1,X2,Z) = H(ξ1 ∧ ξ2; p) into (12) we transform it B,π A∈F(ξ;B,π) ⎩ into H(X1,Z) + H(X2,Z) ≥ H(Z) + H(X1,X2,Z). This 0 otherwise shows that, once we have identified the random variable ⋃ ⋀ with F(ξ) := ξn and Z, this inequality can be recognized as a form of entropy A⊆{1,...,N} n∈A ⎧ ⏐ ⎫ submodularity. ■ ⏐ ⎨ ⏐ ⋀ ⎬ An interesting interpretation of this theorem is as stating F(ξ; B, π) := A ⊆ {1,...,N} ⏐B ∈ ξn . (17) ⏐ that the entropy of the common information between two ⎩ ⏐ n∈π(A) ⎭ r.v.s is upper bounded by their mutual information, since the join partition can be interpreted as the common information Proof: Grouping together terms by common blocks, we can computable individually from either the RVs with partitions rewrite for any entropy vector π(h(ξ; p)) in HN (ξ) ξ1 and ξ2 in a manner that is agreed upon. ⎛ ⎞ In fact, a key property of linear codes, which generate the ∑ ∑ ⋀ cAH(Xπ(A)) = cAH ⎝ ξn; p⎠ r.v.s X as vectors with elements drawn from some finite field n A⊆{1,...,N} A⊆{1,...,N} n∈π(A) GF (q), that resulting from multiplying a vector of independent ∑ ∑ r.v.s uniformly distributed over GF (q) by some matrix, is that = cA fp(B) A⊆{1,...,k} B∈⋀ ξ equality is obtained in the inequality (12). It was this fact that n∈π(A) n enabled Ingleton to prove Ingleton’s inequality, and Dougherty ∑ = dB,πfp(B) (18) Freiling and Zeger derived the region of e.v.s reachable with B∈F(ξ) linear codes for 5 r.v.s also by exploiting this fact. Not only does entropy expressed in a terms of set partitions Lemma 3 establishes that fp(·) is a submodular function for obey a submodular relationship, but also the terms within a every p ∈ Sk. A balanced Shannon-type inequality is one partition’s entropy are submodular, as shown next. which can be expressed as a sum of submodularity inequalities. As such, if (15) is a balanced Shannon-type inequality, (18) Lemma 3 (Submodularity of entropy terms): For any prob- will be ≥ 0 for all p ∈ Sk. ■ ability vector p ∈ Sk, the set function Checking whether or not an information expression is a ( ) ( ) balanced Shannon type information inequality, in turn, can be ∑ ∑ fp(A) := − pi log pi (13) completed by running a linear program. Thus, Thm. 1 enables i∈A i∈A us to determine and prove whether or not a given support gives only entropy vectors restricted to a certain halfspace, is submodular, so that for any A, B ⊆ {1, . . . , k} for instance obeying Ingleton’s inequality, by running a linear fp(A) + fp(B) − fp(A ∩ B) − fp(A ∪ B) ≥ 0. (14) program. Additionally, Thm. 1 also enables us to provide a firm mathematical proof of this fact after running the linear can be used to prove that this support is incapable of program, removing the possibility for numerical errors leading violating Ingleton’s inequality, and is thus restricted to the ¯∗ to incorrect conclusions. part of ΓN reachable with linear codes. Careful evaluation ¯∗ For the purposes of mapping ΓN , Thm. 1’s primary use is as shows a tool to weed out supports that can not be helpful in growing the inner bound, for instance initially for 4-variables, by Ingleton12(ξ) = fp({2}) + fp({3}) − fp({2, 3}) (23) determining that they can never violate Ingleton’s inequality. Ingleton13(ξ) = fp({3}) + fp({4}) − fp({3, 4}) (24) Any support that survives this weeding out process will have Ingleton14(ξ) = fp({2, 3}) + fp({3, 4}) − fp({3}) π o some π ∈ N such that HN (ξ) := π(HN (ξ)) can not be S −fp({2, 3, 4}) (25) proven, via fp(·) submodularity alone, to live in the half space {h|cT h ≥ 0} associated with one of the inner bounding Ingleton23(ξ) = 0 (26) inequalities. We say that the pair (ξ, π) then form a c-violation Ingleton24(ξ) = fp({2, 3}) + fp({4}) − fp({2, 3, 4})(27) candidate. Attention then turns to comparing the amount that Ingleton34(ξ) = fp({2}) + fp({3, 4}) − fp({2, 3, 4})(28) these violation candidates push out in the direction c, and submodularity proves to be useful in this regard as well. The Each of these are a form of simple submodular inequality, following definition makes the notion of “pushing further out or equivalently, a balanced Shannon-type inequality, and in the direction c” precise as the definition of c-domination. thus are ≥ 0 no matter what p ∈ Sk is selected. Using exact linear programs (e.g. [22]) to provide the Definition 2 (Domination of Violation Candidates): The c- submodular constructions, and checking every equivalence violation candidate (ξ, π) dominates the c-violation candi- class of k-atom supports (see [21] for how to apply [23], [24] date (ξ′, π′) if to construct these) enables one to obtain Table I col. 0 - 6. cT h cT h′ 4 75 31 73 29 2 2 1 sup − ≥ sup − ′ (19) 5 2665 349 2558 313 107 36 2 h∈Hπ (ξ) h{1,..,N} ′ π′ ′ h N h ∈HN (ξ ) {1,..,N} 6 105726 6442 99769 5627 5957 815 14 7 5107735 160365 4763013 136776 344722 23589 53 Submodularity is also a powerful tool for proving c- TABLE I IN TERMS OF THE NUMBER OF ATOMS IN THE SUPPORT (COL. 0): THE domination as pointed out in the following theorem. NUMBER (HENCEFORTH #) OFNON-ISOMORPHIC SUPPORTS (COL. 1), AND # OFNEWEQUIVALENCECLASSES (FORMS HENCEFORTH) OF Theorem 2: The c-violation candidate (ξ, π) dominates INGLETONEXPRESSIONS (EQUATION (18)), NOTREACHABLEWITH k′ < k ′ ′ ATOMS, THAT k-ATOMS SUPPORTS MAP TO (COL. 2), # OF SUPPORTS the c-violation candidate (ξ , π ) if for some σ ∈ k S THAT SUBMODULARITY PROVES CAN NEVER VIOLATE ANY INGLETON’S ∑ INEQUALITY (COL. 3), # OFFORMSSUBMODULARITYSHOWSARENEVER qB,σH(Y B) ≥ 0 (20) NEGATIVE (COL. 4), REMAINING # OF SUPPORTS THAT CAN POSSIBLY B⊆{1,...,k} VIOLATE INGLETON’SINEQUALITY (COL. 5), # OF POSSIBLY NEGATIVE FORMS (COL. 6), AND # OF INGLETON-DOMINANTFORMS (COL. 7). is a balanced Shannon-type information inequality in k- IV. MAXIMIZING INGLETON VIOLATION variables, where With those supports that can violate Ingleton (cf. Table I), ′ and thus live in the non-linear part of Γ¯∗ , in-hand, one shifts qB,σ = dB,π′,σ − dB,π (21) N attention to comparing how far their scores can push out in with dB,π defined according to (16) applied to ξ and various directions. Thm. 2 was built for this, as the following

⎧ ∑ ′ two examples, and Table I col. 7 demonstrate. cA σ(B) ∈ F(ξ ) ′ ⎨ dB,π′,σ := A∈F(ξ′;σ(B),π′) (22) Example 2: Thm. 2 can be used to prove that ⎩ 0 otherwise ξ = {ξ1, ξ2, ξ3, ξ4} with ξ1 = {{1}, {2, 3, 4}}, ξ2 = {{1, 2}, {3, 4}}, ξ3 = {{1, 2, 3}, {4}}, ξ4 = {{1, 3}, {2, 4}} T T ′ ′ ′ ′ ′ Proof: Follows from evaluating c h(ξ; p) − c π(h(ξ ; p )) Ingleton-dominates ξ with ξ1 = {{1}, {2, 3, 4}}, ξ2 = ′ ′ ′ and identifying p = σ(p), yielding identical joint entropies {{1, 2, 3}, {4}}, ξ3 = {{1, 2, 4}, {3}}, ξ4 = {{1, 3, 4}, {2}}. ′ h{1,...,N}. ■ For all i, j Ingletonij(ξ ) is some permutation of

III.THEORETICAL CONDITIONSFOR INGLETON −fp({1}) − fp({1, 2, 3}) − fp({1, 2, 4}) − fp({2}) VIOLATING SUPPORTS −f ({3}) − f ({4}) + f ({1, 2}) + f ({1, 3}) + f ({1, 4}) One of the most important applications of Thm. 1 is to rule p p p p p out those supports which always obey Ingleton’s inequality, +fp({2, 3}) + fp({2, 4}) Meanwhile ξ can be proven to obey all forms of Ingletons Ingleton = hij + hik + hiℓ + hjk + hjℓ − hi − hj − hkℓ − ij inequality except Ingleton (ξ) which takes the form, up to hijk − hijℓ. The following example shows how. 13 atom permutation, Example 1 (Support Incapable of Violating Ingleton): −fp({1}) − fp({1, 2, 3}) − fp({1, 2, 4}) − fp({2}) Consider the support ξ = {ξ1, ξ2, ξ3, ξ4} with ξ1 = −f ({3, 4}) + f ({1, 2}) + f ({1, 3}) + f ({1, 4}) {{1}, {2}, {3}, {4}}, ξ2 = {{1}, {2}, {3, 4}}, ξ3 = p p p p {{1}, {2, 3}, {4}}, and ξ4 = {{1}, {2, 3, 4}}. Theorem 1 +fp({2, 3}) + fp({2, 4}) Selecting an appropriate σ we see the difference between REFERENCES ′ the Ingleton value of ξ and that of ξ is of the form [1] Raymond W. Yeung, “A Framework for Linear Information Inequali- fp({3}) + fp({4}) − fp({3, 4}) which is positive due to ties,” IEEE Trans. on Information Theory, vol. 43, no. 6, Nov. 1997. [2] ——, Information Theory and Network Coding. Springer, 2008. the submodularity of fp(·). Since the Ingleton values of ′ [3] Zhen Zhang and Raymond W. Yeung, “On Characterization of Entropy ξ are ≤ the Ingleton values of ξ while they will both have, Function via Information Inequalities,” IEEE Trans. on Information owing to the joint entropy being associated with the meet Theory, vol. 44, no. 4, Jul. 1998. [4] T. Chan and R. Yeung, “On a relation between information inequalities partition, identical joint entropies h1234, the Ingleton score and group theory,” IEEE Trans. on Information Theory, vol. 48, no. 7, reached by ξ is provably lower than the Ingleton score pp. 1992 – 1995, Jul. 2002. ′ reached by ξ . [5] D. Hammer, A. Romashschenko, A. Shen, N. Vereshchagin, “Inequal- ities for Shannon Entropy and Kolmogorov Complexity,” Journal of In fact Thm. 2 can even be used, with appropriate selection, Computer and System Sciences, vol. 60, pp. 442–464, 2000. [6] F. Matu´s,ˇ “Conditional Independences among Four Random Variables to prove Ingleton-dominance of a k-atom support over a k’- III: Final Conclusion,” Combinatorics, Probability and Computing, atom support for k′ > k, as shown in the following example. vol. 8, no. 3, pp. 269–276, May 1999. [7] Xijin Yan, Raymond W. Yeung, and Zhen Zhang, “An Implicit Charac- Example 3: Thm. 2 implies that the support ξ identi- terization of the Achievable Rate Region for Acyclic Multisource Mul- tisink Network Coding,” IEEE Trans. on Information Theory, vol. 58, fied in Example 2 also Ingleton-dominates over the 5- no. 9, pp. 5625–5639, Sep. 2012. ′′ ′′ ′′ [8] T. Chan and A. Grant, “Dualities between entropy functions and network atom support ξ with ξ1 = {{1}, {2, 3}, {4, 5}}, ξ2 = {{1, 2, 3, 4}, {5}}, ξ′′ = {{1, 2, 4, 5}, {3}}, and ξ′′ = codes,” IEEE Transactions on Information Theory, vol. 54, no. 10, pp. 3 4 4470–4487, October 2008. {{1, 3, 5}, {2, 4}}. To see this, first split the atom 2 by [9] J. M. Walsh, S. P. Weber, J. C. de Oliveira, A. Eryilmaz, M. Medard,´ replacing it 2 ↦→ {2, 5} in ξ to get the non-minimal ξo with “Trading Rate for Delay at the Application and Transport Layers (Guest Ingleton value Editorial),” IEEE J. Sel. A. Comm., vol. 29, no. 5, pp. 913–915, May 2011. −f ({1}) − f ({1, 2, 3, 5}) − f ({1, 3, 4}) − f ({2, 5}) [10] J. M. Walsh, S. Weber, and C. wa Maina, “Optimal Rate Delay Tradeoffs p p p p and Delay Mitigating Codes for Multipath Routed and Network Coded −fp({3}) − fp({4}) + fp({1, 2, 5}) + fp({1, 3}) Networks,” IEEE Trans. Inf. Theory, vol. 55, no. 12, pp. 5491–5510, Dec. 2009. +fp({1, 4}) + fp({2, 3, 5}) + fp({3, 4}) [11] L. Csirmaz, “The size of share must be large,” J. Cryptology, vol. 10, ′′ no. 4, pp. 223–231, 1997. Next, evaluating Ingleton23(ξ ) we have [12] A. Dimakis, P. Godfrey, Y. Wu, M. Wainwright, and K. Ramchandran, “Network coding for distributed storage systems,” IEEE Transactions −fp({1}) − fp({1, 2, 3, 4}) − fp({1, 2, 3, 5}) − fp({2}) on Information Theory, vol. 56, no. 9, pp. 4539–4551, Sept 2010. −fp({3}) − fp({4}) − fp({5}) + fp({1, 2}) + fp({1, 2, 3}) [13] C. Tian, “Characterizing the rate region of the (4,3,3) exact-repair re- generating codes,” IEEE Journal on Selected Areas in Communications, +fp({1, 4}) + fp({2, 4}) + fp({3, 4}) + fp({3, 5}) vol. 32, no. 5, pp. 967–975, May 2014. ′′ o [14] Fangwei Ye, Kenneth W. Shum, and Raymond W. Yeung, “The rate So, we see that Ingleton23(ξ ) − Ingleton(ξ ) is of the form region for secure distributed storage systems,” IEEE Trans. Inform. Theory, vol. 63, no. 11, pp. 7038–7051, Nov. 2017. −fp({1, 2, 3, 4}) − fp({1, 2, 5}) − fp({1, 3}) − fp({2}) [15] Randall Dougherty, Chris Freiling, Kenneth Zeger, “Linear rank inequal- −f ({2, 3, 5}) − f ({5}) + f ({1, 2}) + f ({1, 2, 3}) ities on five or more variables,” submitted to SIAM J. Discrete Math. p p p p arXiv:0910.0284. +fp({1, 3, 4}) + fp({3, 5}) + 2fp({2, 5}) (29) [16] Ryan Kinser, “New Inequalities for Subspace Arrangements,” J. of Comb. Theory Ser. A, vol. 188, no. 1, pp. 152–161, Jan. 2011. which can be expressed as the sum of the fol- [17] A. W. Ingleton, “Representation of Matroids,” in Combinatorial Mathe- matics and its Applications, D. J. A. Welsh, Ed. San Diego: Academic lowing three submodularity inequalities fp({1, 2}) + Press, 1971, pp. 149–167. fp({2, 5}) − fp({1, 2, 5}) − fp({2}) ≥ 0, fp({2, 5}) + [18] Wei Mao, Matthew Thill, and Babak Hassibi, “On group network fp({3, 5}) − fp({2, 3, 5}) − fp({5}) ≥ 0, and fp({1, 2, 3}) + codes: Ingleton-bound violations and independent sources,” in IEEE International Symposium on Information Theory (ISIT), Jun. 2010. fp({1, 3, 4}) − fp({1, 2, 3, 4}) − fp({1, 3}) ≥ 0. Thus, ′′ o [19] F. Matu´sˇ and L. Csirmaz, “Entropy region and convolution,” Oct. 2013, Ingleton23(ξ ) − Ingleton(ξ ) ≥ 0 for all values of p. arXiv:1310.5957v1. o Next, we observe that Ingleton(ξ ) = Ingleton(ξ) and [20] Randall Dougherty, Chris Freiling, Kenneth Zeger, “Non-Shannon o Information Inequalities in Four Random Variables,” Apr. 2011, h1234(ξ) = h1234(ξ ), while the refinement inequality (11) o ′′ arXiv:1104.3602v1. shows that h1234(ξ ) ≤ h1234(ξ ). Thus, we see that the [21] Y. Liu and J. M. Walsh, “Non-Isomorphic Distribution Supports for Ingleton score reachable with ξ provably outpowers those Calculating Entropic Vectors,” in 53rd Annual Allerton Conference reachable with ξ′′. on Communication, Control, and Computing, Oct. 2015. [Online]. Available: http://www.ece.drexel.edu/walsh/Liu Allerton 2015.pdf ¯∗ [22] David Applegate, William Cook, Sanjeeb Dash, Daniel Espinoza, “Ex- In order to map ΓN , extremizing other cost functions while act Solutions to Linear Programming Problems,” Operations Research violating Ingleton is also required. Observe that Thm. 2 is Letters, vol. 35, no. 6, pp. 693–699, Nov. 2007. equally suited to this purpose, and is equally amenable to [23] A. Betten, M. Braun, H. Fripertinger, A. Kerber, A. Kohnert, and A. Wassermann, Error-Correcting Linear Codes: Classification by Isom- proofs constructed via linear programs. etry and Applications. Springer, 2006. V. CONCLUSIONS [24] Bernd Schmalz, “t-Designs zu vorgegebener Automorphismengruppe,” By recognizing submodularity of the components of entropy Bayreuther Mathematische Schriften, no. 41, pp. 1–164, 1992, Disser- when restricted to a certain probabilistic support, two theorems tation, Universitat¨ Bayreuth, Bayreuth. were constructed that help determine which supports are best for mapping the boundary of the entropy region.