Tropical Geometry, OMT and Neural Network

Na Lei1

1DUT-RU International School of Information Sciences and Engineering Dalian University of Technology

Geometric Analysis Approach to AI CMSA, Harvard 2019-01-22

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 1 / 62 Thanks

Thanks for the invitation.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 2 / 62 Collaborators

This is a joint work with Professor David Xianfeng Gu, Zhongxuan Luo, Lianbao Jin, Hao wang and so on.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 3 / 62 Outline

1 Introduction to Tropical Geometry 2 Tropical Geometry for Neural Network 3 Optimal Transport Mapping 4 OMT by Neural Network

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 4 / 62 Introduction to Tropical Geometry

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 5 / 62 Tropical Algebra

Tropical geometry is based on tropical semiring. Tropical semiring

1 Tropical Addition: ; ⊕ 2 Tropical Multiplication: ;

3 Tropical semiring: T := (R ∞ , , ). ∪ {− } ⊕ Tropical semiring is also known as the max-plus algebra.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 6 / 62 Tropical Algebra

Tropical operators For x,y R, ∈ 1 their tropical sum is x y := max x,y ; ⊕ { } 2 their tropical product is x y := x + y;

3 the tropical quotient of x over y is x y := x y. −

∞ x = x = 0 x, ∞ x = ∞. − ⊕ − − ∞: the tropical additive identity, 0: the tropical multiplicative identity. − Associativity, Commutativity, Distributivity. Lack of additive inverse.

T := (R ∞ , , ) is a semiring. ∪ {− } ⊕

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 7 / 62 Tropical Algebra

Tropical operators For x,y R, ∈ 1 their tropical sum is x y := max x,y ; ⊕ { } 2 their tropical product is x y := x + y;

3 the tropical quotient of x over y is x y := x y. −

∞ x = x = 0 x, ∞ x = ∞. − ⊕ − − ∞: the tropical additive identity, 0: the tropical multiplicative identity. − Associativity, Commutativity, Distributivity. Lack of additive inverse.

T := (R ∞ , , ) is a semiring. ∪ {− } ⊕

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 7 / 62 Tropical Algebra

Tropical operators For x,y R, ∈ 1 their tropical sum is x y := max x,y ; ⊕ { } 2 their tropical product is x y := x + y;

3 the tropical quotient of x over y is x y := x y. −

∞ x = x = 0 x, ∞ x = ∞. − ⊕ − − ∞: the tropical additive identity, 0: the tropical multiplicative identity. − Associativity, Commutativity, Distributivity. Lack of additive inverse.

T := (R ∞ , , ) is a semiring. ∪ {− } ⊕

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 7 / 62 Tropical Algebra

Tropical operators For x,y R, ∈ 1 their tropical sum is x y := max x,y ; ⊕ { } 2 their tropical product is x y := x + y;

3 the tropical quotient of x over y is x y := x y. −

∞ x = x = 0 x, ∞ x = ∞. − ⊕ − − ∞: the tropical additive identity, 0: the tropical multiplicative identity. − Associativity, Commutativity, Distributivity. Lack of additive inverse.

T := (R ∞ , , ) is a semiring. ∪ {− } ⊕

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 7 / 62 Tropical Algebra

Tropical power Let N = n Z : n 0 . For any integer a N, raising x R to the a th power: { ∈ ≥ } ∈ ∈ − a x := x x = a x ··· · where the last denotes standard product of real numbers. · For any a N, ∈ ( a ∞ if a > 0, ∞ := − − 0 if a = 0.

Every x R has a tropical multiplicative inverse given by its standard additive∈ inverse: 1 x − := x. − Thus T is in fact a semifield.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 8 / 62 Tropical Algebra

Tropical power Let N = n Z : n 0 . For any integer a N, raising x R to the a th power: { ∈ ≥ } ∈ ∈ − a x := x x = a x ··· · where the last denotes standard product of real numbers. · For any a N, ∈ ( a ∞ if a > 0, ∞ := − − 0 if a = 0.

Every x R has a tropical multiplicative inverse given by its standard additive∈ inverse: 1 x − := x. − Thus T is in fact a semifield.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 8 / 62 Tropical Algebra

One may therefore also raise x R to a negative power a Z by raising its tropical multiplicative∈ inverse x to the positive power∈ a, i.e., x a = ( x) a. − − − − ∞ does not have a tropical multiplicative inverse and ( ∞) a is − − undefined for a < 0.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 9 / 62 Tropical Algebra

One may therefore also raise x R to a negative power a Z by raising its tropical multiplicative∈ inverse x to the positive power∈ a, i.e., x a = ( x) a. − − − − ∞ does not have a tropical multiplicative inverse and ( ∞) a is − − undefined for a < 0.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 9 / 62 Tropical Algebra

Tropical monomial

A tropical monomial in d variables x1,...,xd is an expression of the form α a a ad cx = c x 1 x 2 x 1 2 ··· d d where c R ∞ , a1,...,ad N, α = (a1,...,ad ) N and ∈ ∪ {− } ∈ ∈ x = (x1,...,xd ).

Note that xα = 0 xα as 0 is the tropical multiplicative identity.

A tropical monomial is indeed a linear function, or equivalently, a hyperplane: a x + a x + + a x + c. 1 1 2 2 ··· d d

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 10 / 62 Tropical Algebra

Tropical monomial

A tropical monomial in d variables x1,...,xd is an expression of the form α a a ad cx = c x 1 x 2 x 1 2 ··· d d where c R ∞ , a1,...,ad N, α = (a1,...,ad ) N and ∈ ∪ {− } ∈ ∈ x = (x1,...,xd ).

Note that xα = 0 xα as 0 is the tropical multiplicative identity.

A tropical monomial is indeed a linear function, or equivalently, a hyperplane: a x + a x + + a x + c. 1 1 2 2 ··· d d

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 10 / 62 Tropical Algebra

Tropical polynomial

A tropical polynomial f (x) = f (x1,...,xd ) is a finite tropical sum of tropical monomials

α1 αr f (x) = c x cr x , 1 ⊕ ··· ⊕ d where ci R ∞ and αi = (ai1, ,aid ) N for i = 1, ,r. ∈ ∪ {− } ··· ∈ ··· A monomial of a given multiindex appears at most once in the sum, i.e., α = α for any i = j. i 6 j 6 A tropical polynomial is indeed a piecewise linear function, or equivalently, an upper envelope of the r hyperplanes: maxr a x + a x + + a x + c . i=1{ i1 1 i2 2 ··· id d i }

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 11 / 62 Tropical Algebra

Tropical polynomial

A tropical polynomial f (x) = f (x1,...,xd ) is a finite tropical sum of tropical monomials

α1 αr f (x) = c x cr x , 1 ⊕ ··· ⊕ d where ci R ∞ and αi = (ai1, ,aid ) N for i = 1, ,r. ∈ ∪ {− } ··· ∈ ··· A monomial of a given multiindex appears at most once in the sum, i.e., α = α for any i = j. i 6 j 6 A tropical polynomial is indeed a piecewise linear function, or equivalently, an upper envelope of the r hyperplanes: maxr a x + a x + + a x + c . i=1{ i1 1 i2 2 ··· id d i }

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 11 / 62 Tropical Algebra

Tropical rational function A tropical rational function is a standard difference, or, equivalently, a tropical quotient of two tropical polynomials f (x) and g(x):

f (x) g(x) = f (x) g(x). −

The set of tropical polynomials T[x1,...,xd ] forms a semiring under the standard extension of and to tropical polynomials, and likewise the ⊕ set of tropical rational functions T(x1,...,xd ) forms a semifield. We regard a tropical polynomial f = f 0 as a special cases of a tropical rational function and thus

T[x1,...,xd ] T(x1,...,xd ). ⊆

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 12 / 62 Tropical Algebra

Tropical rational function A tropical rational function is a standard difference, or, equivalently, a tropical quotient of two tropical polynomials f (x) and g(x):

f (x) g(x) = f (x) g(x). −

The set of tropical polynomials T[x1,...,xd ] forms a semiring under the standard extension of and to tropical polynomials, and likewise the ⊕ set of tropical rational functions T(x1,...,xd ) forms a semifield. We regard a tropical polynomial f = f 0 as a special cases of a tropical rational function and thus

T[x1,...,xd ] T(x1,...,xd ). ⊆

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 12 / 62 Tropical Algebra

Tropical rational function A tropical rational function is a standard difference, or, equivalently, a tropical quotient of two tropical polynomials f (x) and g(x):

f (x) g(x) = f (x) g(x). −

The set of tropical polynomials T[x1,...,xd ] forms a semiring under the standard extension of and to tropical polynomials, and likewise the ⊕ set of tropical rational functions T(x1,...,xd ) forms a semifield. We regard a tropical polynomial f = f 0 as a special cases of a tropical rational function and thus

T[x1,...,xd ] T(x1,...,xd ). ⊆

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 12 / 62 Tropical Geometry

Tropical hypersurface α α The tropical hypersurface of f (x) = c x 1 cr x r is 1 ⊕ ··· ⊕

d αi αj T (f ) := x R : ci x = cj x = f (x) for some αi = αj . { ∈ 6 } i.e., the set of points x at which the value of f at x is attained by two or more monomials in f .

The tropical curve of

1 x2 1 y2 2 xy 2 x 2 y 2. ⊕ ⊕ ⊕ ⊕ ⊕ i.e. max 1+2x,1+2y,2+x +y,2+x,2+y,2 . { }

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 13 / 62 Tropical Geometry

Tropical hypersurface α α The tropical hypersurface of f (x) = c x 1 cr x r is 1 ⊕ ··· ⊕

d αi αj T (f ) := x R : ci x = cj x = f (x) for some αi = αj . { ∈ 6 } i.e., the set of points x at which the value of f at x is attained by two or more monomials in f .

The tropical curve of

1 x2 1 y2 2 xy 2 x 2 y 2. ⊕ ⊕ ⊕ ⊕ ⊕ i.e. max 1+2x,1+2y,2+x +y,2+x,2+y,2 . { } The tropical hypersurfaces indeed induce a cell decomposition of Rd .

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 14 / 62 Tropical Geometry

Newton Polygon

α αr The Newton polygon of a tropical polynomial f (x) = c1x 1 cr x d ⊕ ···d ⊕ is the convex hull of α1,...,αd N , regarded as points in R , is ∈ d (f ) := Conv αi R : ci = ∞,i = 1, ,r . 4 { ∈ 6 − ··· }

The Newton polygon of

1 x2 1 y2 2 xy 2 x 2 y 2. ⊕ ⊕ ⊕ ⊕ ⊕ i.e. Conv (2,0),(0,2),(1,1),(1,0),(0,1),(0,0) . { }

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 15 / 62 Tropical Geometry

Dual subdivision d d+1 Lift each αi from R into R by appending ci as the last coordinate. Denote the convex hull of the lifted α1,...,αr as

d P(f ) := Conv (αi ,ci ) R R : i = 1, ,r . { ∈ × ··· } UF(P(f )) denote the collection of upper faces in P(f ) and π : Rd R Rd be the projection that drops the last coordinate. The dual subdivision× → determined by f is then

d δ(f ) := π(p) R : p UF(P(f )) { ⊆ ∈ } δ(f ) forms a polyhedral complex with support (f ). 4

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 16 / 62 Tropical Geometry

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 17 / 62 Tropical Geometry

Dual subdivision The tropical hypersurface T (f ) is the (d 1)-skeleton of the − polyhedral complex dual to δ(f ). This means that each vertex in δ(f ) corresponds to one “cell” in Rd where the function f is linear. Thus, the number of vertices in P(f ) provides an upper bound on the number of linear regions of f .

The Newton Polygon and the tropical curve of

1 x2 1 x2 2 x x 2 x 2 x 2. 1 ⊕ 2 ⊕ 1 2 ⊕ 1 ⊕ 2 ⊕

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 18 / 62 Tropical Geometry

Linear region A linear region of F Rat(d,m) is a maximal connected subset of the domain on which F is∈ linear. The number of linear regions of F is denoted N (F).

A tropical polynomial map F Pol(d,m) has convex linear regions but ∈ a tropical rational map F Rat(d,m) generally has nonconvex linear regions. ∈

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 19 / 62 Neural Networks

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 20 / 62 Neural Networks

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 21 / 62 Neural Networks

Feedforward Neural Network An L-layer feedforward neural network is a map ν : Rd Rp given by a composition of functions →

(L) (L) (L 1) (L 1) (1) (1) ν = σ ρ σ − ρ − σ ρ . ◦ ◦ ◦ ◦ ··· ◦ ◦ The preactivation functions ρ(1), ,ρ(L) are affine transformations to ··· be determined and the activation functions σ (1), ,σ (L) are chosen and fixed in advanced. ···

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 22 / 62 Neural Networks

Denote by n the number of nodes of the l-th layer, l = 1, ,L 1. l ··· − n0 := d, nL := p. The output from the l-th layer be denoted by

(l) (l) (l) (l 1) (l 1) (1) (1) ν := σ ρ σ − ρ − σ ρ . ◦ ◦ ◦ ◦ ··· ◦ ◦

(l) n 1 n The affine function ρ : R l − R l is given by a weight matrix ( ) → ( ) ( ) (l) l nl nl 1 (l) l l T nl A = (a ) Z × − and a bias vector b = (b , ,bn ) R : ij ∈ 1 ··· l ∈ (l) (l 1) (l) (l 1) (l) ρ (ν − ) := A ν − + b .

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 23 / 62 Neural Networks

Denote by n the number of nodes of the l-th layer, l = 1, ,L 1. l ··· − n0 := d, nL := p. The output from the l-th layer be denoted by

(l) (l) (l) (l 1) (l 1) (1) (1) ν := σ ρ σ − ρ − σ ρ . ◦ ◦ ◦ ◦ ··· ◦ ◦

(l) n 1 n The affine function ρ : R l − R l is given by a weight matrix ( ) → ( ) ( ) (l) l nl nl 1 (l) l l T nl A = (a ) Z × − and a bias vector b = (b , ,bn ) R : ij ∈ 1 ··· l ∈ (l) (l 1) (l) (l 1) (l) ρ (ν − ) := A ν − + b .

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 23 / 62 Neural Networks

Three Assumptions (a) The weight matices A(1), ,A(L) are integer-valued; ··· (b) The bias vectors b(1), ,b(L) are real-valued; ··· (c) The activation functions σ (l), ,σ (L) take the form ··· σ (l)(x) := max x,t(l) { } ( ) where t l (R ∞ )nl is called a threshold vector. ∈ ∪ {− } real weights can be approximated arbitrarily closely by rational weights; one may then ‘clear denominators’ in these rational weights by multiplying them by the least common multiple of their denominators to obtain integer weights; scaling all weights and biases by the same positive constant has no bearing on the workings of a neural network.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 24 / 62 Neural Networks

Three Assumptions (a) The weight matices A(1), ,A(L) are integer-valued; ··· (b) The bias vectors b(1), ,b(L) are real-valued; ··· (c) The activation functions σ (l), ,σ (L) take the form ··· σ (l)(x) := max x,t(l) { } ( ) where t l (R ∞ )nl is called a threshold vector. ∈ ∪ {− } real weights can be approximated arbitrarily closely by rational weights; one may then ‘clear denominators’ in these rational weights by multiplying them by the least common multiple of their denominators to obtain integer weights; scaling all weights and biases by the same positive constant has no bearing on the workings of a neural network.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 24 / 62 Tropical Algebra of Neural Networks

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 25 / 62 Tropical Algebra of Neural Networks

Consider the output from one layer in the neural network

ν(x) = max Ax + b,t , { } Decompose A as a difference of two nonnegative matrices A = A+ A with entries − − + a := max a ,0 , a− = max a ,0 . ij { ij } ij {− ij } Since max Ax + b,t = max A+x + b,A x + t A x, { } { − } − − every coordinate of one-layer neural network is a difference of two tropical polynomials.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 26 / 62 Tropical Algebra of Neural Networks

Proposition If the nodes of the l-th layer are given by tropical rational functions,

ν(l)(x) = F (l)(x) G(l)(x) = F (l)(x) G(l)(x), − then the outputs of the preactivation and of the (l + 1)-th layer are given by tropical rational functions

ρ(l+1) ν(l)(x) = H(l+1)(x) G(l+1)(x), ◦ − ν(l+1)(x) = σ ρ(l+1) ν(l)(x) = F (l+1)(x) G(l+1)(x) ◦ ◦ − respectively, where

F (l+1)(x) = max H(l+1)(x),G(l+1)(x) + t , (l+1) {(l) (l) } G (x) = A+G (x) + A F (x), (l+1) (l) − (l) H (x) = A+F (x) + A G (x) + b. −

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 27 / 62 Tropical Algebra of Neural Networks

Tropical characterization of neural networks A feedforward neural network under assumptions (a)-(c) is a function ν : Rd Rp whose coordinates are tropical rational functions of the input, i.e.,→ ν(x) = F(x) G(x) = F(x) G(x), − where F and G are tropical polynomial maps. Thus ν is a tropical rational map.

Corollary (1) (L 1) (L) d Setting t = = t − = 0 and t = ∞. Let ν : R R be an ReLU activated··· feedforward neural network− with integer→ weights and linear output. Then ν is a tropical rational function.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 28 / 62 Tropical Algebra of Neural Networks

Equivalence of neural networks and tropical rational functions (i) Let ν : Rd R. Then ν is a tropical rational function if and only if ν is a feedforward→ neural network satisfying assumptions (a)-(c). (ii) A tropical rational function f g can be represented as an L-layer neural network, with

L max log r , log rg + 2, ≤ {d 2 f e d 2 e}

where rf and rg are the number of monomials in the tropical polynomials f and g respectively.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 29 / 62 Tropical Algebra of Neural Networks

Continuous piecewise linear function Let ν : Rd R. Then ν is a continuous piecewise linear function with → integer coefficients if and only if ν is a tropical rational function.

Equivalence (i) Tropical rational functions, (ii) Continuous piecewise linear functions with integer coefficients, (iii) Neural networks satisfying assumptions (a)-(c).

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 30 / 62 Tropical Algebra of Neural Networks

Continuous piecewise linear function Let ν : Rd R. Then ν is a continuous piecewise linear function with → integer coefficients if and only if ν is a tropical rational function.

Equivalence (i) Tropical rational functions, (ii) Continuous piecewise linear functions with integer coefficients, (iii) Neural networks satisfying assumptions (a)-(c).

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 30 / 62 Optimal Transport Mapping

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 31 / 62 Optimal Transport Problem

(Ω, ) 

p

(p)

(D, )

Earth movement cost.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 32 / 62 Optimal Mass Transportation

Problem Setting Find the best scheme of transporting one mass distribution (µ,U) to another one (ν,V ) such that the total cost is minimized, where U,V are two bounded domains in Rn, such that Z Z µ(x)dx = ν(y)dy, U V

0 µ L1(U) and 0 ν L1(V ) are density functions. ≤ ∈ ≤ ∈

f x f(x) (µ, U) (ν, V )

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 33 / 62 Optimal Mass Transportation

For a transport scheme s ( a mapping from U to V )

s : x U y V , ∈ → ∈

the total cost is Z C(s) = µ(x)c(x,s(x))dx U where c(x,y) is the cost function.

f x f(x) (µ, U) (ν, V )

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 34 / 62 Cost Function c(x,y)

The cost of moving a unit mass from point x to point y.

Monge(1781): c(x,y) = x y . | − | This is the natural cost function. Other cost functions include

c(x,y) = x y p,p = 0 | − | 6 c(x,y) = log x y p− | − | c(x,y) = ε + x y 2,ε > 0 | − | Any function can be cost function. It can be negative.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 35 / 62 Optimal Transportation Map

Problem Is there an optimal mapping T : U V such that the total cost C is minimized, → C (T ) = inf C (s): s S { ∈ } where S is the set of all measure preserving mappings, namely s : U V satisfies → Z Z µ(x)dx = ν(y)dy, Borel set E V 1 s− (E) E ∀ ⊂

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 36 / 62 Solutions

Three categories: 1 Discrete category: both (µ,U) and (ν,V ) are discrete, 2 Semi-continuous category: (µ,U) is continuous, (ν,V ) is discrete, 3 Continuous category: both (µ,U) and (ν,V ) are continuous.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 37 / 62 Kantorovich’s Approach

Both (µ,U) and (ν,V ) are discrete. µ and ν are Dirac measures. (µ,U) is represented as

(µ ,p ),(µ ,p ), ,(µm,pm) , { 1 1 2 2 ··· } (ν,V ) is (ν ,q ),(ν ,q ), ,(νn,qn) . { 1 1 2 2 ··· } A transportation plan f : p q , f = f , f means how much { i } → { j } { ij } ij mass is moved from (µi ,pi ) to (νj ,qj ), i m,j n. The optimal mass transportation plan is: ≤ ≤ minfij c(pi ,qj ) f with constraints: n m ∑ fij = µi , ∑ fij = νj . j=1 i=1 Optimizing a linear energy on a convex set, solvable by linear programming method.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 38 / 62 Kantorovich’s Approach

Kantorovich won Nobel’s prize in economics.

(1, p1) fij

min f c(p ,p ), ∑ ij i j (2, p2) (1, q1) f ij such that (2, q2)

∑fij = µi ,∑fij = νj . j i mn unknowns in total. The complexity is quite high. (n, qn)

(m, pm)

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 39 / 62 Brenier’s Approach

Theorem (Brenier) If µ,ν > 0 and U is convex, and the cost function is quadratic distance,

c(x,y) = x y 2 | − | then there exists a convex function f : U R unique upto a constant, such that the unique optimal transportation→ map is given by the gradient map T : x ∇f (x). →

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 40 / 62 Brenier’s Approach

Continuous Category: In smooth case, the Brenier potential f : U R statisfies the Monge-Ampere equation →

 ∂ 2f  µ(x) det = , ∂xi ∂xj ν(∇f (x))

and ∇f : U V minimizes the quadratic cost → Z min x ∇f (x) 2dx. f U | − |

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 41 / 62 Semi-Continuous Category

Discrete Optimal Mass Transportation Problem

T

(pi,Ai) Wi

Ω

n n Given a compact convex domain U in R and p1, ,pk in R and ··· A1, ,Ak > 0, find a transport map T : U p1, ,pk with ··· 1 → { ··· } vol(T − (pi )) = Ai , so that T minimizes the transport cost Z x T (x) 2dx. U | − |

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 42 / 62 Power Diagram vs Optimal Transport Map

Theorem (Aurenhammer-Hoffmann-Aronov 1998) n n Given a compact convex domain U in R and p1, ,pk in R and ··· A , ,A > 0, ∑ A = vol(U), there exists a unique power diagram 1 ··· k i i k [ U = Wi , i=1

vol(W ) = A , the map T : W p minimizes the transport cost i i i 7→ i Z x T (x) 2dx. U | − |

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 43 / 62 Convex Geometry

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 44 / 62 Voronoi Decomposition

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 45 / 62 Voronoi Diagram

Voronoi Diagram n Given p1, ,pk in R , the Voronoi cell Wi ··· at pi is

W = x x p 2 x p 2, j . i { || − i | ≤ | − j | ∀ }

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 46 / 62 Power Distance

Power Distance

Given p associated with a pow(pi, q) i ri sphere (pi ,ri ) the power n distance from q R to pi is q ∈ pi pow(p ,q) = p q 2 r 2. i | i − | − i

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 47 / 62 Power Diagram

n Given p1, ,pk in R and power weights h1, ,hk , the power Voronoi ··· ··· cell Wi at pi is W = x x p 2 + h x p 2 + h , j . i { || − i | i ≤ | − j | j ∀ }

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 48 / 62 PL convex function vs. Power diagram

Lemma

Suppose f (x) = max x,pi + hi is a piecewise linear convex{h function,i } then its gradient map induces a power diagram, uh(x)

πj W = x ∇f = p . i { | i } Fi Proof. x,p + h x,p + h is equivalent to i i j j Ω h i ≥ h i Wi x p 2 2h p 2 x p 2 2h p 2. | − i | − i − | i | ≤ | − j | − j − | j |

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 49 / 62 Piecewise Linear Convex Function

A Piecewise Linear convex function

uh(x)

f (x) := max x,p + h i = 1, ,k πj {h i i i | ··· } Fi produces a convex cell decomposition Wi of Rn:

Wi = x x,pi + hi x,pj + hj , j Ω { |h i ≥ h i ∀ } Wi Namely, W = x ∇f (x) = p . i { | i }

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 50 / 62 Alexandrov Theorem

Theorem (Alexandrov 1950) Given Ω compact convex domain in Rn, n p1, ,pk distinct in R ,A1, ,Ak > 0, ··· ··· such that ∑Ai = Vol(Ω), there exists PL uh(x) convex function πj

Fi f (x) := max x,p + h i = 1, ,k {h i i i | ··· } unique up to translation such that

Ω Vol(W ) = Vol( x ∇f (x) = p ) = A . Wi i { | i } i Alexandrov’s proof is topological, not variational. It has been open for years to find a constructive proof.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 51 / 62 Variational Proof

Theorem (Gu-Luo-Sun-Yau 2013) n n Ω is a compact convex domain in R , y1, ,yk distinct in R , µ a ··· positive continuous measure on Ω. For any ν , ,ν > 0 with 1 ··· k ∑ν = µ(Ω), there exists a vector (h , ,h ) so that i 1 ··· k u(x) = max x,p + h {h i i i }

satisfies µ(Wi Ω) = νi , where Wi = x ∇f (x) = pi . Furthermore, h is the maximum point∩ of the convex function{ | }

k Z h k E(h) = ∑ νi hi ∑ wi (η)dηi , i=1 − 0 i=1

where w (η) = µ(W (η) Ω) is the µ-volume of the cell. i i ∩

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 52 / 62 Geometric Interpretation

One can define a cylinder through ∂Ω, the cylinder is truncated by the R h xy-plane and the convex polyhedron. The energy term ∑wi (η)dηi equals to the volume of the truncated cylinder.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 53 / 62 Computational Algorithm

uh(x)

πj

Fi

Ω

Wi

Definition (Alexandrov Potential) The concave energy is

k Z h k E(h1,h2, ,hk ) = ∑ νi hi ∑ wj (η)dηj , ··· i=1 − 0 j=1

Geometrically, the energy is the volume beneath the parabola.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 54 / 62 Computational Algorithm

uh(x)

πj

Fi

Ω

Wi

The gradient of the Alexanrov potential is the differences between the target measure and the current measure of each cell

∇E(h ,h , ,h ) = (ν w ,ν w , ,ν w ) 1 2 ··· k 1 − 1 2 − 2 ··· k − k

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 55 / 62 Key Point

uh(x)

πj

Fi The essential computation of OMT is to compute a piecewise linear function.

Ω

Wi

Equivalence (i) Tropical rational functions, (ii) Continuous piecewise linear function: OMT, (iii) Neural networks.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 56 / 62 OMT by Neural Networks

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 57 / 62 OT vs. Tropical Geometry vs. NN

Optimal Transport Tropical Geometry Neural Network

sample data pi tropical power pi weight matrix hyperplane < x,p > +h tropical monomial h xpi one neuron i i i upper envelope tropical polynomial neural network maxk < x,p > +h k h xpi i=1{ i i } ⊕i=1 i power diagram tropical hypersurface activated path power weights hi unknowns hi bias

Tropical Polynomial Upper Envelope of Polytope

πi(h) πi∗

proj proj∗ u ∇ h

W (h) i yi

Tropical Hypersurfaces Subdivision of Newton Polygon

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 58 / 62 Merits

OMT map can be represented as a tropical polynomial, and further realized using neural network, the weights are fixed, only train the biases; the number of parameters equal to that of the training samples; the method is independent of the dimension, general for arbitrary dimension; convex optimization, fast convergence, easy to train.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 59 / 62 OMT-NN Generative Model

(a) training data (b) generated data

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 60 / 62 Conclusion

The fundamental concepts of tropical geometry; The equivalence relation between tropical rational function and ReLU DNN; The tropical geometric representation of OMT; The realization of OMT using neural network.

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 61 / 62 Thanks

Thank you!

For more information, please email to [email protected].

Na Lei ([email protected]) (DLUT) Tropical Geometry, OMT and NN 01/22/2019 62 / 62