Fundamental Convex Euclidean Geometry and its Applications Jon Dattorro June 2003 c 2001 Jon Dattorro, all rights reserved. ° for Jennie Columba ♦ Antonio ♦ ♦ and Sze Wan Sooner or later just like the world first day, Sooner or later we learn to throw the past away. Gordon Sumner − Preface Philosophy in Ph.D., to some these pages may seem stark; quite the con- trary... Very few places in life is it possible to achieve a glimpse of perfection, e.g., musical performance, sports,... Why do this... purpose is belief will make life better for others in some re- spect, reflects hope for the future. Passion...art...what we leave behind...life is short... do this regardless -Jon Dattorro, Stanford 2003 i § § ´ G § ´ § ´ § ´ § ´ § ´ ? § ´ § ? § ´ § ´ § ´ ? ? § ´ § ´ § § ´ ii Acknowledgements Reflecting on help given me all along the way, I am awed by my debt of gratitude: Anthony, Teppo Aatola, Bob Adams, Chuck Bagnaschi, Jeffrey Barish, Keith Barr, Steven Bass, David Beausejour, Leonard Bellin, Dave Berners, Barry Blesser, Stephen Boyd, Merril Bradshaw, Gary Brooks, Jack Brown, Patti Brown, Joe Bryan, Barb Burk, John Calderone, Kathryn Cal´ı,Terri Carter, Reginald Centracchio, Bonnie Charvez, Michael Chille, Jan Chomysin, John Cioffi, Sandra, Josie, and Freddy Colletta, Judy Copice, Kathy and Rita Copley, Gary Corsini, Domenico, Yvonne, Christine, and John D’Attoro, Cosmo Dellagrotta, Robin DeLuca, Lou D’Orsaneo, Tom Destefano, Peggy Donatelli, Jeannie Duguay, Pozzi Escot, Terry Fayer, Maryam Fazel, Carol Feola, Valery and Evelyn Ferrullo, Martin Fischer...(Brown U.), Salvatore Fransosi, Stanley Freedman, Tony Garnett, Ed Gauthier, Jeanette Genovese, Emilia Giglio, Chae Teok Goh, Gene Golub, Tricia Goodrich, Heather Graham, Robert Gray, David Griesinger, Robert Haas, George Heath et al..., Roy Hendrickson, Martin Henry, Mar Hershenson, Haitham Hindi, Evan Holland, Leland Jackson, Rene Jaeger, Joakim Jalden, Sung Dae Jun, Max Kamenetsky, Seymore Kass..., Patricia Keefe, James Koch, Paul Kruger, Helen Kung, Kurzweil..., Mike Ladra, Lynn Lamberis, Paul Lansky, Karen Law, Francis Lee, Scott Levine, Hal Lipset, Nando Lopez-Lezcano, Pat Macdonald, Ed Mahler, Sylvia and David Martinelli, Joyce, Al, and Tilly Menchini, Lisa McFall, Ginny Magee, Jamie Jacobs May, Paul and Angelia McLean, Teresa Meng, Tom Metcalf, Andy Moorer, Denise Murphy, Walter Murray, John O’Donnell, Ray Ouellette, Sal Pandozzi, Bob Peretti, Manny Pokotilow, Timothy Rapa, Doris Rapisardi, Katharina Reissing, Myron Romanul, Marvin Rous, Antonetta and Carlo Russo, Eric Safire, Anthony, Susan, and Esther Scorpio, John Senior, Robin Silver, Josh, Richard, and Jeremy Singer, Terry Stancil, Timothy Stilson, Brigitte Strain, John Strawn, Albert Siu-Toung, David Taraborelli, Robin Tepper, Susan Thomas, Stavros Toumpis, Georgette Trezvant, Jack, Dan, Stella, and Rebecca Tuttle, Maryanne Uccello, Joe Votta, Julia Wade, Shelley Marged Weber, Jennifer Westerlind, Bernard Widrow, Todd Wilson, WinEdt, George and MaryAnn Young, Donna and Joe Zagami iii 1 INTRODUCTION 1 1 Introduction There will be a virtual flood of applications with the realization that many problems hitherto believed non-convex can be transformed or relaxed into convexity. example, LMI books [1] [2]. Circuit design: analog/digital filter synthesis [3], chip design [4, 4.7], [Barcelona], economics [5] [6] [7] ... § [4, 4.3] [8] discuss the relaxation of NP-hard combinatorial problems by semidefinite§ programming. antenna array design, structural mechanics, med- ical imaging, ... Convex geometry and linear algebra are inextricably bonded... Summarize what lies ahead... 2 ESSENTIAL CONVEXITY THEOREMS, DEFINITIONS 2 2 Essential convexity theorems, definitions There is relatively less published pertaining to matrix-valued convex sets and functions. We present only a few pertinent results; purposely abbreviated. We assume the reader to be comfortable with chapters 2 and 3 from [9], while familiar with chapters 4 and 5 there. The essential references are [10] [9] [11]. The reader is referred to [12] [13] [14] [15] [16] (in that order) for a more comprehensive treatment of convexity. 2.1 Sets Definition. Convex set. A set is convex iff for all Y , Z and 0 µ 1, C ∈ C ≤ ≤ µY + (1 µ)Z (1) − ∈ C ⋄ Under that defining constraint on µ , the linear sum in (1) is called a convex combination of Y and Z . If Y and Z are points in Euclidean space n m n R or R × , then (1) represents the closed line segment joining them. All line segments are thereby convex sets; so are all affine sets ( 3.1.1), any ray ( 3.4.2), halfspace ( 3.2.1), hyperplane ( 3.2.2), and subspace.§ 1 Apparent from§ the definition, a§ convex set is a connected§ set. [18, 3.4, 3.5] § § Example. Orthant: the name given to a higher-dimensional generaliza- 2 tion of the term quadrant from the classical Cartesian partition of R . The n n orthant Ri- for example, identifies the region in R whose members’ sole neg- ative coordinate is their ith (analog to quadrant II or IV). The most common n n n is the nonnegative orthant R+ or R+× (I) to which membership denotes all nonnegative vector- or matrix-entries respectively. The nonpositive orthant n n n R- or R-× (III) denotes all negative or zero entries. Orthant convexity is easily verified by definition (1).2 ¤ 1A nonempty subset of a vector space Rn is called a subspace if every vector of the form αx + βy , for α,β R , is in the subset whenever x and y are both in the subset. [5, 2.3] A subspace contains∈ the origin, by definition, and is a convex set. Any subspace§ that does not constitute the whole vector space is called a proper subspace; for example, any hyperplane through the origin. The vector space is itself a subspace, [17, 2.1] inclusively, although not proper. § 2We will later learn that all orthants are self-dual simplicial cones. ( 3.6.2) § 2 ESSENTIAL CONVEXITY THEOREMS, DEFINITIONS 3 Theorem. Intersection. [9, 2] [11, 2] The intersection of an arbitrary collection of convex sets is convex.§ § ⋄ Theorem. Image, Inverse image. [11, 3] [9, 2] Let f be a mapping p k m n § § from R × to R × . The image of a convex set under any affine function • C m n f( )= f(X) X R × (2) C { | ∈ C} ⊆ is convex. The inverse image of a convex set , • F 1 p k f − ( )= X f(X) R × (3) F { | ∈F} ⊆ a single or many-valued mapping, under any affine function f is convex. ⋄ Each converse of this two-part theorem is generally not true; id est, given f affine, a convex image f( ) does not imply that set is convex, and C 1 C neither does a convex inverse image f − ( ) imply that set is convex. A counter-example is easy to visualize whenF the affine function isF an orthogonal projector [19] [5]: Corollary. Projection on subspace. [11, 3] The orthogonal projection of a convex set on a subspace is another convex§ set. ⋄ The corollary is true more generally for projection on hyperplanes. [14, 6.6] Again, the converse is not true. Shadows, for example, are umbral projections§ that can be convex when the object providing the shade is not. 2.1.1 Vectorized matrix inner product Euclidean space Rn comes equipped with a vector inner product <y,z> = yTz (4) that is linear. The vector inner product for matrices is calculated just as it p k pk is for vectors by first transforming a matrix in R × to a vector in R by 2 ESSENTIAL CONVEXITY THEOREMS, DEFINITIONS 4 concatenating its columns in the natural order. For lack of a better term, we shall call that transformation vectorization. For example, the vectorization p k of Y = [y y y ] R × is [20] [21] 1 2 ··· k ∈ y1 ∆ y2 vec Y = . Rpk (5) . ∈ yk Then the vectorized-matrix inner product is the trace of the matrix inner p k product; for Z R × , [9, 2] [10, 0.3.1] [22, 8] [23, 2.2] ∈ § § § § <Y,Z> =∆ tr(Y TZ) = vecT Y vec Z (6) where tr(Y TZ) = tr(ZY T ) = tr(Y ZT ) = tr(ZT Y )= 1T(Y Z)1 (7) ◦ and where denotes the Hadamard product3 of matrices [24] [25, 1.1.4]. ◦ § For example, consider any particular vectors v and w and take any ele- p k ment 1 from a matrix-valued set in R × . Then the vector inner product of withC vwT is C1 <vwT , > = vT w = tr(wvT ) = 1T (vwT ) 1 (8) C1 C1 C1 ◦ C1 ¡ ¢ p k Example. Application of the image theorem. Suppose the set R × p k C ⊆ is convex. Then for any particular vectors v R and w R , the set of vector inner products ∈ ∈ ∆ = vT w = <vwT , > R (9) ℜ C C ⊆ is convex. This result is a consequence of the image theorem. Yet it is easy to show directly that a convex combination of inner products from remains an element of . ℜ ¤ ℜ To verify that, take any two elements and from the convex matrix- C1 C2 valued set , and then form the vector inner products (9) that are two C 3The Hadamard product is a simple entry-wise product of corresponding entries from two matrices of like size; id est, not necessarily square. 2 ESSENTIAL CONVEXITY THEOREMS, DEFINITIONS 5 elements of by definition. Now make a convex combination of those inner products; videlicetℜ , for 0 µ 1, ≤ ≤ µ <vwT , > + (1 µ) <vwT , > = <vwT , µ + (1 µ) > (10) C1 − C2 C1 − C2 The two sides are equivalent by linearity of the inner product. The right-hand side remains a vector inner product of vwT with an element µ + (1 µ) C1 − C2 from the convex set ; hence belongs to . Since that holds true for any two elements from C, then it must be a convexℜ set. ¨ ℜ T p k More generally, vw may be replaced with any particular matrix Z R × ∈ while convexity of the set <Z , > R persists.