When Is a Set of Lines in Space Convex? Jacob E. Goodman The Definition of a Convex Set 2. idempotence: conv (conv S) = conv S. d In R , a set S of points is convex if the line seg- 3. antiexchange: conv S = S, x, y / S, x = y , ment joining any two points of S lies completely y conv (S x )= x/conv (S ∈y ) (Fig-6 within S (Figure 1). The purpose of this article is ure∈ 3). ∪{ } ⇒ ∈ ∪{ } to describe a recent extension of this concept of convexity to the Grassmannian and to discuss its connection with some other ideas in geometry. More specifically, the extension is to the so-called “affine Grassmannian” Gk,d0 , the open manifold that parametrizes all the k-dimensional flats (trans- lates of linear subspaces) in Rd. In other words, rather than convex sets of points, we will be talk- ing about convex sets of lines, for example, or of planes. Much of the material that this article deals with is based on joint work of the author’s with Richard Pollack [7, 8], as well as with Rephael Wenger and others [3, 10]. Figure 1. A convex point set. Which properties of convex point sets would we expect to hold also for convex sets of k-flats? The basic setup for point sets is this. To any set S of points is associated a set conv S containing S, called its convex hull (Figure 2), that satisfies: 1. monotonicity: S T = conv S conv T. ⊂ ⇒ ⊂ Jacob E. Goodman is professor of mathematics at the City College of the City University of New York. His e-mail ad- Figure 2. The convex hull of a set of four points. dress is [email protected]. Research partially supported by NSF grant DMS-9322475, NSA grant MDA904-95-H-1012, and PSC-CUNY grant 666366. This article is based on an address presented at the Montreal meeting of the American Mathematical So- ciety in September 1997 under the title “Convex geome- try on affine Grassmannians and geometric transver- S y x sals”. I am grateful to Hugo Rossi for inviting me to write an article for the Notices on my recent work and to Richard Pollack and Rephael Wenger for their helpful comments. —J. E. Goodman Figure 3. The antiexchange property. 222 NOTICES OF THE AMS VOLUME 45, NUMBER 2 (Property 3 says essentially that conv induces a par- tial ordering on the complement of any convex set, which can be thought of intuitively as represent- ing how “far away” points are from the set.) 4. nonsingular affine invariance: conv commutes with the action of the affine group. (If A(d,R) is the group of nonsingular affine transformations— nonsingular linear transformations plus transla- tions— Figure 4. x conv a, b, c = x0 conv a0,b0,c0 . Lα d ∈ { } ⇒ ∈ { } A(d,R)= L GL(d,R),α R , (" 01# ∈ ∈ ) and if σ A(d,R), then conv (σS)=σ (conv S) for ∈ S Rd; see Figure 4.) Then once we know what the ⊂ H1 convex hull of a set is, a set S is said to be convex S if conv S = S. H The first three of these are usually taken to be the defining properties of what is known as an ab- stract “convex hull” operation, i.e., of a convexity x structure, and the last condition means that this convexity structure is a natural one in the affine space Rd. But if we want to define a similar convexity H structure on Gk,d0 , for example on the set of lines 2 F in 3-space, we immediately run up against the fol- lowing apparent obstacle. One of the characteris- Figure 5. Point x is surrounded by set S. tic features of a convex set of points is that it is connected. Yet we can show: A) x conv S iff there is a flat F containing x ∈ Theorem 1 [8]. There is no notion of convexity for within which x is surrounded by the points of S lines or higher-dimensional flats that is nonsin- lying in F. (One cannot simply say “iff x is sur- gular-affine-invariant, that satisfies the antiex- rounded by the points of S”, since S may be a change property, and in which all convex sets are lower-dimensional set.) connected. And of course we also have This is not difficult to prove. It means that we B) x conv S iff every convex point set meeting have to give up something. It turns out that if one ∈ is willing to give up the connectedness itself, to every point of S also meets x. (This is trivially true: drop one’s insistence that a convex set of flats it just amounts to saying that conv S is the inter- should always be connected, then a rich theory section of all the convex point sets containing S.) emerges, one that extends many of the properties It turns out that “surrounded by” still makes per- of convexity for point sets and that links up to a fectly good sense when the basic objects are flats number of problems that have recently been stud- of some fixed dimension k from 1 to d 1 rather ied under the heading of what is sometimes called − than simply points, and that (A) and (B) are still “geometric transversal theory”. equivalent in that setting. And it also turns out that There are two ways to describe what the con- they imply the four basic properties of the convex vex hull of a set of k-flats is, both of which extend d characterizations of the convex hull of a point set. hull operator in R : monotonicity, idempotence, First let us go back to points for a moment. antiexchange, and invariance with respect to the affine group, and, happily, many other properties Definition 1. Let us say that a point x in a flat F as well. is surrounded by a set of points S in F if any hy- perplane (flat of codimension 1) H in F passing So here is our definition of the convex hull of a through x lies strictly between two parallel hy- set of k-flats in d-space: perplanes H1 and H2 in F, each containing a point Definition 2. A k-flat l0 belongs to the convex hull of S; in other words, any such hyperplane is conv of a set of k-flats in Rd if it satisfies ei- “trapped” by points of S if it tries to escape by con- L L ther of the following two conditions: tinuous translation to infinity (Figure 5). A) There is a flat F containing l0 within which l0 is Then it is easy to see that surrounded by the flats of lying in F; L FEBRUARY 1998 NOTICES OF THE AMS 223 l 2 H 2 If G10 ,3, a line l0 is in conv if and only if (FigureL⊂ 6) L l’ H A 0) l , or 0 ∈L l A 1) l0 lies between two parallel lines l1,l2 , or l’ 1 H ∈L l l’ l 1 1 2 A 2) any plane passing through l0 can be moved (A )(A )(A ) parallel to itself a positive distance in either di- 0 1 2 rection until itΠ contains lines of . L (A ) describes the “generic” way in which a line is Figure 6. Line l0 is surrounded by other lines. 2 surrounded by other lines, while (A 0) and (A 1) de- scribe the degenerate ways. And then we have Theorem 2 [8]. (A) and (B) in Definition 2 are equiv- alent. Theorem 3. The operator conv on Gk,d0 satisfies 1. monotonicity: 1 2 = conv 1 conv 2. s ε L* L ⊂L ⇒ L ⊂ L 2. idempotence: conv (conv ) = conv . L L L 3. antiexchange: conv = , l1,l2 / ,l1 = l2, Figure 7. The dual of a set of lines. L L ∈L 6 l2 conv ( l1 )= l1 / conv ( l2 ). ∈ L∪{ } ⇒ ∈ L∪{ } 4. nonsingular affine invariance: conv commutes with the action of the affine group. (If σ A(d,R) ε ∈ l S* induces σ : G0 G0 , then conv (σ )= k k,d → k,d kL σ (conv ) for G0 .) k L L⊂ k,d There is a nice way to look at condition (B) in the definition. Given a set of k-flats in Rd, let L us define its dual, ∗, to be the set of all convex point sets S, each ofL which meets all the flats of S (Figure 7). And given a family S of convex point L S sets, define its k-dual S ∗ (actually S ∗k, but sim- ply S∗ if we fix k) to be the set of all k-flats l meet- Figure 8. The dual of a set of convex point sets. ing all the members of S, the so-called k-trans- versals of the family S (Figure 8). Then condition (B) just amounts to saying that the convex hull of a set of flats is its double dual ∗∗. So if k =1, say, startL with a set of four linesL as in Figure 9, L ** look at all the convex point sets meeting them LL( ), and then look at all the lines meeting those L∗ convex sets ( ∗∗). They constitute the convex hull. It is trivial Lto see that this concept of duality sat- isfies the usual conditions: Figure 9. The convex hull of a set of lines. 1. 1 2 = ∗ ∗; S 1 S 2 = S ∗ S ∗; L ⊂L ⇒L1 ⊃L2 ⊂ ⇒ 1 ⊃ 2 2.