A Bit About Hilbert Spaces David Rosenberg New York University October 29, 2016 David Rosenberg (New York University ) DS-GA 1003 October 29, 2016 1 / 9 Inner Product Space (or “Pre-Hilbert” Spaces) An inner product space (over reals) is a vector space V and an inner product, which is a mapping h·,·i : V × V ! R that has the following properties 8x,y,z 2 V and a,b 2 R: Symmetry: hx,yi = hy,xi Linearity: hax + by,zi = ahx,zi + b hy,zi Postive-definiteness: hx,xi > 0 and hx,xi = 0 () x = 0. David Rosenberg (New York University ) DS-GA 1003 October 29, 2016 2 / 9 Norm from Inner Product For an inner product space, we define a norm as p kxk = hx,xi. Example Rd with standard Euclidean inner product is an inner product space: hx,yi := xT y 8x,y 2 Rd . Norm is p kxk = xT y. David Rosenberg (New York University ) DS-GA 1003 October 29, 2016 3 / 9 What norms can we get from an inner product? Theorem (Parallelogram Law) A norm kvk can be generated by an inner product on V iff 8x,y 2 V 2kxk2 + 2kyk2 = kx + yk2 + kx - yk2, and if it can, the inner product is given by the polarization identity jjxjj2 + jjyjj2 - jjx - yjj2 hx,yi = . 2 Example d `1 norm on R is NOT generated by an inner product. [Exercise] d Is `2 norm on R generated by an inner product? David Rosenberg (New York University ) DS-GA 1003 October 29, 2016 4 / 9 Pythagorean Theroem Definition Two vectors are orthogonal if hx,yi = 0. We denote this by x ? y. Definition x is orthogonal to a set S, i.e. x ? S, if x ? s for all x 2 S. Theorem (Pythagorean Theorem) If x ? y, then kx + yk2 = kxk2 + kyk2. Proof. We have kx + yk2 = hx + y,x + yi = hx,xi + hx,yi + hy,xi + hy,yi = kxk2 + kyk2. David Rosenberg (New York University ) DS-GA 1003 October 29, 2016 5 / 9 Projection onto a Plane (Rough Definition) Choose some x 2 V. Let M be a subspace of inner product space V. Then m0 is the projection of x onto M, if m0 2 M and is the closest point to x in M. In math: For all m 2 M, kx - m0k 6 kx - mk. David Rosenberg (New York University ) DS-GA 1003 October 29, 2016 6 / 9 Hilbert Space Projections exist for all finite-dimensional inner product spaces. We want to allow infinite-dimensional spaces. Need an extra condition called completeness. A space is complete if all Cauchy sequences in the space converge. Definition A Hilbert space is a complete inner product space. Example Any finite dimensional inner product space is a Hilbert space. David Rosenberg (New York University ) DS-GA 1003 October 29, 2016 7 / 9 The Projection Theorem Theorem (Classical Projection Theorem) H a Hilbert space M a closed subspace of H For any x 2 H, there exists a unique m0 2 M for which kx - m0k 6 kx - mk 8m 2 M. This m0 is called the [orthogonal] projection of x onto M. Furthermore, m0 2 M is the projection of x onto M iff x - m0 ? M. David Rosenberg (New York University ) DS-GA 1003 October 29, 2016 8 / 9 Projection Reduces Norm Theorem Let M be a closed subspace of H. For any x 2 H, let m0 = ProjM x be the projection of x onto M. Then km0k 6 kxk, with equality only when m0 = x. Proof. 2 2 kxk = km0 + (x - m0)k (note: x - m0 ? m0) 2 2 = km0k + kx - m0k by Pythagorean theorem 2 2 2 km0k = kxk - kx - m0k 2 If kx - m0k = 0, then x = m0, by definition of norm. David Rosenberg (New York University ) DS-GA 1003 October 29, 2016 9 / 9.