MATH 355 Supplemental Notes Quadratic Forms

Quadratic Forms

Each of 2 x2 ?2x 7, 3x18 x11, and 0 ´ ` ´ ⇡ is a polynomial in the single variable x. But polynomials can involve more than one variable. For instance, 1 ?5x8 x x4 x3x and 3x2 2x x 7x2 1 ´ 3 1 2 ` 1 2 1 ` 1 2 ` 2 are polynomials in the two variables x1, x2; products between powers of variables in terms are permissible, but all exponents in such powers must be nonnegative integers to fit the classification polynomial. The degrees of the terms of

1 ?5x8 x x4 x3x 1 ´ 3 1 2 ` 1 2 are 8, 5 and 4, respectively. When all terms in a polynomial are of the same degree k, we call that polynomial a k-form. Thus, 3x2 2x x 7x2 1 ` 1 2 ` 2 is a 2-form (also known as a quadratic form) in two variables, while the dot product of a constant vector a and a vector x Rn of unknowns P

a1 x1 »a2fi »x2fi a x a x a x a x “ . . “ 1 1 ` 2 2 `¨¨¨` n n ¨ — . ¨ — . — — —a —x — n — n – fl – fl is a 1-form, or linear form in the n variables found in x. The quadratic form in variables x1, x2

T 2 2 x1 ab2 x1 ax1 bx1x2 cx2 { Ax, x , ` ` “ «x2 «b 2 c «x2 “ x y { for ab2 x1 A { and x . “ «b 2 c “ «x2 { Similarly, a quadratic form in variables x1, x2, x3 like

2x2 3x x x2 4x x 5x x can be written as Ax, x , 1 ´ 1 2 ´ 2 ` 1 3 ` 2 3 x y where 2 1.52 x ´ 1 A 1.5 12.5 and x x . “ »´ ´ fi “ » 2fi 22.50 x — — 3 – fl – fl Note that the coecient 5 of the x2x3 term could have been partitioned between entries a23 and a32 of A in any manner which sums to 5, but to put half as a23 and the rest as a32 (and similarly with

10 MATH 355 Supplemental Notes Approximating Functions of Multiple Variables other terms) turns A into a symmetric matrix. In fact, we can simply define a quadratic form to be any expression of the form x, Ax xTAx ATx, x , x y “ “ where A is symmetric. @ D

Approximating Functions of Multiple Variables

Polynomials are easy to di↵erentiate and evaluate, and we like to use them to approximate other functions. This is the content of Taylor’s theorem, encountered in Calculus:

Theorem 4 (Taylor’s Theorem for Real-Valued Functions): Suppose f : a R, a R R, p ´ ` qÑ where I is the open interval a R, a R centered at a of some positive radius R 0. Suppose p ´ ` q ° also that k 0 be an integer such that f k is continuous on I and f k 1 exists throughout ° p q p ` q I. Given any x I there exists a number c between x and a such that P k j k 1 f p q a j f p ` q c k 1 f x p q x a p q x a ` . p q“ j! p ´ q ` `! p ´ q j 0 ÿ“

The expression k j f p q a T x p q x a j kp q“ j! p ´ q j 0 ÿ“ is a kth degree polynomial known as the kth degree Taylor polynomial of f centered at a. It’s form arises from being the only polynomial of degree k which has an identical value as f at x a, “ an identical 1st derivative value as f at x a, right up to an identical kth derivative value as f at “ x a. In particular, the 2nd degree Taylor polynomial looks like “

f 2 a T x f a f 1 a x a x a p q x a . 2p q“ p q` p qp ´ q`p ´ q 2 p ´ q

Example 2:

Find the 2nd degree Taylor polynomial of sin x at ⇡ 4 . The answer is p q p´ { q ?2 ⇡ 2 ?2 ⇡ ?2 T x x x . 2p q“ 4 ` 4 ` 2 ` 4 ´ 2 ´ ¯ ´ ¯ See the video at https://www.youtube.com/watch?v=44PeKBY_ySQ for details, if interested.

11 MATH 355 Supplemental Notes Approximating Functions of Multiple Variables

When f x f x ,...,x is a suciently smooth, real-valued function of n real variables x , x ,...,x , p q“ p 1 nq 1 2 n there is a version of Taylor’s theorem which guides the approximation of f by polynomials in x1, x2,...,xn. I will not state that theorem here. But I will point out that if, as above, we focus on T , the Taylor polynomial in x ,...,x centered at x a whose terms are of degree two or less, it 2 1 n “ has a particularly nice formula:

1 T T2 x f a f a x a x a H a x a , p q“ p q`r p q¨p ´ q`2p ´ q f p qp ´ q where the gradient vector and Hessian matrix are

2 f 2 f 2 f 2 f B 2 a B a B a B a x x1 x2 x1 x3 x1 xn f B 1 p q B B p q B B p q¨¨¨ B B p q B a 2 2 2 2 x1 p q » f f f f fi B B a B 2 a B a B a f x2 x1 x x2 x3 x2 xn » B fi B B p q B 2 p q B B p q¨¨¨ B B p q x a — 2 2 2 2 B 2 p q — f f f f and B a B a B 2 a B a . f a — H f a — x3 x1 x3 x2 x x3 xn r p q“— . p q“— B B p q B B p q B 3 p q¨¨¨B B p q — . — . . . . . — — ...... — f — B — — x a — 2 2 2 2 — B n p q — f f f f – fl B a B a B a B 2 a — xn x1 xn x2 xn x3 x — B B p q B B p q B B p q¨¨¨ B n p q – fl 2 f B Notice that, under smoothness conditions discussed in Calculus, cross-partial derivatives x x B iB j 2 f B and x x with respect to the same two variables are equal, making the Hessian matrix symmetric. B jB i Hence, setting h x a, the expressions “ ´ 1 1 f a x a f a h and x a TH a x a hT H a h r p q p ´ q“r p q 2p ´ q f p qp ´ q“ 2 f p q ¨ ¨ ˆ ˙ are linear and quadratic forms in the variables of h, respectively, so we have

1 T 1 T2 a h f a f a h h H a h f a f a , h H a h, h (3) p ` q“ p q`r p q¨ ` 2 f p q “ p q`xr p q y ` 2 f p q B F as an approximation to values of f a h when h is small. p ` q } }

Example 3:

Find the gradient vector and Hessian matrix of

f x, y, z x3z yz2 p q“ ` at the point a 1, 2, 3 . “p q

The three 1st partial derivatives of f are f f f B 3x2z, B z2, B x3 2yz, x “ y “ z “ ` B B B 12 MATH 355 Supplemental Notes Approximating Functions of Multiple Variables

so 9 f 1, 2, 3 9 . r p q“» fi 13 — – fl As cross-partial derivatives are equal, we list only 6 di↵erent 2nd partial derivatives:

2 f 2 f 2 f 2 f 2 f 2 f B 0, B 3x2, B 2z, B 6xz, B 0, B 2y. x y “ x z “ y z “ x2 “ y2 “ z2 “ B B B B B B B B B Thus, the Hessian matrix generally uses formulas

6xz 03x2 18 0 3 002z and H 1, 2, 3 006. » fi f p q“» fi 3x2 2z 2y 364 — — – fl – fl Now, the f in this example is already a polynomial, chosen to be so in order to make the calculation of derivatives simple. In practice, you would probably find a 2nd degree polyno- mial approximation when f is not a polynomial. But to carry out the approximation of f near a 1, 2, 3 , we have “p q

1 T f 1 h , 2 h2, 3 h3 f 1, 2, 3 f 1, 2, 3 h h H 1, 2, 3 h p ` 1 ` ` q« p q`r p q ¨ ` 2 f p q 9 h1 901.5 h1 21 9 h h h h 003 h “ ` » fi ¨ » 2fi ` 1 2 3 » fi » 2fi 13 h ” ı 1.53 2 h — — 3 — — 3 – fl – fl – fl – fl 9h2 2h2 3h h 6h h 9h 9h 13h 21. “ 1 ` 3 ` 1 3 ` 2 3 ` 1 ` 2 ` 3 ` Compare the function value at 1.1, 1.95, 3.08 p q f 1.1, 1.95, 3.08 1.1 3 3.08 1.95 3.08 2 ⌘ 22.598, p q“p q p q`p qp q with the estimate at h 0.1, 0.05, 0.08 “p ´ q 9 0.1 2 2 0.08 2 3 0.1 0.08 6 0.05 0.08 9 0.1 9 0.05 13 0.08 21 ⌘ 22.593. p q ` p q ` p qp q` p´ qp q` p q` p´ q` p q`

13