2. Norm, Distance, Angle

L. Vandenberghe ECE133A (Spring 2021) 2. Norm, distance, angle

norm • distance • :-means algorithm • angle • complex vectors •

2.1 Euclidean norm

= (Euclidean) norm of vector 0 R : ∈ q 0 = 02 02 02= k k 1 + 2 + · · · + p = 0) 0

if = = 1, 0 reduces to absolute value 0 • k k | | measures the magnitude of 0 • sometimes written as 0 to distinguish from other norms, e.g., • k k2

0 = 0 0 0= k k1 | 1| + | 2| + · · · + | |

Norm, distance, angle 2.2 Properties

Positive deﬁniteness

0 0 for all 0, 0 = 0 only if 0 = 0 k k ≥ k k

Homogeneity

V0 = V 0 for all vectors 0 and scalars V k k | |k k

Triangle inequality (proved on page 2.7)

0 1 0 1 for all vectors 0 and 1 of equal length k + k ≤ k k + k k

Norm of block vector: if 0, 1 are vectors, 0 q = 0 2 1 2 1 k k + k k

Norm, distance, angle 2.3 Cauchy–Schwarz inequality

) = 0 1 0 1 for all 0, 1 R | | ≤ k kk k ∈

) moreover, equality 0 1 = 0 1 holds if: | | k kk k ) 0 = 0 or 1 = 0; in this case 0 1 = 0 = 0 1 • k kk k 0 ≠ 0 and 1 ≠ 0, and 1 = W0 for some W > 0; in this case • ) 0 < 0 1 = W 0 2 = 0 1 k k k kk k

0 ≠ 0 and 1 ≠ 0, and 1 = W0 for some W > 0; in this case • − ) 0 > 0 1 = W 0 2 = 0 1 − k k −k kk k

Norm, distance, angle 2.4 Proof of Cauchy–Schwarz inequality

1. trivial if 0 = 0 or 1 = 0 ) 2. assume 0 = 1 = 1; we show that 1 0 1 1 k k k k − ≤ ≤ 0 0 1 2 0 0 1 2 ≤ k − k ≤ k + k = 0 1 ) 0 1 = 0 1 ) 0 1 ( − ) ( ) − ) ( + ) ( ) + ) = 0 2 20 1 1 2 = 0 2 20 1 1 2 k k − ) + k k k k + ) + k k = 2 1 0 1 = 2 1 0 1 ( − ) ( + )

with equality only if 0 = 1 with equality only if 0 = 1 −

3. for general nonzero 0, 1, apply case 2 to the unit-norm vectors

1 0, 1 1 0 1 k k k k

Norm, distance, angle 2.5 Average and RMS value let 0 be a real =-vector

the average of the elements of 0 is • ) 01 02 0= 1 0 avg 0 = + + · · · + = ( ) = =

the root-mean-square value is the root of the average squared entry • s 02 02 02= 0 rms 0 = 1 + 2 + · · · + = k k ( ) = √=

Exercise: show that avg 0 rms 0 | ( )| ≤ ( )

Norm, distance, angle 2.6 Triangle inequality from Cauchy–Schwarz inequality for vectors 0, 1 of equal size

0 1 2 = 0 1 ) 0 1 k + k ( + ) ( + ) = 0) 0 1) 0 0) 1 1) 1 + )+ + = 0 2 20 1 1 2 k k + + k k 0 2 2 0 1 1 2 (by Cauchy–Schwarz) ≤ k k + k kk k + k k = 0 1 2 (k k + k k)

taking squareroots gives the triangle inequality • ) triangle inequality is an equality if and only if 0 1 = 0 1 (see page 2.4) • k kk k ) also note from line 3 that 0 1 2 = 0 2 1 2 if 0 1 = 0 • k + k k k + k k

Norm, distance, angle 2.7 Outline

norm • distance • :-means algorithm • angle • complex vectors • Distance the (Euclidean) distance between vectors 0 and 1 is deﬁned as 0 1 k − k

0 1 0 for all 0, 1 and 0 1 = 0 only if 0 = 1 •k − k ≥ k − k triangle inequality •

0 2 0 1 1 2 for all 0, 1, 2 k − k ≤ k − k + k − k c

a c k − k b c k − k

a a b b k − k 0 1 RMS deviation between =-vectors 0 and 1 is rms 0 1 = k − k • ( − ) √=

Norm, distance, angle 2.8 Standard deviation let 0 be a real =-vector

the de-meaned vector is the vector of deviations from the average • 0 ) 0 =  01 avg 0   1 1   0 − (0)   − ( ) )/   2 avg   02 1 0 =  0 avg 0 1 =  − . ( )  =  − ( . )/  − ( )  .   .     )   0= avg 0   0= 1 0 =   − ( )   − ( )/ 

the standard deviation is the RMS deviation from the average • ) 0 1 0 = 1 std 0 = rms 0 avg 0 1 = − (( )/ ) ( ) ( − ( ) ) √=

the de-meaned vector in standard units is • 1 0 avg 0 1 std 0 ( − ( ) ) ( )

Norm, distance, angle 2.9 Mean return and risk of investment

vectors represent time series of returns on an investment (as a percentage) • average value is (mean) return of the investment • standard deviation measures variation around the mean, i.e., risk • ak bk ck dk 10 10 10 10 5 5 5 5 0 k 0 k 0 k 0 k 5 5 5 5 − − − − (mean) return

3 c b 2 d 1 a

0 risk 0 1 2 3 4 5

Norm, distance, angle 2.10 Exercise show that avg 0 2 std 0 2 = rms 0 2 ( ) + ( ) ( )

Solution

0 avg 0 1 2 std 0 2 = k − ( ) k ( ) = ) ) ) 1 1 0 1 0 = 0 1 0 1 = − = − = ) 2 ) 2 ) 2 ! 1 ) 1 0 1 0 1 0 = 0 0 ( ) ( ) = = − = − = + =

) 2 1 ) 1 0 = 0 0 ( ) = − = = rms 0 2 avg 0 2 ( ) − ( )

Norm, distance, angle 2.11 Exercise: nearest scalar multiple

= given two vectors 0, 1 R , with 0 ≠ 0, ﬁnd scalar multiple C0 closest to 1 ∈ b

line ta t R taˆ { | ∈ }

Solution

squared distance between C0 and 1 is • ) ) ) ) C0 1 2 = C0 1 C0 1 = C20 0 2C0 1 1 1 k − k ( − ) ( − ) − + ) a quadratic function of C with positive leading coeﬃcient 0 0 derivative with respect to C is zero for • 0) 1 0) 1 Cˆ = = 0) 0 0 2 k k Norm, distance, angle 2.12 Exercise: average of collection of vectors

= given # vectors G ,..., G# R , ﬁnd the =-vector I that minimizes 1 ∈

I G 2 I G 2 I G# 2 k − 1k + k − 2k + · · · + k − k

x5 z

x2 x1

I is also known as the centroid of the points G1,..., G#

Norm, distance, angle 2.13 Solution: sum of squared distances is

2 2 2 I G1 I G2 I G# k − k= + k − k + · · · + k − k X 2 2 2 = I8 G1 8 I8 G2 8 I8 G# 8 8=1 ( − ( ) ) + ( − ( ) ) + · · · + ( − ( ) ) = X 2 2 2 = #I8 2I8 G1 8 G2 8 G# 8 G1 8 G# 8 8=1 − (( ) + ( ) + · · · + ( ) ) + ( ) + · · · + ( ) here G 9 8 is 8th element of the vector G 9 ( )

term 8 in the sum is minimized by • 1 I8 = G 8 G 8 G# 8 # (( 1) + ( 2) + · · · + ( ) )

solution I is component-wise average of the points G ,..., G#: • 1 1 I = G G G# # ( 1 + 2 + · · · + )

Norm, distance, angle 2.14 Outline

norm • distance • :-means algorithm • angle • complex vectors • :-means clustering

a popular iterative algorithm for partitioning # vectors G1,..., G# in : clusters

Norm, distance, angle 2.15 Algorithm

choose initial ‘representatives’ I1,..., I: for the : groups and repeat:

1. assign each vector G8 to the nearest group representative I 9

2. set the representative I 9 to the mean of the vectors assigned to it

initial representatives are often chosen randomly • as a variation, choose a random initial partition and start with step 2 • solution depends on choice of initial representatives or partition • can be shown to converge in a ﬁnite number of iterations • in practice, often restarted a few times, with diﬀerent starting points •

Norm, distance, angle 2.16 Example: ﬁrst iteration