Eigenpairs Lecture 1 January 20th, 2021

1 Eigenpairs

2 . 1 Relevant sources for background self- study Ian Goodfellow, Yoshua Bengio and Aaron Courville: Deep Learning MIT Press, 2016. Jure Lescovec, Anand Rajaraman, Jeffrey D. Ullmann Mining of Massive datasets MIT Press, 2016.

2 . 2 Spectral Analysis

3 . 1 Eigenpairs A has a real λ and a vector e→ s.t. Ae→ = λe→

λ is an eigenvalue and e→ an eigenvector of A. In principle, if |A| = n there should be n such pairs. In practice,

they might not be real, nor ≠ 0

are always costly to nd.

3 . 2 A A is called positive semidenite iff

T A is symmetric (A = A ) T x→ Ax→ ≥ 0 for any x→

In such case its eigenvalues are non-negative: λi ≥ 0.

3 . 3 Underlying idea, I Eigenvectors, i.e., solutions to Ax = λx describe the direction along which matrix A operates an expansion as opposed to

deformation

3 . 4 Example: shear mapping

[[1, .27], [0, 1]]

x x + 3 y [ ] ⟶ [ 11 ] y y

3 . 5 The blue is unchanged:

T an [x, 0] eigenvector corresponding to λ = 1

3 . 6 Eigen. of user-activity matrices Eigenvectors are always orthogonal with each other: they describe alternative directions, interpretable as topic Eigenvalues expand one’s afliation to a specic topic.

3 . 7 Simple all-pairs extraction via Numpy/LA:

√ 2 2 Caveat: e-values come normalized: λ1 + … λn = 1 1 hence, multiply them by √n

3 . 8 def find_eigenpairs(mat): """ Testing the quality of eigenvector/eigenvalue computation by numpy """

n = len(mat)

m = len(mat[0])

eig_vals, eig_vects = la.eig(mat)

# they come in ascending order, take the last one on the right dominant_eig = abs(eig_vals[-1])

print(dominant_eig)

print(eig_vals)

3 . 9 Norms, distances etc.

4 . 1 Euclidean norm √ T √ m 2 ||x|| = x x = ∑i=1 xi

General case: 1 m 1 p p p p p p ||x||p = (|x1| + |x1| + … |xm| ) = (∑i=1 |xi| )

4 . 2 The Frobenius norm || ⋅ ||F extends || ⋅ ||2 to matrices

4 . 3 Normalization The unit or normalized vector of x x 1 u = = ( )x ||x|| ||x||

1. keeps the directon 2. norm is set to 1.

4 . 4 Computing Eigenpairs

5 . 1 With Maths Mx = λx

Handbook solution: solve the equivalent (Mx − λI)x = 0

A non-zero x is associated to a solution of |Mx − λI| = 0

α In Numerical Analysis Θ(n ) methods are available.

nd the λs that makes | … | = 0, then nd the associate x eig.

5 . 2 With Computer Science α At Google scale, Θ(n ) methods are not advised. Ideas:

1. nd the e-vectors rst, with an iterated method. 2. interleave iteration with control on the expansion in value

T x0 = [1, 1, … 1]

Mxk xk+1 = ||Mxk||

until a x point: xl+1 ≈ xl.

5 . 3 Now, eliminate the contribution of the rst eigenpair:

∗ ′ T M = M − λ1x1x1

∗ 2 Repeat the iteration on M . Times are in Θ(dn )

5 . 4