Eigenpairs Lecture 1 January 20th, 2021 1 Eigenpairs 2 . 1 Relevant sources for background self- study Ian Goodfellow, Yoshua Bengio and Aaron Courville: Deep Learning MIT Press, 2016. Jure Lescovec, Anand Rajaraman, Jeffrey D. Ullmann Mining of Massive datasets MIT Press, 2016. 2 . 2 Spectral Analysis 3 . 1 Eigenpairs Matrix A has a real λ and a vector e→ s.t. Ae→ = λe→ λ is an eigenvalue and e→ an eigenvector of A. In principle, if |A| = n there should be n such pairs. In practice, they might not be real, nor ≠ 0 are always costly to nd. 3 . 2 A square matrix A is called positive semidenite iff T A is symmetric (A = A ) T x→ Ax→ ≥ 0 for any x→ In such case its eigenvalues are non-negative: λi ≥ 0. 3 . 3 Underlying idea, I Eigenvectors, i.e., solutions to Ax = λx describe the direction along which matrix A operates an expansion as opposed to rotation deformation 3 . 4 Example: shear mapping [[1, .27], [0, 1]] x x + 3 y [ ] ⟶ [ 11 ] y y 3 . 5 The blue line is unchanged: T an [x, 0] eigenvector corresponding to λ = 1 3 . 6 Eigen. of user-activity matrices Eigenvectors are always orthogonal with each other: they describe alternative directions, interpretable as topic Eigenvalues expand one’s afliation to a specic topic. 3 . 7 Simple all-pairs extraction via Numpy/LA: √ 2 2 Caveat: e-values come normalized: λ1 + … λn = 1 1 hence, multiply them by √n 3 . 8 def find_eigenpairs(mat): """ Testing the quality of eigenvector/eigenvalue computation by numpy """ n = len(mat) m = len(mat[0]) eig_vals, eig_vects = la.eig(mat) # they come in ascending order, take the last one on the right dominant_eig = abs(eig_vals[-1]) print(dominant_eig) print(eig_vals) 3 . 9 Norms, distances etc. 4 . 1 Euclidean norm √ T √ m 2 ||x|| = x x = ∑i=1 xi General case: 1 m 1 p p p p p p ||x||p = (|x1| + |x1| + … |xm| ) = (∑i=1 |xi| ) 4 . 2 The Frobenius norm || ⋅ ||F extends || ⋅ ||2 to matrices 4 . 3 Normalization The unit or normalized vector of x x 1 u = = ( )x ||x|| ||x|| 1. keeps the directon 2. norm is set to 1. 4 . 4 Computing Eigenpairs 5 . 1 With Maths Mx = λx Handbook solution: solve the equivalent (Mx − λI)x = 0 A non-zero x is associated to a solution of |Mx − λI| = 0 α In Numerical Analysis Θ(n ) methods are available. nd the λs that makes | … | = 0, then nd the associate x eig. 5 . 2 With Computer Science α At Google scale, Θ(n ) methods are not advised. Ideas: 1. nd the e-vectors rst, with an iterated method. 2. interleave iteration with control on the expansion in value T x0 = [1, 1, … 1] Mxk xk+1 = ||Mxk|| until a x point: xl+1 ≈ xl. 5 . 3 Now, eliminate the contribution of the rst eigenpair: ∗ ′ T M = M − λ1x1x1 ∗ 2 Repeat the iteration on M . Times are in Θ(dn ) 5 . 4.