Fundamentals of Media Processing Lecturer: 池畑 諭（Prof. IKEHATA Satoshi） 児玉 和也（Prof. KODAMA Kazuya） Support: 佐藤 真一（Prof. SATO Shinichi） 孟 洋（Prof. MO Hiroshi） Course Overview (15 classes in total) 1-10 Machine Learning by Prof. Satoshi Ikehata 11-15 Signal Processing by Prof. Kazuya Kodama Grading will be based on the final report. Fundamentals of Media Processing, Deep Learning Chapter 1-9 (out of 20) An introduction to a broad range of topics in deep learning, covering mathematical and conceptual background, deep learning techniques used in industry, and research perspectives. • Due to my background, I will mainly talk about “image” • I will introduce some applications beyond this book Fundamentals of Media Processing, Deep Learning 10/16 (Today) Introduction Chap. 1 Basic of Machine Learning (Maybe for beginners) 10/23 Basic mathematics (1) (Linear algebra, probability, numerical computation) Chap. 2,3,4 10/30 Basic mathematics (2) (Linear algebra, probability, numerical computation) Chap. 2,3,4 11/6 Machine Learning Basics (1) Chap. 5 11/13 Machine Learning Basics (2) Chap. 5 Basic of Deep Learning 11/20 Deep Feedforward Networks Chap. 6 11/27 Regularization and Deep Learning Chap. 7 12/4 Optimization for Training Deep Models Chap. 8 CNN and its Application Chap. 9 and more 12/11 Convolutional Neural Networks and Its Application (1) 12/18 Convolutional Neural Networks and Its Application (2) Chap. 9 and more Fundamentals of Media Processing, Deep Learning Linear Algebra Fundamentals of Media Processing, Deep Learning Scalars, Vectors, Matrices and Tensors Scalars 푎 = 1 0 풂 = = 0,1 푇 (1-D) Vector 1 0 1 0 2 (2-D) Matrix 퐴 = 퐴푇 = 퐴 ∈ ℝ2×2 2 3 1 3 퐴1,1 = 0, 퐴1,2 = 1, 퐴2,1 = 2, 퐴2,2 = 3 퐴 (N-D) Tensor 푖,푗,푘… Fundamentals of Media Processing, Deep Learning Multiplying Matrices and Vectors 풙푇풚 = 풚푇풙 퐴 퐵 + 퐶 = 퐴퐵 + 퐴퐶 풙 ∗ 풙 = 풙푇풙 = 풙 ퟐ 퐴퐵 푇 = 퐵푇퐴푇 푇 풙 2 = 1 Unit vector 풙 풚 = 0 Orthogonal −1 푇 Symmetric matrix 퐴퐴 = 퐼 Inverse matrix 퐴 = 퐴 푇 푇 퐴 퐴 = 퐴퐴 = 퐼 Orthogonal matrix Tr 퐴 = ෍ 퐴푖,푖 Trace operator 푖 L1 Norm 풙 ퟏ = ෍ |푥푖| L∞ Norm 풙 ∞ = max|푥푖| 풊 풊 ퟏ ퟐ ퟐ Frobenius Norm 퐴 = ෍ 퐴2 L2 Norm 풙 ퟐ = ෍ |푥푖| 퐹 푖,푗 풊 푖,푗 푇 L0 Norm 풙 ퟎ = # of nonzero entries 퐴 퐹 = Tr 퐴퐴 Fundamentals of Media Processing, Deep Learning Matrix Decomposition ◼ Decomposition of matrices shows us information about their functional properties that is not obvious from the representation of the matrix ⚫ Eigendecomposition ⚫ Singular value decomposition Differs in ⚫ LU decomposition ⚫ QR decomposition ◼ Applicable matrix ⚫ Jordan decomposition ◼ Decomposition type ⚫ Cholesky decomposition ◼ Application ⚫ Schur decomposition ⚫ Rank factorization ⚫ And more… Fundamentals of Media Processing, Deep Learning Eigendecomposition ⚫ Eigenvector and eigenvalue: 풗: Eigenvector 퐴풗 = 휆풗 휆: Eigenvalue ⚫ Eigendecomposition of a diagonalizable matrix : 푉: A set of eigenvectors 퐴 = 푉diag 휆 푉−1 휆: A set of eigenvalues ⚫ Every symmetric matrix has a real, orthogonal eigendecomposition 퐴 = 푄훬푄푇 ⚫ A matrix whose eigenvalues are all positive is positive definite, positive or zero is positive semidefinite Fundamentals of Media Processing, Deep Learning What Eigenvalues and Eigenvectors are? ◼ Eigenvalues and eigenvectors feature prominently in the analysis of linear transformations. The prefix eigen- is adopted from the German word eigen for "proper", "characteristic". Originally utilized to study principal axes of the rotational motion of rigid bodies, eigenvalues and eigenvectors have a wide range of applications, for example in stability analysis, vibration analysis, atomic orbitals, facial recognition, and matrix diagonalization. In this shear mapping the red arrow changes direction but the blue arrow does not. The blue arrow is an eigenvector of this shear mapping because it does not change direction, and since its length is unchanged, its eigenvalue is 1. https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors Fundamentals of Media Processing, Deep Learning Calculation of Eigendecomposition ⚫ Simply solve a linear equation 퐴풗 = 휆풗 → 퐴 − 휆퐼 풗 = 0 ⚫ 퐴 − 휆퐼 must not have inverse matrix, otherwise 푣 = 0 푛1 2 푛3 det 퐴 − 휆퐼 = 0 → 푝 휆 = 휆 − 휆1 휆 − 휆2 휆 − 휆3 … ⚫ Once 휆푠 are calculated, eigenvectors are calculated by: 퐴 − 휆퐼 풗 = 0 Fundamentals of Media Processing, Deep Learning Singular Value Decomposition (SVD) ⚫ Similar to eigendecomposition, but matrix need not be square. Singular value decomposition of a real matrix A is 퐴 = 푈퐷푉푇 퐴풖 = 풖 퐴푇풗 = 풗 ⚫ The left singular vectors of 퐵 are the eigenvectors of 퐴퐴푇 ⚫ The right singular vectors of 퐵 are the eigenvectors of 퐴푇퐴 ⚫ SVD is useful for the matrix inversion of non-square matrix (pseudo inverse matrix) or rank approximation (E.g., find nearest matrix whose rank is k) or solving a homogenous system Fundamentals of Media Processing, Deep Learning Solving Linear Systems ◼ Number of Eq = Number of Unknown (Determined) 푎 + 2b + 3c = 3 1 2 3 푎 3 −1 2푎 − b + c = 2 2 −1 1 푏 = 2 풙 = 퐴 풚 5 1 −2 푐 1 5a + b − 2c = 1 퐴 풙 풚 ◼ Number of Eq < Number of Unknown (Underdetermined) 푎 푎 + 2b + 3c = 3 1 2 3 3 푏 = ? 2푎 − b + c = 2 2 −1 1 푐 2 ◼ Number of Eq > Number of Unknown (Overdetermined) 푎 + 2b + 3c = 3 1 2 3 푎 3 2푎 − b + c = 2 2 −1 1 2 푏 = 5 1 −2 1 ? 5a + b − 2c = 1 푐 8 −1 6 0 8a − b + 6c = 0 Fundamentals of Media Processing, Deep Learning Moore-Penrose Pseudoinverse ◼ SVD gives the pseudoinverse of A as + + 푇 푇 + 1 퐴 = 푉퐷 푈 (퐴 = 푈퐷푉 , 퐷푖푖 = ) 퐷푖푖 ◼ The solution to any linear systems is given by 풙 = 퐴+풚 ◼ If the linear system is: ⚫ Determined: this is same as the inverse ⚫ Underdetermined: this gives us the solution with the smallest norm of x ⚫ Overdetermined: this gives us the solution with the smallest error Fundamentals of Media Processing, Deep Learning The Homogeneous System ◼ We may want to solve the homogenous system that is defined as 퐴풙 = 0 ◼ If det(퐴푇퐴) = 0, numerical solutions to a homogeneous system can be found with SVD or eigendecompositoin 퐴푥 = 0 퐴푇퐴푥 = 0 퐴푇퐴푣 = 휆푣 ⚫ Eigenvector corresponding to smallest eigen value (~=0) ⚫ Or, the column vector in V which is corresponding to the smallest singular value (퐴 = 푈퐷푉푇) Fundamentals of Media Processing, Deep Learning Example: Structure-from-Motion https://www.youtube.com/watch?v=i7ierVkXYa8 Fundamentals of Media Processing, Deep Learning Example: Structure-from-Motion • Relative camera poses could be computed from the point correspondences on images • Given camera pose (R,t), and camera intrinsics (K), 3D points X is given by triangulation 푥1 = 퐾1 푅1 푇1 푋෨ 푥2 = 퐾2 푅2 푇2 푋෨ 퐴푋 = 0 Homogenous system Fundamentals of Media Processing, Deep Learning Probability and Information Theory Fundamentals of Media Processing, Deep Learning Why probability? Most real problem is not deterministic. Which is this picture about Dog or Cat? Fundamentals of Media Processing, Deep Learning Three possible sources of uncertainty ⚫ Inherent stochasticity in the system e.g., Randomly shuffled card ⚫ Incomplete observability e.g., Monty Hall problem ⚫ Incomplete modeling e.g., A robot that only sees the discretized space ⚫ A random variable is a variable that can take on different values randomly. e. g., 푥 ∈ x Fundamentals of Media Processing, Deep Learning Monty Hall problem Fundamentals of Media Processing, Deep Learning Discrete case: Probability Mass Function 푃 푥 ⚫ The domain of P must be the set of all possible states of x • ∀푥 ∈ x, 0 ≤ 푃 푥 ≤ 1. An impossible event has probability 0 and no state can be less probable than that. Likewise, an event that is guaranteed to happen has probability 1, and no state can have a greater chance of occurring • σ푥∈x 푃 푥 = 1. We refer to this property as being normalized. Without this property, we could obtain probabilities greater than one by computing the probability of one of many events occurring ⚫ Example • Dice : P(x)=1/6, x is an event where f(x)=1,2,3,4,5,6 Fundamentals of Media Processing, Deep Learning Continuous case: Probability Density Function 푝 푥 ⚫ The domain of 푝 must be the set of all possible states of x • ∀푥 ∈ x, 푝 푥 ≥ 0. Note that we do not require 푝 푥 ≤ 1 • ∫ 푝 푥 푑푥 = 1 • Does not give the probability of a specific state directly e.g., p(0.0001) + p(0.0002) +….. >= 100%! ⚫ Example • What is the probability that randomly selected value within [0,1] is more than 0.5? Fundamentals of Media Processing, Deep Learning Marginal Probability ◼ The probability distribution over the subset ⚫ ∀푥 ∈ x, P 푥 = σ푦 푃 푥, 푦 (Discrete) ⚫ 푝 푥 = ∫ 푝 푥, 푦 푑푦 (Continuous) Table: The statistics about the student Male Female Tokyo 0.4 0.3 Q. How often students come from Tokyo? Outside Tokyo 0.1 0.2 Fundamentals of Media Processing, Deep Learning Conditional Probability ◼ The probability of an event, given that some other event has happened ⚫ 푃 푦 푥 = 푃 푦, 푥 /푃(푥) ⚫ 푃 푦, 푥 = 푃 푦 푥 푃(푥) ◼ Example: The boy and girl problem Mr. Jones has two children. One is a girl. What is the probability that the other is a boy? • Each child is either male or female. • Each child has the same chance of being male as of being female. • The sex of each child is independent of the sex of the other. Fundamentals of Media Processing, Deep Learning Conditional Probability [Boy/Boy, Boy/Girl, Girl/Boy, Girl/Girl] ⚫ 푃 푦 : The probability that “The other is a boy” ⚫ 푃 푥 : The probability that “One is a girl” ⚫ 푃 푦|푥 : The probability that “The other is a boy” when “One is a girl” ⚫ 푃 푦, 푥 : The probability that “One is a girl, the other is a boy” 푃 푦, 푥 2/4 2 푃 푦 푥 = = = 푃 푥 3/4 3 Fundamentals of Media Processing, Deep Learning Chain rule of Probability 1 (푛) 1 푛 푖 1 (푖−1) ⚫ 푃 푥 , … , 푥 = 푃 푥 Π푖=2푃(푥 |푥 , … , 푥 )