Modelling 1 SUMMER TERM 2020 ′ 푓 퐱 푥1 = 퐯, 퐛1 = 2 퐯, 퐛1 푥1 = 1.5 ′ 퐛1 퐛1 푥2 = 1 ′ 푥2 = 퐯, 퐛ퟐ 퐛2 ′ 퐛2 = 2 퐯, 퐛ퟐ ADDENDUM Co- and Contravariance Michael Wand · Institut für Informatik · . Covariance & Contravariance Representing Vectors Two operations in linear algebra 푓 퐱 ▪ Contravariant: Linear combination of vectors 푥1 = 3 푛 푥1 퐛 퐯 = ෍ 푥푖퐛푖 → 퐯 ≡ ⋮ 1 푥2 = 2 푥푛 푖=1 퐛2 ▪ Covariant: Projection on vectors (w/scalar product) 푥1 = 퐯, 퐛1 퐯, 퐛1 퐯 ≡ ⋮ 퐯, 퐛푛 퐛1 푥2 = 퐯, 퐛ퟐ 퐛2 Where is the difference? Change of basis 푓 퐱 ▪ Contravariant: 푥1 = 1.5 ▪ Keep same output vector: 퐛1 −1 푥2 = 1 퐛푖 → 퐓퐛푖 requires 퐱 → 퐓 퐱 퐛2 ▪ Covariant: ′ 퐱, 퐛1 푥1 = 퐯, 퐛1 푓 퐱 = ⋮ = 2 퐯, 퐛1 퐱, 퐛푛 ▪ Keep same output vector: ′ 퐛1 ′ 퐛 → 퐓퐛 requires 퐱 → 퐓퐱 푥2 = 퐯, 퐛ퟐ 푖 푖 ′ 퐛2 = 2 퐯, 퐛ퟐ Awesome Video Tensors, Co-/Contra-Variance ▪ „Tensors Explained Intuitively: Covariant, Contravariant, Rank“ Physics Videos by Eugene Khutoryansky https://www.youtube.com/watch?v=CliW7kSxxWU Covariance & Contravariance Linear map 퐟: 푉1 → 푉2 Matrix representation (standard basis) 퐌 ∈ ℝ푑1×푑2 Change of basis | | | | 퐁 = 퐛 1 ⋯ 퐛 1 , 퐁 = 퐛 2 ⋯ 퐛 2 1 1 푑1 2 1 푑2 | | | | New matrix representation (bases 퐁1, 퐁2) −1 푑1×푑2 퐁2 퐌퐁1 ∈ ℝ Covariance & Contravariance Situation covariant 퐟: 푉1 → 푉2 퐱 퐈 퐌 퐈 mapping M −1 퐁1 퐁2 퐌퐁1 퐁2 퐲 contravariant Transformation law ▪ Input vectors 퐱 (퐌퐱): 퐱 퐁1 = 퐁1퐱 퐈 (covariant) −1 ▪ Output vectors 퐲 = 퐌퐱: 퐲 퐁2 = 퐁2 퐲 퐈 (contravariant) Covariance & Contravariance Situation covariant 퐟: 푉1 → 푉2 퐱 퐈 퐌 퐈 mapping M −1 퐁1 퐁2 퐌퐁1 퐁2 퐲 contravariant Transformation law ▪ Input vectors 퐱 (퐌퐱): 퐱 퐁1 = 퐁1퐱 퐈 (covariant) −1 ▪ Output vectors 퐲 = 퐌퐱: 퐲 퐁2 = 퐁2 퐲 퐈 (contravariant) Covariance & Contravariance −1 푓 퐱 ← 퐁2 퐌퐁1 ← 퐱 −1 퐁2 퐌퐁1 transforms row-vectors −1 퐁2 퐌 퐁1 transforms column-vectors Scalar Product covariant 퐱 퐲 General scalar product 푑 퐱, 퐲 ∈ ℝ mapping M 퐱, 퐲 = 퐱T 퐐 퐲, T 퐐 = 퐐 , 퐐 > 0 ⟨퐱, 퐲⟩ scalar (no coordinate system) Change of basis 퐈 퐁 퐱, 퐲 = 퐱T 퐐 퐲 퐱 → 퐁퐱, 퐈 퐲 → 퐁퐲 T 퐓 퐱, 퐲 퐁 = 퐱 ⋅ 퐁 ⋅ 퐐 ⋅ 퐁 ⋅ 퐲 Three shades of dual PCA, SVD, MDS Inputs and Outputs Input (“covariant”) side of the matrix Output (“contravariant”) side of the matrix Squaring a Matrix ▪ Possibility 1: 퐀 ⋅ 퐀푇 ⋅ = ▪ Possibility 2: 퐀푇 ⋅ 퐀 ⋅ = A Story about Dual Spaces VT X U D 1 0 0 0 0 0 = 0 2 0 0 0 0 0 0 3 0 0 0 SVD 0 0 0 4 0 0 XT X U D2 UT 1² 0 0 0 0 2² 0 0 = PCA 0 0 3² 0 0 0 0 4² T X V D2 VT X 1² 0 0 0 0 0 0 2² 0 0 0 0 0 0 3² 0 0 0 = MDS 0 0 0 4² 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Tensors: Multi-Linear Maps Tensors General notion: Tensor ▪ Tensor: multi-linear form*) with 푟 ∈ ℕ input vectors 퐓: 푉1, … , 푉푛, 푉푛+1, … , 푉푟 → 퐹 (usually: field 퐹 = ℝ) ▪ “Rank 푟” tensor ▪ Linear in each input (when keeping the rest constant) ▪ Each input can be covariant or contravariant ▪ (푛, 푚) tensor ▪ 푟 = 푛 + 푚 ▪ 푛 – contravariant inputs ▪ 푚 – covariant inputs *) i.e.: multi-linear mapping / function Tensors Representation ▪ Represented as r-dimensional array 푡푖1,푖2,…,푖푛 푗1,푗2,…,푗푚 ▪ n – contravariant inputs (“indices”) ▪ m – covariant inputs (“indices”) ▪ Mapping rule 퐓 퐯 1 , … , 퐯 푛 , 퐰 1 , … , 퐰 푚 ≔ ෍ ⋯ ෍ ෍ ⋯ ෍ 푣 1 ⋯ 푣 푛 푤 1 ⋯ 푤 푚 푡푖1,푖2,…,푖푛 푖1 푖푛 푗1 푗푚 푗1,푗2,…,푗푚 푖1=0,…,푛푖1 푖푛=0,…,푛푖푛 푗1=0,…,푛푗1 푗푚=0,…,푛푗푚 (Note: writing the application of 퐓 as multi-linear mapping here) Tensors Remarks ▪ No difference between co-/contravariant dimensions in terms of numerical representation ▪ Generalization of matrix Example 푧1 푥1 푦1 퐓 , , 푧2 = 42푥1푦1푧1 + 23푥1푦1푧2 + ⋯ + 16푥2푦2푧3 푥2 푦2 푧3 ▪ Purely linear polynomial in each input parameter when all others remain constant. ▪ 3D array - 2 × 2 × 3 combinations of coefficients Einstein Notation Example: Quadratic polynomial ℝ3 → ℝ3 푝푗 퐱 = 퐱T퐀퐱 + 퐛퐱 + c 3 3 3 푗 푗 푗 푗 푝 = ෍ ෍ 푥푘푥푙 푎푘푙 + ෍ 푥푘푏푘 + c 푘=1 푙=1 푘=1 Tensor notation ▪ Input: 푥푖, 푖 = 1. 3 ▪ Quadratic form (Matrix) 퐀: 푎푘푙 푗 ▪ Output: 푝 , 푗 = 1. 3 ▪ Linear form (Co-Vector) 퐛: 푏푘 ▪ Constant 푐 Einstein Notation Example: Quadratic polynomial ℝ3 → ℝ3 푝푗 퐱 = 퐱T퐀퐱 + 퐛퐱 + c Einstein notation (implicit sums over common indices) 푗 푗 푗 푗 푝 = 푥푘푥푙푎푘푙 + 푥푘푏푘 + c Tensor notation ▪ Input: 푥푖, 푖 = 1. 3 ▪ Quadratic form (Matrix) 퐀: 푎푘푙 푗 ▪ Output: 푝 , 푗 = 1. 3 ▪ Linear form (Co-Vector) 퐛: 푏푘 ▪ Constant 푐 Further Examples Examples ▪ (푛, 푚)-tensor ▪ n contravariant “indices” ▪ m covariant “indices” ▪ Matrix: (1,1)-tensor ▪ Scalar product: (0,2)-tensor ▪ Vector: (1,0)-tensor ▪ Co-vector: (0,1)-tensor ▪ Geometric vectors: (1,0)-tensors Covariant Derivatives? Examples ▪ Geometric vectors: (1,0) tensors ▪ Derivatives*) / gradients / normal vectors: (0,1) tensors *) to be precise: ▪ Spatial derivatives co-vary for changes of the basis of the space – 푓: ℝ푛 → ℝ, 푓 퐱 = 퐲, ⇒ 훻푓 is covariant (0,1). – Examples: Gradient vector ▪ Derivatives of vector functions by unrelated dimensions remain contravariant d – 푓: ℝ → ℝ푛, 푓 푡 = 퐲, ⇒ 푓 remains contravariant (1,0). d푡 – Examples: velocity, acceleration 푛 푚 ▪ Mixed case: 푓: ℝ → ℝ , 훻푓 = 퐽푓 is a (1,1)-tensor (1,1) Example: Plane Equation Plane equation(s) ▪ Parametric: 퐱 = 휆1퐫1 + 휆2퐫2 + 퐨 ▪ Implicit: 퐧, 퐱 − 푑 = 0 Transformation 퐱 → 퐓퐱 ▪ Parametric: 퐓퐱 = 퐓 휆1퐫1 + 휆2퐫2 + 퐨 = 휆1퐓퐫1 + 휆2퐓퐫2 + 퐓퐨 ▪ Implicit: 퐧, 퐓퐱 − 푑 = 퐧T퐓 퐱 − 푑0 More Structure? Connecting ▪ Integrals ▪ Derivatives ▪ In higher dimensions ▪ And their transformation rules “Exterior Calculus” ▪ Unified framework ▪ Beyond this lecture (take a real math course :-) ) Vectors & Covectors in Function Spaces Remark: Function Spaces Discrete vector spaces ▪ Picking entries by index is a linear operation ▪ Can be represented by projection to vector (multiplication with “co-vector”) Example ▪ 퐱 = (푥1, 푥2, 푥3, 푥4, 푥5) ▪ 퐱 ↦ 푥4 is a linear maps ▪ Represented by 0,0,0,1,0 , 퐱 ▪ “Linear form”: 퐱 ↦ 0,0,0,1,0 , 퐱 , ퟎ T ퟎ in short, ⋅, 0,0,0,1,0 , shorter: 0,0,0,1,0 = ퟎ ퟏ ퟎ Linear Forms in Function Spaces In function spaces ▪ Picking entries by x-axis is a linear operation ▪ Cannot be represented by projection to another function (multiplication with “co-vector”) 푔 1, if 푥 = 4 푔 푥 = ቊ 푓 0, elsewhere Example න 푓 푥 푔 푥 푑푥 = 0 ℝ ▪ 푓: ℝ → ℝ, 푓 푥 = sin 푥 ▪ 퐿: ℝ → ℝ → ℝ, 퐿: 푓 ↦ 푓 4.0 is a linear map ▪ A function 푔 with 푔, 푓 = 푓 4.0 does not exist Dirac’s “Delta Function” f(x) f(x) x x f(x) f(x) න 푓 푥 푑푥 = 1 x x Ω Dirac Delta “Function” ▪ ׬ℝ 훿 푥 푑푥 = 1, zero everywhere but at 푥 = 0 ▪ Idealization (“distribution”) – think of very sharp peak Distributions Distributions ▪ Adding all linear forms to the vector space ▪ All linear mappings from the function space to ℝ ▪ This makes the situation symmetric ▪ 훿 is a distribution, not a (traditional) function Formalization ▪ Different approaches (details beyond our course) ▪ Limits of “bumps” ▪ Space of linear-forms (“co-vectors”, “dual functions”) ▪ Difference of complex functions on Riemann sphere (exotic).