Exercise 6: Orthogonal Functions, Radial Basis Functions (RBF) and Cross-Validation (CV) CSElab Computational Science & Engineering Laboratory Outline 1. Goals 2. Theory/ Examples 3. Questions Goals ⚫ Orthonormal Basis Functions ⚫ How to construct an orthonormal basis ? ⚫ Benefit of using an orthonormal basis? ⚫ Radial Basis Functions (RBF) ⚫ Cross Validation (CV) Motivation: Orthonormal Basis Functions ⚫ Computation of interpolation coefficients can be compute intensive ⚫ Adding or removing basis functions normally requires a re-computation of the coefficients ⚫ Overcome by orthonormal basis functions Gram-Schmidt Orthonormalization 푛 ⚫ Given: A set of vectors 푣1, … , 푣푘 ⊂ 푅 푛 ⚫ Goal: Generate a set of vectors 푢1, … , 푢푘 ⊂ 푅 such that 푢1, … , 푢푘 is orthonormal and spans the same subspace as 푣1, … , 푣푘 ⚫ Click here if the animation is not playing Recap: Projection of Vectors ⚫ Notation: Denote dot product as 푎Ԧ ⋅ 푏 = 푎Ԧ, 푏 1 (bilinear form). This implies the norm 푢 = 푎Ԧ, 푎Ԧ 2 ⚫ Define the scalar projection of v onto u as 푃푢 푣 = 푢,푣 | 푢 | 푢,푣 푢 ⚫ Hence, is the part of v pointing in direction of u | 푢 | | 푢 | 푢 푢 and 푣∗ = 푣 − , 푣 is orthogonal to 푢 푢 | 푢 | Gram-Schmidt for Vectors 푛 ⚫ Given: A set of vectors 푣1, … , 푣푘 ⊂ 푅 푣1 푢1 = | 푣1 | 푣2,푢1 푢1 푢2 = 푣2 − = 푣2 − 푣2, 푢1 푢1 as 푢1 normalized 푢1 푢1 푢෤3 = 푣3 − 푣3, 푢1 푢1 What’s missing? Gram-Schmidt for Vectors 푛 ⚫ Given: A set of vectors 푣1, … , 푣푘 ⊂ 푅 푣1 푢1 = | 푣1 | 푣2,푢1 푢1 푢2 = 푣2 − = 푣2 − 푣2, 푢1 푢1 as 푢1 normalized 푢1 푢1 푢෤3 = 푣3 − 푣3, 푢1 푢1 What’s missing? 푢෤3is orthogonal to 푢1, but not yet to 푢2 푢3 = 푢෤3 − 푢෤3, 푢2 푢2 If the animation is not playing, please click here Gram-Schmidt for Functions ⚫ Given: A set of functions 푔1, … , 푔푘 .We want 휙1, … , 휙푘 ⚫ Bridge to Gram-Schmidt for vectors: ∞ ⚫ Define 푔푖, 푔푗 = ׬−∞ 푔푖 푥 푔푗 푥 푑푥 ⚫ What’s the corresponding norm? Gram-Schmidt for Functions ⚫ Given: A set of functions 푔1, … , 푔푘 .We want 휙1, … , 휙푘 ⚫ Bridge to Gram-Schmidt for vectors: ∞ ⚫ 푔 , 푔 = 푔 푥 푔 푥 푑푥 . Define 푖 푗 ׬−∞ 푖 푗 What is the 1 ∞ 1 2 2 corresponding norm? : gi = 푔푖,푔푖 = ׬−∞푔푖 푥 푔푖 푥 푑푥 ⚫ Orthogonality property: 푔푖, 푔푗 = 훿푖,푗 푔1 휙1 = | 푔1 | 푢2 = 푣2 − 푣2, 푢1 푢1 becomes 휙2 = 푔2 − 푔2, 휙1 휙1 휙෨3 = 푔3 − 푔3, 휙1 휙1, 휙3 = 휙෨3 − 휙෨3, 휙2 휙2 Idea: Interpolation Target function: Weighted Inner Product: Coefficients: 푁 1 푝 푥 ≈ ෍ 훿(푥 − 푥 ) Sampling data: 푁 0 푛=1 Recap: ׬ 푓 푥 훿 푥 − 푥푛 푑푥 = 푓 푥푛 Approximation: Radial Basis Functions (RBF) Example: 1D Gaussian Radial Basis functions - The characteristic of RBFs is that it only depends on the distance to a set of centers and some Parameter ( in this case: centers = µ, parameter=σ) Question: How would you define 2D? 3D? … Cross-Validation • Problems with interpolation • Overfitting: fit data exactly = fit noise? • Predictability: fit well around data = fit well everywhere else? • Deciding which type of interpolation to use: • Estimate predictability with given data set: Cross-Validation • Basic metric: mean squared error K-Fold Cross-Validation 1. Split training data into k disjoint samples of roughly equal size (N/K) yielding K folds Z1 to Zk 2. For each Zi 1. Use remaining folds to interpolate f 2. For interpolation f, evaluate for each Zi 3. Average ri over all folds Image source: http://nbviewer.jupyter.org/github/rasbt/python-machine-learning- book/blob/master/code/ch06/ch06.ipynb#K-fold-cross-validation Question 1 • (a) Find an orthonormalized set of polynomials 2 • Start with g1(x) = 1, g2(x) = x, g3(x) = x • (b,c) Perform interpolation • Find coefficients • What happens when you add a basis function? Question 2 • Basis functions: 2D Gaussian functions • Centers: at each data point • Free parameter: σx and σy • Feature: non-uniform data • Python template provided Question 3 Leave-one-out: K-fold cross validation with K = N (number of data points) • Imagine given an interpolation function • input data points and vector x • output the final value based on x using a predefined interpolation model Write pseudocode for leave-one-out cross validation.