Exercise 6: Orthogonal Functions, Radial Functions (RBF) and Cross-Validation (CV)

CSElab Computational Science & Engineering Laboratory Outline

1. Goals 2. Theory/ Examples 3. Questions Goals

⚫ Functions

⚫ How to construct an orthonormal basis ?

⚫ Benefit of using an orthonormal basis?

⚫ Radial Basis Functions (RBF)

⚫ Cross Validation (CV) Motivation: Orthonormal Basis Functions

⚫ Computation of coefficients can be compute intensive

⚫ Adding or removing basis functions normally requires a re-computation of the coefficients

⚫ Overcome by orthonormal basis functions Gram-Schmidt Orthonormalization

푛 ⚫ Given: A set of vectors 푣1, … , 푣푘 ⊂ 푅 푛 ⚫ Goal: Generate a set of vectors 푢1, … , 푢푘 ⊂ 푅 such that 푢1, … , 푢푘 is orthonormal and spans the same subspace as 푣1, … , 푣푘

⚫ Click here if the animation is not playing Recap: Projection of Vectors

⚫ Notation: Denote dot product as 푎Ԧ ⋅ 푏 = 푎Ԧ, 푏 1 (bilinear form). This implies the norm 푢 = 푎Ԧ, 푎Ԧ 2

⚫ Define the scalar projection of v onto u as 푃푢 푣 = 푢,푣 | 푢 |

푢,푣 푢 ⚫ Hence, is the part of v pointing in direction of u | 푢 | | 푢 | 푢 푢 and 푣∗ = 푣 − , 푣 is orthogonal to 푢 푢 | 푢 | Gram-Schmidt for Vectors

푛 ⚫ Given: A set of vectors 푣1, … , 푣푘 ⊂ 푅 푣1 푢1 = | 푣1 |

푣2,푢1 푢1 푢2 = 푣2 − = 푣2 − 푣2, 푢1 푢1 as 푢1 normalized 푢1 푢1

푢෤3 = 푣3 − 푣3, 푢1 푢1 What’s missing? Gram-Schmidt for Vectors

푛 ⚫ Given: A set of vectors 푣1, … , 푣푘 ⊂ 푅 푣1 푢1 = | 푣1 |

푣2,푢1 푢1 푢2 = 푣2 − = 푣2 − 푣2, 푢1 푢1 as 푢1 normalized 푢1 푢1

푢෤3 = 푣3 − 푣3, 푢1 푢1 What’s missing?

푢෤3is orthogonal to 푢1, but not yet to 푢2

푢3 = 푢෤3 − 푢෤3, 푢2 푢2 If the animation is not playing, please click here Gram-Schmidt for Functions

⚫ Given: A set of functions 푔1, … , 푔푘 .We want 휙1, … , 휙푘

⚫ Bridge to Gram-Schmidt for vectors: ∞ ⚫ Define 푔푖, 푔푗 = ׬−∞ 푔푖 푥 푔푗 푥 푑푥

⚫ What’s the corresponding norm? Gram-Schmidt for Functions

⚫ Given: A set of functions 푔1, … , 푔푘 .We want 휙1, … , 휙푘

⚫ Bridge to Gram-Schmidt for vectors: ∞ ⚫ 푔 , 푔 = 푔 푥 푔 푥 푑푥 . Define 푖 푗 ׬−∞ 푖 푗 What is the 1 ∞ 1 2 2 corresponding norm? : gi = 푔푖,푔푖 = ׬−∞푔푖 푥 푔푖 푥 푑푥

⚫ property: 푔푖, 푔푗 = 훿푖,푗

푔1 휙1 = | 푔1 |

푢2 = 푣2 − 푣2, 푢1 푢1 becomes

휙2 = 푔2 − 푔2, 휙1 휙1

휙෨3 = 푔3 − 푔3, 휙1 휙1, 휙3 = 휙෨3 − 휙෨3, 휙2 휙2 Idea: Interpolation

Target :

Weighted Inner Product:

Coefficients:

푁 1 푝 푥 ≈ ෍ 훿(푥 − 푥 ) Sampling data: 푁 0 푛=1

Recap: ׬ 푓 푥 훿 푥 − 푥푛 푑푥 = 푓 푥푛

Approximation: Radial Basis Functions (RBF)

Example: 1D Gaussian Radial Basis functions

- The characteristic of RBFs is that it only depends on the distance to a set of centers and some Parameter ( in this case: centers = µ, parameter=σ)

Question: How would you define 2D? 3D? … Cross-Validation

• Problems with interpolation • Overfitting: fit data exactly = fit noise? • Predictability: fit well around data = fit well everywhere else?

• Deciding which type of interpolation to use: • Estimate predictability with given data set: Cross-Validation • Basic metric: mean squared error K-Fold Cross-Validation

1. Split training data into k disjoint samples of roughly equal size (N/K) yielding K folds Z1 to Zk 2. For each Zi 1. Use remaining folds to interpolate f 2. For interpolation f, evaluate for each Zi

3. Average ri over all folds Image source: http://nbviewer.jupyter.org/github/rasbt/python-machine-learning- book/blob/master/code/ch06/ch06.ipynb#K-fold-cross-validation Question 1

• (a) Find an orthonormalized set of 2 • Start with g1(x) = 1, g2(x) = x, g3(x) = x

• (b,c) Perform interpolation • Find coefficients • What happens when you add a basis function? Question 2

• Basis functions: 2D Gaussian functions

• Centers: at each data point • Free parameter: σx and σy • Feature: non-uniform data • Python template provided Question 3

Leave-one-out: K-fold cross validation with K = N (number of data points) • Imagine given an interpolation function • input data points and vector x • output the final value based on x using a predefined interpolation model Write pseudocode for leave-one-out cross validation