> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by synthesis Marcel Lüthi, 03. Juni 2019 Graphics and Vision Research Group Department of Mathematics and Computer Science University of Basel University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by synthesis Comparison Parameters 휃 Update 휃 Synthesis 휑(휃) Being able to synthesize data means we can understand how it was formed. − Allows reasoning about unseen parts. University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by synthesis Comparison Parameters 휃 Update 휃 Synthesis 휑(휃) University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by synthesis – Computer vision Comparison Parameters 휃 Update 휃 Synthesis 휑(휃) Computer graphics University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Mathematical Framework: Bayesian inference Comparison: 푝 data 휃) Prior 푝(휃) Parameters 휃 Update using 푝(휃|data) Synthesis 휑(휃) • Principled way of dealing with uncertainty. University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Algorithmic implementation: MCMC Comparison: 푝 data 휃) Prior 푝(휃) Parameters 휃 Sample from 푝(휃|data) Synthesis 휑(휃) Posterior distribution over parameters University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE The course in context Pattern Theory Text Music Ulf Grenander Computational Research at anatomy Gravis Medical Images Fotos Speech Natural language This course University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Pattern theory – The mathematics University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Pattern theory vs PMM • Pattern theory is about developing a theory for understanding real- world signals • Probabilistic Morphable Models are about using theoretical well founded concepts to analyse images. • GPs for modelling • MCMC for model fitting • Working software University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by Synthesis in 5 (simple) steps University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by synthesis in 5 simple steps 1. Define a parametric model • a representation of the world 휃푛 • State of the world is determined by parameters 휃 = (휃1, … , 휃푛) … 휃1 11 University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by synthesis in 5 simple steps 2. Define a synthesis function 휑 휃1, … , 휃푛 • generates/synthesize the data given the “state of the world” • 휑 can be deterministic or stochastic 휃푛 휑(휃1, … , 휃푛) … 휃1 12 University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis-by-Synthesis in 5 simple steps 3. Define likelihood function: • Define a probabilistic model 푝 data 휃1, … , 휃푛 that models how the synthesized data compares to the real data • Includes stochastic factors on the data, such as noise Comparison 푝(data|휃1, … , 휃푛) 휃푛 휑(휃1, … , 휃푛) … 휃1 University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Bayesian inference We have: 푃 푑푎푡푎|휃1, … , 휃푛 We want: 푃 휃1, … , 휃푛|푑푎푡푎 Bayes rule: 푃 퐷|휃 푃 휃 푃 휃|퐷 = 푃 퐷 Lets us compute from 푝 퐷 휃 its “inverse” 푝(휃|퐷) University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by synthesis in 5 simple steps 4. Define prior distribution: 푝 휃 = 푝(휃1, … , 휃푛) • Our believe about the “state of the world” • Makes it possible to invert mapping 푝(data|휃1, … , 휃푛) Comparison 푝(image|휃1, … , 휃푛) 휃푛 푝 휃 휑(휃1, … , 휃푛) … 휃1 University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by synthesis in 5 simple steps 5. Do inference 푝 휃 , … , 휃 푝 data 휃 , … , 휃 푝 휃 , … , 휃 data = 1 푛 1 푛 1 푛 푝 data Comparison 푝(image|휑(휃1, … , 휃푛) Purely conceptual formulation: 휃푛 • Independent of algorithmic 휑(휃 , … , 휃 ) implementation 1 푛 • … But usually done iteratively 휃1 Parameters 휃 Update using 푝(휃|Image) Synthesis 휑(휃) 16 University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by synthesis in 5 simple steps 5. Possibility 1: Find best (most likely) solution: 푝 휃1, … , 휃푛 푝 data 휃1, … , 휃푛 arg max 푝 휃1, … , 휃푛 data = arg max 휃1,…,휃푛 휃1,…,휃푛 푝 data MAP Solution Local Most popular approach Maxima • Usually based on gradient- descent • May miss good solutions 17 University of Basel > DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE Analysis by synthesis in 5 simple steps 5. Possibility 2: Find posterior distribution: 푝 휃 , … , 휃 푝 data 휃 , … , 휃 푝 휃 , … , 휃 data = 1 푛 1 푛 1 푛 푝 data Core of this course • Obtain samples from the distribution • Based on Markov Chain Monte Carlo methods 18.