Surrogate minimisation in high dimensions Fabrizia Guglielmetti (ESO) In collaboration with: Torsten Enßlin (MPA), Henrik Junklewitz (AIfA), Theo Steininger (MPA) Bayes Forum Seminar - 24.02.2017 (1) Outline ❖ Motivation! • Performance enhancement of optimisation schemes ! ❖ Global optimisation as a Bayesian decision problem! • Two connected statistical layers! ❖ Surrogates basic principles ! ❖ Application of Kriging Surrogate within NIFTY (Selig, M. et al., 2013, A&A, 554, 26) framework Bayes Forum Seminar - 24.02.2017 (2) Introduction ! ❖ Complex computer codes are essentials! ! ! ! ! ! ! Bayes Forum Seminar - 24.02.2017 (3) Introduction A 48-hour computer simulation of Typhoon Mawar using the Weather Research and Forecasting model! credit Wikipedia Bayes Forum Seminar - 24.02.2017 (3) Introduction See, e.g., Cox D.D, Park J.S., Singer C.E. (2001) Preuss, R. & von Toussaint, U. (2016) Bayes Forum Seminar - 24.02.2017 (3) Introduction Bayes Forum Seminar - 24.02.2017 (3) Introduction Bayes Forum Seminar - 24.02.2017 (3) Introduction Bayes Forum Seminar - 24.02.2017 (3) Introduction Bayes Forum Seminar - 24.02.2017 (3) Introduction ! ❖ Complex computer codes are essentials! ! ! Computer ! x Code z(x) ! Simulator ! ! Bayes Forum Seminar - 24.02.2017 (4) Introduction ! ❖ Complex computer codes are essentials:! ! ! Computer ! x Code z(x) ! ! Simulator Minimisation! of Complex Objective function. For reasonable data size, current estimation techniques:! ! a. SLOW if full objective function is accounted! b. FAST if do not account for full objective function! Bayes Forum Seminar - 24.02.2017 (4) Introduction ! ❖ Complex computer codes are essentials:! ! ! Computer ! x Code z(x) ! ! Emulator ! • statistical representation of x ! ! • expresses knowledge about z(x) at any given x! • built using prior information and training set of model runs Bayes Forum Seminar - 24.02.2017 (4) Introduction ! ❖ Complex computer codes are essentials:! ! ! Computer ! x Code z(x) ! Emulator ! Kriging surrogate sampler is used to emulate the original Complex ! Objective function Bayes Forum Seminar - 24.02.2017 (4) Surrogates (or metamodels, emulators, …) Bayes Forum Seminar - 24.02.2017 (5) Surrogates (or metamodels, emulators, …) • A map is a surrogate that predicts terrain elevation • Prediction of the map allows one to locate the summit without climbing it • Searching for the highest peak is a form of global optimisation Bayes Forum Seminar - 24.02.2017 (5) Surrogates (or metamodels, emulators, …) … in the same fashion we can use metamodels to find the optimal signal configuration Bayes Forum Seminar - 24.02.2017 (5) Bayesian inference, parametric model ❖ Data: x, y! ❖ Model: !y = fk(x)+✏ ! 1 Gaussian likelihood:! p(y x, k,M ) exp (y f (x ))2/σ2 | i / − 2 j − k j j ! Y Prior over the param:!p(k M ) | i ! p(y x, k,Mi)p(k Mi) Posterior param distr: p! (k x, y,Mi)= | | | p(y x,Mi) ! | Make predictions: ! p(y⇤ x⇤, x, y,M )= p(y⇤ k,x⇤,M )p(k x, y,M )dk | i | i | i Bayes Forum Seminar - 24.02.2017 Z (6) Bayesian inference, parametric model ❖ Data: x, y! ❖ Model: !y = fk(x)+✏ ! 1 Gaussian likelihood:! p(y x, k,M ) exp (y f (x ))2/σ2 | i / − 2 j − k j j ! Y Prior over the param:!p(k M ) | i ! p(y x, k,Mi)p(k Mi) Posterior param distr: p! (k x, y,Mi)= | | | p(y x,Mi) ! | Make predictions: ! p(y⇤ x⇤, x, y,M )= p(y⇤ k,x⇤,M )p(k x, y,M )dk | i | i | i Bayes Forum Seminar - 24.02.2017 Z (6) Bayesian inference, parametric model ❖ Data: x, y! ❖ Model: !y = fk(x)+✏ ! 1 Gaussian likelihood:! p(y x, k,M ) exp (y f (x ))2/σ2 | i / − 2 j − k j j ! Y Prior over the param:!p(k M ) | i ! p(y x, k,Mi)p(k Mi) Posterior param distr: p! (k x, y,Mi)= | | | p(y x,Mi) ! | Make predictions: ! p(y⇤ x⇤, x, y,M )= p(y⇤ k,x⇤,M )p(k x, y,M )dk | i | i | i Bayes Forum Seminar - 24.02.2017 Z (6) IFT for signal inference ❖ Information Field Theory (Enßlin, T. et al. 2009), information theory for fields! ❖ Signal field (s) estimation: ! ! H(d,s) P (d s)P (s) e− ! P (s d)= | | P (d) ⌘ Zd ! ! ! Bayes Forum Seminar - 24.02.2017 (7) IFT for signal inference ❖ Information Field Theory (Enßlin, T. et al. 2009), information theory for fields! ❖ Signal field (s) estimation: ! Information Hamiltonian ! H(d,s) P (d s)P (s) e− ! P (s d)= | | P (d) ⌘ Zd ! ! ! Bayes Forum Seminar - 24.02.2017 (7) IFT for signal inference ❖ Information Field Theory (Enßlin, T. et al. 2009), information theory for fields! ❖ Signal field (s) estimation: ! Information Hamiltonian ! H(d,s) P (d s)P (s) e− ! P (s d)= | | P (d) ⌘ Zd ! ! Partition function ! Bayes Forum Seminar - 24.02.2017 (7) IFT for signal inference ❖ Information Field Theory (Enßlin, T. et al. 2009), information theory for fields! ❖ Signal field (s) estimation: ! ! H(d,s) P (d s)P (s) e− ! P (s d)= | | P (d) ⌘ Zd ! ! T d =(d1,d2,...,dn) n N finite dataset 2 ! IFT exploits known or inferred correlation structures of the field of interest s = s ( x ) over some domain ⌦ = x to regularise the ill- posed inverse problem of determining an { } number of dof from a finite dataset 1 Bayes Forum Seminar - 24.02.2017 (7) IFT for signal inference ❖ Information Field Theory (Enßlin, T. et al. 2009), information theory for fields! ❖ Signal field (s) estimation: ! ! H(d,s) P (d s)P (s) e− ! P (s d)= | | P (d) ⌘ Zd ! ! most probable signal configuration ! Bayes Forum Seminar - 24.02.2017 (7) IFT for signal inference ❖ Information Field Theory (Enßlin, T. et al. 2009), information theory for fields! ❖ Signal field (s) estimation: ! ! H(d,s) P (d s)P (s) e− ! P (s d)= | | P (d) ⌘ Zd ! ! most probable signal configuration ! s = argmin s d H(d, s) h i h | i Bayes Forum Seminar - 24.02.2017 (7) Minimization of (complex) energy function Expensive computer code Bayes Forum Seminar - 24.02.2017 (8) Minimization of (complex) energy function Expensive computer code Optimizer Bayes Forum Seminar - 24.02.2017 (8) Minimization of (complex) energy function Expensive computer code Optimizer Find optimal image reconstruction Bayes Forum Seminar - 24.02.2017 (8) Minimization of (complex) energy function Expensive computer code Response Optimizer/ Surface Adjoint simulation Model (RSM) Find optimal image reconstruction Bayes Forum Seminar - 24.02.2017 (8) Minimization of (complex) energy function Expensive computer code Used to increase the speed Response Optimizer/ Surface Adjoint simulation Model (RSM) Find optimal image reconstruction Bayes Forum Seminar - 24.02.2017 (8) Minimization of (complex) energy function Expensive computer code Used to increase the speed Response Optimizer/ Surface Adjoint simulation Model (RSM) Cheap-to-run surrogate model of the original model function Find optimal image is used and combined with reconstruction the full objective function! ! (same model is used throughout the optimisation process) Bayes Forum Seminar - 24.02.2017 (8) Minimization of (complex) energy function Expensive computer code Response Optimizer/ Surface Adjoint simulation Model (RSM) Kriging (Wiener, 1938; Krige, 1951; Matheron, 1962; Sacks et al, 1989; Find optimal image Cressie, 1989, 1993; Jones et al, 1989; reconstruction Kleijnen, 2009; Razavi et al; 2012) Bayes Forum Seminar - 24.02.2017 (8) Minimization of (complex) energy function Expensive computer code Response Optimizer/ Surface Adjoint simulation Model (RSM) Kriging (Kitanidis, 1986; Handcock & Stein, 1993; Mira & Sanchez,2002; More & Halvorsen, 1989; Morris, 1993; KennedyFind & optimal image O’Hagan, 2000; Bayarri & Berger, 2007; reconstruction Preuss et al. 2016) Bayes Forum Seminar - 24.02.2017 (8) Minimization of (complex) energy function Expensive computer code Partially converged simulations Response Optimizer/ Surface Adjoint simulation Model (RSM) Find optimal image reconstruction Bayes Forum Seminar - 24.02.2017 (8) Minimization of (complex) energy function Expensive computer code Partially converged simulations Response Optimizer/ Position of individual surface Surface Adjoint simulation points are moved in the direction Model (RSM) of the gradients predicted by the adjoint simulation Find optimal image reconstruction Bayes Forum Seminar - 24.02.2017 (8) Minimization of (complex) energy function Expensive computer code Partially converged simulations Response Optimizer/ Position of individual surface Surface Adjoint simulation points are moved in the direction Model (RSM) of the gradients predicted by the adjoint simulation Gradients take correct direction early in Find optimal image convergence reconstruction Bayes Forum Seminar - 24.02.2017 (8) Minimization of (complex) energy function Expensive computer code Partially converged simulations Response Optimizer/ Position of individual surface Surface Adjoint simulation points are moved in the direction Model (RSM) of the gradients predicted by the adjoint simulation Proper shape of RSM is provided Find optimal image reconstruction Bayes Forum Seminar - 24.02.2017 (8) Minimization of (complex) energy function Expensive computer code Partially converged simulations Response Optimizer/ Position of individual surface Surface Adjoint simulation points are moved in the direction Model (RSM) of the gradients predicted by the adjoint simulation Find optimal image reconstruction Max shape control in image variations afforded to the optimiser for a minimum number of Bayes Forum Seminar - 24.02.2017 variables Minimization of (complex) energy function Expensive computer code Partially converged simulations Response Optimizer/ Position of individual surface Surface Adjoint simulation points are moved in the direction Model (RSM) of the gradients predicted by the adjoint simulation Find