Supervised Learning: The Setup Laplace approximation Fall 2019 Instructor: Shandian Zhe Machine [email protected] SchoolFall 2017 of Computing 1 1 Outline • Laplace approximation • Bayesian logistic regression 2 Outline • Laplace approximation • Bayesian logistic regression 3 Laplace approximation • Objective: construct a Gaussian distribution to approximate the target distribution • Method: second order Taylor expansion at the posterior mode (i.e., MAP estimation) 4 648 p(x)= p xi pa(xi) 649 | i 648 650 pY(x)= p xi pa( xi) 649 q(x ) exp [log|p(x)] 651 j Eq(x ij ) 650 / { Y¬ } x 652 q(xj) expj Eq(x j )[log p(x)] 651 / { ¬ } 653648 652 p(x)= p xi pa(xi) xj 654649 q(xj) exp E log p(x|j pa(xj)) 653 / { i | 650 q(x ) Yexp log p(x pa(x )) 655 654 q(x ) j exp E [log p(jx)] j 651 Laplace approximation+j /⇥ Eq{E(x jlog) p(xt| x⇤j, pa(xt) xj ) (5) 656 655 / +{ ¬ log| p}(x x , pa(\{x ) }x }) (5) xj pa(xt)x ⇥ E t ⇤j t j 652 656 2X j | \{ } } 657 xj ⇥pa(xt) ⇤ 653 657 2X ⇥ ⇤ 658 • Given a joint probability p(✓, ) 654 658 q(xj) exp E log p(xj pa(xj))- Dp(✓, ) 659 / { | D 655 659 p(✓ ) • How to+ compute⇥ (approximate)E log p(xt x|D⇤j , pa( p ( ✓ x t ) ) ?x j ) (5) 660656 660 | |D \{ } } xj pa(xt) 661657 661 2X ⇥ ⇤ Let us do MAP estimation first 662658 p(✓, ) 662 D 663659 663 p(✓ ) 660 664 |D 664 ✓0 = argmax log p(✓, ) = argmax log p( ✓) 665661 665 ✓ D ✓ D| 662 666 666 f(✓) , log p(✓, ) 663 667 D 667 664 668 668 665 669 5 669666 670 670667 671 671668 672 672669 673 674 673670 675 674671 672 676 675 673 677 676 674 678 677 675 679 680 678676 681 679677 682 680678 683 681679 684 682680 685 681 683 686 682 684 687 683 685 688 684 686 689 685 690 687686 691 688687 692 689688 693 690689 694 691690 695 692691 696 693692 697 693 694 698 694 695 699 695 700 696 696 701 697697 698698 699699 13 700700 701701 13 13 648 648 p(x)= p xi pa(xi) 649 648648 p(x)= p xi| pa(xi) 649 i | pp((xx)=)= pp xxiipa(pa(xxii)) 650 649649 Yi || 650 Y ii 650 YY 648 650 q(xqj()x ) expexpEq(x j )[log[log pp((xx)])] 651 651 648 p(x)=j p xi Epa(q(x¬xij)) / / { { ¬ qqp(((xxxjj)=))}} expexpp EExqqi((xpa(x jj))x[log[logi) pp((xx)])] 649 651651 | // {{ ¬¬ }} 652 649 i xxj j | 652 652652 i xxjj 650 650 Y Y 653 653 653653 q(xj) exp q(x )[log p(x)]q(x ) exp [log p(x)] 651 651 E j j Eq(x j ) q(x ) exp/ log{ p(¬x pa(x )) } / { ¬ } 654 q(x654654j) jexp Elog p(xj jpa(qq((xxxjjjj)))) expexp EE loglogpp((xxjj pa(pa(xxjj)))) 654 652 652 / E{ xj | // {{ xj || 655 655655 / { | 655 653 653 + ⇥ E log p(xt x⇤j+,+pa(x⇥t⇥) xEEj loglog) pp((xxttxx⇤j⇤j,,pa(pa(xxtt)) xxjj )(5)) (5)(5) 656 + log p(xt xj, pa(xt) xj ) | \{ } } (5) 656 656 ⇥ E q(xj) |exp⇤ x logpa(px\{(x) j pa(} x}j)) | \{ } } 654 q654(xj) exp E logxj ppa((xxjt)pa(xj)) Exjj pa(xtt) 656 657 2X /| { 22XX \{|⇥ } } ⇤ 657 655657/ {xj pa(xt) | ⇥ ⇥ ⇤ ⇤ 657 655 658 2X ⇥ + ⇥ E logp(p✓(,x⇤t )x⇤j, pa(xt) xj ) (5) 658 Laplace656658 + approximation⇥ E log p(px(✓t ,x⇤j,)pa(xt) xj ) p(✓,D|) \{(5) } } 656 659 | D xj \{pa(xt}) } D 658 659 657659 xj pa(xt) p(✓, ) 2X ⇥ pp((✓✓ )) ⇤ 657 2X ⇥ p(✓ ) ⇤ |D 660660 D|D |D 659 660 658 ✓✓0 == argmax argmax loglogppp((✓(✓✓,,, ))) = = argmax argmax loglogpp(( ✓✓)) 658 661✓0 = argmax plog(✓,pp((✓)✓, )) =0 argmax✓ log p( ✓DD) ✓ D| 661 •659We661 then expand theDlog|Djoint probability✓ p(at✓ theD) ✓ D| 660 662 ✓ D ✓ D| 659 662 p(✓ ) |D 662 660✓0 = argmax log p(✓, ) = argmax logff((✓✓p))(, loglog✓)pp((✓✓,, )) 661 660 posterior663 mode f(✓|D) Dlog✓0 p=(✓ argmax, ) log p(✓, D|,) = argmaxD log p( ✓) 661663 ✓ , ✓✓ D ✓ D D| 663 ✓0664= argmax log p(✓, ) = argmax Dlog p( ✓) 662 661 662664 ✓ D ✓ f(✓)D|f(✓0)+ f(✓0)>(✓ ✓0) 664 665 f(✓) log p(✓, f)(✓)f⇡(✓f)(,✓0log)+pr(✓f,(✓0))>(✓− ✓0) 662 663665 f(✓) f(✓,0)+ f(✓0)>(✓1 ⇡✓0) r D − 663 f(✓) log p(✓, ) - D 665 666 ⇡, r + 1−(✓ ✓0)> f(✓0)(✓ ✓0) 663 664666 1 D + 2 (✓ ✓0)> f(✓0)(✓ ✓0) 664 667 f(✓) f(✓−0)+ rrf(✓0)>(✓ −✓0) 666 + (✓ ✓0)> f(✓0)(✓ ⇡2✓0)− rrr − − 664 665667 f(✓) 2 f(✓0)+ f(✓0)>(✓ ✓0) 665 667 668 f(✓) f⇡(✓0)+− f(✓rrr0)>(✓ ✓10)−− f(✓0)=0 665 666668 ⇡ r −+ (✓ ✓0)>r f(✓f0(✓)=0)(0✓ ✓0) 669 1 1 2 − rrrf(✓ ) 0 − 666 668 667669 f(✓0)=0 0 666 670 ++(✓(✓✓0)>✓0)r>f(✓0)(f✓(✓0)(✓0✓) ✓0) rr f(✓0)≺ 0 Why? 669 668670 2 2 − − rr rr − − frr(✓0)=- 0 ≺ 667 667 671 f(✓0) 0 r 670 669671 rr ≺ f(✓ ) 0 668 668 672 1 rr 0 ≺ 670 = f(✓ ) (✓ ✓ )>A(✓ ✓ ) 669 671 672673 0 0 0 1 669 − 2 − = f(✓−) (✓ ✓ )>A(✓ ✓ ) 671673 0 0 0 670 672 674 − 2 − − 670 672 A = f(✓0) 673 674675 rr A = f(✓0) 0 671 673 rr 671 674 675676 6 672 674 676677 672 673 675 675 677678 673 674 676 676 678679 675 677 677 674 679680 676 678 678 675 680681 679 679 677 681682 676 680 680 678 682683 677 681 679 681 683684 682 678 680 682 685 683684 683 686 679 681 684685 684 687 680 682 685686 688 683 685 686687 681 689 684 686 687688 682 690 685 687 688689 691 683 686 688 689690 692 684 689 690691 687 693 691692 685 688 690 694 692693 686 689 691 695 693694 690 692 696 687 694695 691 693 697 688 695696 692 694 698 696697 689 695 699 693 697698 690 700 694 696 698699 701 691 695 697 699700 692 696 698 700701 697 699 701 693 13 698 700 694 699 701 13 695 700 13 696 701 697 13 698 699 13 700 701 13 648 p(x)= p x pa(x ) p(x)= p xi pa(i xi) i 649 | | i i 650 YY q(xj) exp q(x )[log p(x)] 651 q(xj) exp EqE(x j )[logj p(x)] 651 / / { { ¬ ¬ } } 652 xj xj 648 653 648 p(x)= p xi pa(xi) 649 654 q(qx(jx)jp)(xexp)=expE Elogplogpx(xipj(pa(xpa(j| xpa(xij)))xj)) 649 // { { i | | | 650 655 i Y 650 q(x++) Y⇥exp⇥ E Eloglogp[log(xpt(xpx⇤(tjx,xpa()]⇤j, pa(xt)xt)xj x) j ) (5) (5) 651 656 j Eq(x j ) | | \{ \{} }} } q(x ) exp/x pa({x ) [log¬ p(x)] } 651 648j xj jpa(Extq)(xt j ) 652 657 / 2X2{X x¬⇥j ⇥ } p(x)= ⇤ p x⇤i pa(xi) 657 649 | 652 xj i 653 658 650 p(✓p,(✓,) ) Y D D q(x ) exp [log p(x)] 653 654 659 q(xj) 651exp E log p(xj pa(xj)) j Eq(x j ) / { |p(✓p(✓) ) / { ¬ } 654 655 660 q(xj) exp652E log p(xj pa(xj))|D|D xj + log p(xt xj, pa(xt) xj ) (5) / ✓0✓653{0== argmax argmax⇥ logE|logp(✓p,(✓,) =)⇤ argmax = argmaxloglogp( p(✓) ✓) 655 656 661 ✓ D D| ✓ \{ }D|}D| xj pa(✓xt) q(x ) ✓exp log p(x pa(x )) + 654 ⇥ 2XE log p(xt x⇤j, pa(j xt) xEj ) j j (5) 657 662 ⇥ / { ⇤ | 656 655 f(f✓()✓) log| logp(✓p,(✓,) \{) } } 658 xj pa(xt) p(✓,, ,) + ⇥ E log p(xt x⇤j, pa(xt) xj ) (5) 657 663 6562X ⇥ D D ⇤ | \{ } } D xj pa(xt) 659 664 657 p(✓ ) 2X ⇥ ⇤ 658 f(f✓()✓)p(f✓(f✓, (0✓)+)0|D)+f(✓f0()✓>0()✓>(✓✓0)✓0) 660 665 658 ⇡ D r − p(✓, ) 659 665 ✓0 = argmax log⇡p(✓, ) = argmaxr log p−( ✓) D 661 659 1 1 p(✓ D) D| p(✓ ) 666 ✓ ++ (✓(✓ ✓0✓)>) f(✓f✓0()(✓✓)(✓✓0)✓ ) 660 660 2 − |D 0 rr> 0 − 0 |D 662 667 ✓ = argmaxLaplacelog p 2approximation(✓, −) = argmaxrr ✓0 =log argmaxp(− ✓log) p(✓, ) = argmax log p( ✓) 667 0 661 f(✓) , log p(✓, ) D D| 661 663 ✓ D ✓D ✓ D| ✓ 668 662 f(✓f0()=✓0)=0 0 662 664 r r f(✓) , log p(✓, ) 669 663f(✓f) (✓)f(✓0log)+p(f✓f(✓,(✓)0))>(0✓ ✓0) D ⇡ , r f0(✓0) −0 663 665 670 664 rrrr D ≺ ≺ 670 1 648 1 f(✓) f(✓0)+ f(✓0)>(✓ ✓0) 666 665+ (✓ ✓0)> 1 f(✓0)(✓ ✓0) ⇡ p(xr)= p xi pa(− xi) 664 671 f(f✓(0✓649) ) (✓(✓ ✓0)✓>A)>(A✓ (✓1✓0)✓ ) | 671 f(✓666) 2f⇡(✓−0)+0− rr2f(✓−0)>(✓0 −✓0−) 0 i 667 ⇡ 650 − 2 − + −(✓ ✓0)> f(Y✓0)(✓ ✓0) 665 672 ⇡ r − 2 − q(x )rrexp −[log p(x)] 667 A651=f(✓ )=f0(✓ ) 0 j Eq(x j ) 668 1 A =0 f0(✓0) 0 / { ¬ } 666 673 + 668(✓ ✓ )>r rrf(✓ )(✓ ✓ ) f(✓0)=0x 669 673 0 652 rr0 0 r j 674 2669 − 653rrf(✓0) 0 −1 667 674 rr ≺ f(✓0) 0 670 p(✓, ) p(✓6540, )exp (✓ ✓q0(x)j>) Aexp(rr✓ E ✓log0)p(≺xj pa(xj)) 675 670- D ⇡ 1 D − 2 − / 1{− | 668 675 f(✓ ) 655(✓ ✓ ) A(✓ ✓ ) 671 0 0 > - 0 671 f(✓0) + (✓⇥ ✓E0)>logAp((✓xt x⇤✓j,0pa() xt) xj ) (5) 676 ⇡ −6562 − − ⇡ Gaussian!− 2 − −| \{ } } 669 672 676 xj pa(xt) 672 2X 677 A =657 f(✓0) 0 A = f(✓⇥0) 0 ⇤ 670 673 677 673 658rr rr p(✓, ) 678 674 1 1 D 674 678 659 p(✓, ) p(✓0, )exp (p✓(✓ ✓)0)>A(✓ ✓0) 671 p(✓, ) p(✓0, )exp (✓ ✓0)>A(✓ ✓0) 2 |D 679 D 675⇡ D 660 − 2 − D ⇡ −D − − − 672 675 679 ✓0 = argmax log p(✓, ) =1 argmax log p( ✓) 680 676 661 1 p(✓ )✓ (✓ ✓0,DA− ) ✓ D| 676 680 p(✓ ) (✓ ✓ , A− ) |D ⇡ N | 677 662 0 673 681 |D ⇡ N | A = logf(✓p)(✓,, log) p✓(=✓✓, ) 663 D0 677 681 678 rr D | 674 682 664 EenE- 678 679 f(✓) f(✓0)+ f(✓0)7>(✓ ✓0) 682 665 ⇡ r − 675 683 680 1 679 683 666 + (✓ ✓0)> f(✓0)(✓ ✓0) 676 684 681 2 − rr − 680 684 667 685 682 668 f(✓0)=0 677 681 685 r 683 669 f(✓0) 0 678 682 686 rr ≺ 686 684 670 1 687 671 f(✓0) (✓ ✓0)>A(✓ ✓0) 679 683 687 685 ⇡ − 2 − − 688 672 684 686 A = f(✓0) 0 680 688 673 rr 689 685 687 674 1 681 689 p(✓, ) p(✓0, )exp (✓ ✓0)>A(✓ ✓0) 690 688 675 D ⇡ D − 2 − − 686 1 682 690 689 676 p(✓ ) (✓ ✓0, A− ) 687 691 |D ⇡ N | 683 691 690 677 688 692 678 692 691 684 693 679 689 692 693 680 685 690 694 693 694 681 686 691 695 694 682 695 687 692 696 695 683 696 684 693 697696 688 697 685 694 698697 686 689 698 698 687 695 699 699 690 688 696 700699 700 691 689 697 701700 701 690 692 698 701 691 693 692 699 693 13 694 700 694 13 695 701 695 696 13 696 697 697 698 699 698 700 13 699 701 700 701 13 13 4.4.