Interior Point Methods ME575 – Optimization Methods John Hedengren

40 =0 30 =0.1 =1.0

ln(x)  20 =2.0 =5.0 10 =10

Barrier Function, Barrier - -10

-20 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Variable (x) Overview

• Nonlinear constrained minimization • Slack variables • Barrier function • Newton’s method for KKT conditions • Interior Point Method (IPM) • Tutorial Examples Nonlinear Constrained Optimization

min 푓(푥) 푥∈푅푛 푠. 푡. 푐 푥 = 0 푥 ≥ 0 • Linear, quadratic, or general nonlinear objective and constraints • Convex optimization, local solution possible for non-convex problems • Convert maximization by minimizing negative of the objective • Convert general inequalities to simple inequalities with slack variables Slack Variables

min 푓(푥) min 푓(푥) 푥∈푅푛 푥∈푅푛 푠. 푡. 푔 푥 ≥ 푏 푠. 푡. 푐 푥 = 0 ℎ 푥 = 0 푥 ≥ 0 • Complete worksheet on slack variables

푚𝑖푛 푥1푥4 푥1 + 푥2 + 푥3 + 푥3 푥 푠. 푡. 푥1푥2푥3푥4 ≥ 25 2 2 2 2 푥1 + 푥2 + 푥3 + 푥4 = 40

• Additional slack variable tutorial: • http://apmonitor.com/me575/index.php/Main/SlackVariables Barrier Function

푛

min 푓(푥) min 푓 푥 − 휇 ෍ ln 푥푖 푥∈푅푛 푥∈푅푛 푠. 푡. 푐 푥 = 0 푖=1 푥 ≥ 0 푠. 푡. 푐 푥 = 0

40 • Natural log barrier term for =0 30 =0.1 =1.0

inequality constraints ln(x)  20 =2.0 =5.0 • The term ln 푥푖 is undefined 10 =10 for 푥푖 < 0 0

• Search points only in interior Function, Barrier - -10

-20 feasible space 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Variable (x) Barrier Function Example 1

min 푥 − 3 2 min 푥 − 3 2 − 휇 ln 푥 푥∈푅 푥∈푅 푠. 푡. 푥 ≥ 0 40 Unconstrained =1 30 =2 =5 =10 20

Augmented ObjectiveAugmented Function

-10

-20 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Dependent Variable (x) Barrier Function Example 2

min 푥 + 1 2 min 푥 + 1 2 − 휇 ln 푥 푥∈푅 푥∈푅

푠. 푡. 푥 ≥ 0 10 Unconstrained 9 =1 8 =2 =5 7 =10

3 Augmented ObjectiveAugmented Function 2

0 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 Dependent Variable (x) How to Solve a Barrier Problem: Step 1

• KKT Conditions for Barrier Problem 푛 푛 1 min푛 푓 푥 − 휇 ෍ ln 푥푖 훻푓 푥 + 훻푐 푥 휆 − 휇 ෍ = 0 푥∈푅 푥푖 푖=1 푖=1 푠. 푡. 푐 푥 = 0 푐 푥 = 0 1 • Define 푧푖 = and solve modified version of KKT conditions 푥푖

훻푓 푥 + 훻푐 푥 휆 − 푧 = 0 푐 푥 = 0 푋푍푒 − 휇푒 = 0 How to Solve a Barrier Problem: Step 2

• Find KKT solution with Newton-Raphson method 푥 훻푓 푥 + 훻푐 푥 휆 − 푧 = 0 푊푘 훻푐 푥푘 −퐼 푑푘 훻푓 푥푘 + 훻푐 푥푘 휆푘 − 푧푘 푐 푥 = 0 푇 휆 훻푐 푥푘 0 0 푑푘 = − 푐 푥푘 푋푍푒 − 휇푒 = 0 푧 푋 푍 푒 − 휇 푒 푍푘 0 푋푘 푑푘 푘 푘 푗 • Where: 푧1 0 0 푥1 0 0 2 2 푇 푊푘 = 훻푥푥퐿 푥푘, 휆푘, 푧푘 = 훻푥푥 푓 푥푘 + 푐 푥푘 휆푘 − 푧푘 푍푘 = 0 ⋱ 0 푋푘 = 0 ⋱ 0 0 0 푧푛 0 0 푥푛 Find a Search Direction and Step Size: Example 3

푑푥 푚𝑖푛 푓 = 푦2 푊푘 훻푐 푥푘 −퐼 푘 훻푓 푥푘 + 훻푐 푥푘 휆푘 − 푧푘 1 푇 휆 푥∈푅 훻푐 푥푘 0 0 푑푘 = − 푐 푥푘 푧 푋 푍 푒 − 휇 푒 푠. 푡. 2푦 ≤ 9 푍푘 0 푋푘 푑푘 푘 푘 푗 푦 ≥ 0 2 2 푇 푊푘 = 훻푥푥퐿 푥푘, 휆푘, 푧푘 = 훻푥푥 푓 푥푘 + 푐 푥푘 휆푘 − 푧푘 푠푡푎푟푡𝑖푛푔 푎푡 푦 = 3, 휇 = 10, 휆 = 1, 훼 = 0.5 How to Solve a Barrier Problem: Step 2

• Find KKT solution with Newton-Raphson method 푥 훻푓 푥 + 훻푐 푥 휆 − 푧 = 0 푊푘 훻푐 푥푘 −퐼 푑푘 훻푓 푥푘 + 훻푐 푥푘 휆푘 − 푧푘 푐 푥 = 0 푇 휆 훻푐 푥푘 0 0 푑푘 = − 푐 푥푘 푋푍푒 − 휇푒 = 0 푧 푋 푍 푒 − 휇 푒 푍푘 0 푋푘 푑푘 푘 푘 푗 • Where: 푧1 0 0 푥1 0 0 2 2 푇 푊푘 = 훻푥푥퐿 푥푘, 휆푘, 푧푘 = 훻푥푥 푓 푥푘 + 푐 푥푘 휆푘 − 푧푘 푍푘 = 0 ⋱ 0 푋푘 = 0 ⋱ 0 0 0 푧 0 0 푥 • Rearrange into symmetric linear system 푛 푛

푥 푊푘 + Σ푘 훻푐 푥푘 푑푘 훻푓 푥푘 + 훻푐 푥푘 휆푘 −1 푇 휆 = − Σ푘 = 푋푘 푍푘 훻푐 푥푘 0 푑푘 푐 푥푘 푧 푥 휆 • Solve for 푑푘 after the linear solution to 푑푘 and 푑푘 with explicit solution 푧 −1 푥 푑푘 = 휇푘푋푘 푒 − 푧푘 − Σ푘푑푘 Step Size (훼)

• Two objectives in evaluating progress • Minimize objective • Minimize constraint violations • Two popular approaches • Decrease in merit function, 푚푒푟𝑖푡 = 푓 푥 + 휈 σ 푐(푥) • Filter methods • Cut back step size until improvement 푥 푥푘+1 = 푥푘 + 훼푘푑푘 휆 휆푘+1 = 휆푘 + 훼푘푑푘 푧 푧푘+1 = 푧푘 + 훼푘푑푘 Convergence Criteria

• Convergence when KKT conditions are satisfied with a tolerance

max 훻푓 푥 + 훻푐 푥 휆 − 푧 ≤ 휖푡표푙

max 푐(푥) ≤ 휖푡표푙

max 푋푍푒 − 휇푒 ≤ 휖푡표푙 • Tolerance for constraint violation may be more restrictive Interior Point Method Overview

Initialize 휆 Initialize x , l , z 푥 = 푓푒푎푠𝑖푏푙푒 0 0 0 0 휇 퐼 훻푐 푥0 푤 훻푓 푥 − 푧 − 푧 푧0 = 0 퐿,0 푈,0 푥 푇 휆 = − 0 훻푐 푥0 0 0 0

Yes Check for Convergence Optimal Solution

No 퐸 푥, 휆, 푧 ≤ 휖푡표푙

푥 Compute the Search Direction −1 푊푘 + Σ푘 훻푐 푥푘 푑푘 훻푓 푥푘 + 훻푐 푥푘 휆푘 − 푍푘 Σ푘 = 푋푘 푍푘 푇 휆 = − with the linearized Barrier Problem 훻푐 푥푘 0 푑푘 푐(푥푘) 푧 −1 푥 푑푘 = 휇푋푘 푒 − 푧푘 − Σ푘푑푘

푥 Backtracking Line Search 푥푘+1 = 푥푘 + 훼푘푑푘 Determine 훼 by decrease in Merit Function or 휆 = 휆 + 훼 푑휆 푘+1 푘 푘 푘 with Filter Methods 푧 푧푘+1 = 푧푘 + 훼푘푑푘 Barrier Function Example 4

푚𝑖푛 푥2 5 + 푥1 푥∈푅2 푥1푥2 ≥ 5 푠. 푡. 2 2 푥1 + 푥2 ≤ 20 Barrier Function: Example 4

min 푥2 5 + 푥1 푥∈푅2 푥1푥2 ≥ 5 푠. 푡. 2 2 푥1 + 푥2 ≤ 20 Comparing Interior Point and Active Set Methods

100

80 APOPT+BPOPT APOPT 70 1.0 BPOPT 1.0 60 IPOPT 3.10 IPOPT 50 2.3 SNOPT

Percentage (%) Percentage 6.1 40 MINOS 5.5

20 NLP Benchmark – Summary (494 Problems)

10 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Not worse than 2 times slower than the best solver () APOPT Solver

• AMPL / MATLAB / Python Interfaces from http://apopt.com IPOPT Solver

• Download Source Code from https://projects.coin-or.org/Ipopt Interior Point Homework Problem 1

2 2 min 푥1 − 2푥1푥2 + 4푥2 푥∈푅2 푠. 푡. 0.1푥1 − 푥2 > 1 Interior Point Homework Problem 2

2 2 min 푥1 + 2푥2 푥∈푅2 푠. 푡. 2푥1 + 푥2 ≤ 9 푥1 + 2푥2 푥1 > 0, 푥2 > 0 Homework Help: IPOPT Output

• Visit: http://apmonitor.com/online/view_pass.php?f=ipm.apm Homework Help: IPOPT Output IPOPT Output Headings

• iter: The current iteration count. • objective: The unscaled objective value at the current point. • inf_pr: The unscaled max constraint violation at the current point. • inf_du: The max scaled dual infeasibility at the current point. • lg(mu): log10 of the value of the barrier parameter m • ||d||: The infinity norm (max) of the primal step • lg(rg): log10 of the value of the regularization term for the Hessian of the Lagrangian. Dash (-) indicates that no regularization was done. • alpha_du: The stepsize for the dual variables • alpha_pr: The stepsize for the primal variables • ls: The number of backtracking line search steps Homework Help: IPOPT Output Homework Help

• Download BPOPT Solver (MATLAB version) from Interior Point Page • http://apmonitor.com/me575/index.php/Main/InteriorPointMethod • Homework Problem 1 = BPOPT problem 10 • Homework Problem 2 = BPOPT problem 6

% load bpopt libraries addpath('bpopt')

% Solve problem 1 prob = bp_create(10); % create problem sol = bpopt(prob); % solve problem