An Introduction to Complementarity
Michael C. Ferris
University of Wisconsin, Madison
Nonsmooth Mechanics Meeting: June 14, 2010
Ferris (Univ. Wisconsi) EMP Aussois, June 2010 1 / 63 Outline
Introduction to Complementarity Models Extension to Variational Inequalities Extended Mathematical Programming Heirarchical Optimization
Introduction: Transportation Model Application: World Dairy Market Model Algorithms: Feasible Descent Framework Implementation: PATH Results
Ferris (Univ. Wisconsi) EMP Aussois, June 2010 2 / 63 Sample Network
D1
S1
D2
S2
D3
S3
D4
Ferris (Univ. Wisconsi) EMP Aussois, June 2010 3 / 63 Transportation Model
Suppliers ship good from warehouses to customers
I Satisfy demand for commodity I Minimize transportation cost Transportation network provided as set A of arcs
Variables xi,j - amount shipped over (i, j) ∈ A Parameters
I si - supply at node i I di - demand at node i I ci,j - cost to ship good from nodes i to j
Ferris (Univ. Wisconsi) EMP Aussois, June 2010 4 / 63 Linear Program
P minx≥0 (i,j)∈A ci,j xi,j P subject to j:(i,j)∈A xi,j ≤ si ∀ i P i:(i,j)∈A xi,j ≥ dj ∀ j
Ferris (Univ. Wisconsi) EMP Aussois, June 2010 5 / 63 Multipliers
Introduce multipliers (marginal prices) ps and pd
P s j:(i,j)∈A xi,j ≤ si pi ≥ 0
In a competitive marketplace
P s j:(i,j)∈A xi,j < si ⇒ pi = 0
At solution P s j:(i,j)∈A xi,j = si or pi = 0 Complementarity relationship
P s j:(i,j)∈A xi,j ≤ si ⊥ pi ≥ 0
Ferris (Univ. Wisconsi) EMP Aussois, June 2010 6 / 63 Wardropian Equilibrium
Delivery cost exceeds market price
s d pi + ci,j ≥ pj
Strict inequality implies no shipment xi,j = 0 Linear complementarity problem
P s j:(i,j)∈A xi,j ≤ si ⊥ pi ≥ 0 ∀ i P d dj ≤ i:(i,j)∈A xi,j ⊥ pi ≥ 0 ∀ j d s pj ≤ pi + ci,j ⊥ xi,j ≥ 0 ∀ (i, j) ∈ A • First order conditions for linear program
Ferris (Univ. Wisconsi) EMP Aussois, June 2010 7 / 63 Nonlinear Complementarity Problems Given F : 0 ≤ F (x) x ≥ 0 xT F (x) = 0 Compactly written 0 ≤ F (x) ⊥ x ≥ 0 Equivalent to nonsmooth equation: min(x, F (x)) = 0 Ferris (Univ. Wisconsi) EMP Aussois, June 2010 8 / 63 Extensions Original problem has fixed demand Use general demand function d(p) Examples I Linear demand X d xi,j ≥ dj (1 − pj ) i:(i,j)∈A I Nonlinear demand F Cobb-Douglas F CES function Use more general cost functions c(x) Ferris (Univ. Wisconsi) EMP Aussois, June 2010 9 / 63 Taxes and Tariffs Exogenous supply tax ti s d pi (1 + ti ) + ci,j ≥ pj Endogenous taxes I Make ti a variable I Add driving equation No longer optimality conditions Most complementarity problems do not correspond to first order conditions of optimization problems Ferris (Univ. Wisconsi) EMP Aussois, June 2010 10 / 63 Use of complementarity Pricing electricity markets and options Contact Problems with Friction Video games: model contact problems I Friction only occurs if bodies are in contact Crack Propagation Structure design I how springy is concrete I optimal sailboat rig design Congestion in Networks Computer/traffic networks I The price of anarchy measures difference between “system optimal” (MPCC) and “individual optimization” (MCP) Electricity Market Deregulation Game Theory (Nash Equilibria) General Equilibria with Incomplete Markets Impact of Environmental Policy Reform Ferris (Univ. Wisconsi) EMP Aussois, June 2010 11 / 63 World Dairy Market Model Spatial equilibrium model of world dairy sector I 5 farm milk types I 8 processed goods I 21 regions Regions trade raw and processed goods Barriers to free trade I Import policies: quotas, tariffs I Export policies: subsidies • Study impact of policy decisions I GATT/URAA I Future trade negotiations Ferris (Univ. Wisconsi) EMP Aussois, June 2010 12 / 63 Formulation Quadratic program I Variables: quantities I Constraints: production and transportation I Objective: maximize net social welfare • Difficulty is ad valorem tariffs I Tariff based on value of goods I Market value is multiplier on constraint • Complementarity problem I Formulate optimality conditions I Market price is now a variable I Directly model ad valorem tariffs Ferris (Univ. Wisconsi) EMP Aussois, June 2010 13 / 63 World Dairy Market Model Statistics Quadratic program I 31,772 variables I 14,118 constraints Linear complementarity problem I 45,890 variables and constraints I 131,831 nonzeros Ferris (Univ. Wisconsi) EMP Aussois, June 2010 14 / 63 European Put Option Contract where holder can sell an asset 1 At a fixed expiration time T 2 For a fixed price E Asset value S(t) satisfies dS = (σ dX + µ dt)S I σ dX is random return I µ dt is deterministic return Black-Scholes Equation ∂V 1 ∂2V ∂V + σ2S2 + rS − rV = 0 ∂t 2 ∂S2 ∂S Ferris (Univ. Wisconsi) EMP Aussois, June 2010 15 / 63 American Put Options Contract where holder can sell an asset 1 At any time prior to a fixed expiration T 2 For a fixed price E Free Boundary Sf (t) – optimal exercise price V (Sf (t), t) = max(E − Sf (t), 0) ∂V (S (t), t) = −1 ∂S f If S ≥ Sf (t) then satisfy Black-Scholes Equation ∂V 1 ∂2V ∂V + σ2S2 + rS − rV = 0 ∂t 2 ∂S2 ∂S Ferris (Univ. Wisconsi) EMP Aussois, June 2010 16 / 63 Complementarity Problem Reformulate to remove dependence on boundary Partial differential complementarity problem ∂V 1 2 2 ∂2V ∂V F (S, t) ≡ ∂t + 2 σ S ∂S2 + rS ∂S − rV 0 ≤ −F (S, t) ⊥ V (S, t) ≥ max(E − S, 0) Free boundary recovered after solving Ferris (Univ. Wisconsi) EMP Aussois, June 2010 17 / 63 Solution Method Finite differences used to discretize I Central differences for space I Forward differences for time I Crank-Nicolson method Step through time from T to present At each t a linear complementarity problem is solved Used GAMS/PATH to model and solve Ferris (Univ. Wisconsi) EMP Aussois, June 2010 18 / 63 Results 7 payoff t = 0 6 5 4 3 Option Value (V) 2 1 0 4 6 8 10 12 14 16 Asset Price (S) Ferris (Univ. Wisconsi) EMP Aussois, June 2010 19 / 63 Background Nonlinear equations F (x) = 0 Newton’s Method F (xk ) + ∇F (xk )dk = 0 xk+1 = xk + dk 1 2 Damp using Armijo linesearch on 2 kF (x)k2 Descent direction - gradient of merit function Properties I Well defined I Global and local-fast convergence Ferris (Univ. Wisconsi) EMP Aussois, June 2010 20 / 63 Cycling Example 2.5 2 1.5 1 0.5 0 f(x) −0.5 −1 −1.5 −2 −2.5 −1.5 −1 −0.5 0 0.5 1 1.5 x Ferris (Univ. Wisconsi) EMP Aussois, June 2010 21 / 63 Merit Function 3.5 3 2.5 2 2 0.5(f(x)) 1.5 1 0.5 0 −1.5 −1 −0.5 0 0.5 1 1.5 x Ferris (Univ. Wisconsi) EMP Aussois, June 2010 22 / 63 Normal Map Equivalent piecewise smooth equation F+(x) = 0 F+(x) ≡ F (x+) + x − x+ Nonsmooth Newton Method I Piecewise linear system of equations I Solve via a pivotal method 1 2 I Damp using Armijo search on 2 kF+(x)k2 Properties I Global and local-fast convergence I Merit function not differentiable Ferris (Univ. Wisconsi) EMP Aussois, June 2010 23 / 63 Nonlinear Complementarity Problem Assume F+(¯x) = 0 1 Ifx ¯i ≥ 0 thenx ¯i − (¯xi )+ = 0 and Fi (¯x) = 0 2 Ifx ¯i ≤ 0 thenx ¯i − (¯xi )+ ≤ 0 and Fi (¯x) ≥ 0 Thereforex ¯+ solves 0 ≤ F (x) x ≥ 0 xT F (x) = 0 Compact representation 0 ≤ F (x) ⊥ x ≥ 0 Ifz ¯ solves NCP(F ) then F+(¯z − F (¯z)) = 0 Ferris (Univ. Wisconsi) EMP Aussois, June 2010 24 / 63 Piecewise Linear Example 0.5 0.4 0.3 0.2 0.1 x2 0 −0.1 −0.2 −0.3 −0.4 −0.5 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5 x1 Ferris (Univ. Wisconsi) EMP Aussois, June 2010 25 / 63 Fischer-Burmeister Function Φ(x) defined componentwise q 2 2 Φi (x) ≡ (xi ) + (Fi (x)) − xi − Fi (x) Φ(x) = 0 if and only if x solves NCP(F ) Not continuously differentiable - semismooth 1 2 Natural merit function ( 2 kΦ(x)k2) is differentiable Ferris (Univ. Wisconsi) EMP Aussois, June 2010 26 / 63 Fischer-Burmeister Example 0.5 0.4 0.3 0.2 0.1 x2 0 −0.1 −0.2 −0.3 −0.4 −0.5 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5 x1 Ferris (Univ. Wisconsi) EMP Aussois, June 2010 27 / 63 Review Nonlinear Complementarity Problem Piecewise smooth system of equations I Use nonsmooth Newton Method I Solve linear complementarity problem per iteration I Merit function not differentiable Fischer-Burmeister I Differentiable merit function Combine to obtain new algorithm I Well defined I Global and local-fast convergence Ferris (Univ. Wisconsi) EMP Aussois, June 2010 28 / 63 Feasible Descent Framework Calculate direction using a local method I Generates feasible iterates I Local fast convergence I Used nonsmooth Newton Method 1 2 Accept direction if descent for 2 kΦ(x)k Otherwise use projected gradient step Ferris (Univ. Wisconsi) EMP Aussois, June 2010 29 / 63 Theorem Let {xk } ⊆ Ferris (Univ. Wisconsi) EMP Aussois, June 2010 30 / 63 Computational Details Crashing method to quickly identify basis Nonmonotone search with watchdog Perturbation scheme for rank deficiency Stable interpolating pathsearch Restart strategy Projected gradient searches Diagnostic information Ferris (Univ. Wisconsi) EMP Aussois, June 2010 31 / 63 Restarts User provides solver with information I Starting point I Resource limits Effectively use resources to solve problem Determine when at a stationary point and kΦ(x)k > 0 I Restart from starting point I Modify algorithmic parameters Parameter choices based on empirical studies Ferris (Univ. Wisconsi) EMP Aussois, June 2010 32 / 63 Comparative Results Models from GAMSLIB and MCPLIB 100% Success Rate 95% ( 1255 Instances 90% ) 85% PATH 2.9 PATH 3.2 PATH 4.2 Ferris (Univ. Wisconsi) EMP Aussois, June 2010 33 / 63 Preprocessing Discover information about a problem Use to reduce size and complexity I Improve algorithm performance I Detect unsolvable models Main idea I Identify special structure F Polyhedral constraints F Separability I Use complementarity theory to eliminate variables Ferris (Univ. Wisconsi) EMP Aussois, June 2010 34 / 63 Linear Solve Majority of time spent finding direction Advanced starts Ill-conditioning and rank-deficiency Degeneracy in pivot sequence Cycling rules Stable regeneration of search path Ferris (Univ. Wisconsi) EMP Aussois, June 2010 35 / 63 Core Computation Each step solves a (large, sparse) linear system Pivot step updates system matrix by a rank-1 modification (see details later) Require factor, solve, update technology I Dense version: uses Fletcher Matthews updates of LU factors I Default version: uses LUSOL (Markovitz sparsity, Bartels Golub factor updates, rank revealing factorization) I New version: uses UMFPACK (unsymmetric multifrontal method, block LU updating (Schur Complement) for updates) I Compressed version: much more complicated to implement, not as efficient in practice over complete set of models Ferris (Univ. Wisconsi) EMP Aussois, June 2010 36 / 63 Availability Modeling Languages I GAMS I AMPL MATLAB I MEX interface NEOS I FORTRAN specification I ADIFOR to obtain Jacobian I Large problems solved via CONDOR Callable Library Ferris (Univ. Wisconsi) EMP Aussois, June 2010 37 / 63 Conclusions Complementarity generalizes nonlinear equations Nonsmooth Newton method proposed I Differentiable merit function I Well defined I Global and local-fast convergence Developed sophisticated implementation Applied to several problems I Transportation model I Options pricing Future – improve speed and reliability Ferris (Univ. Wisconsi) EMP Aussois, June 2010 38 / 63 Discussion Very robust on standard test set Obtained large, difficult models from colleagues I World Dairy Market Model I Several quadratic programs Improve performance on large scale problems I Robustness I Speed Ferris (Univ. Wisconsi) EMP Aussois, June 2010 39 / 63 Variational Inequality Formulation F : 0 ∈ F (x)+ NX (x) • Special Cases n I Nonlinear Equations (X ≡ < ) F (x) = 0 n I Nonlinear Complementarity Problem (X ≡ <+) 0 ≤ F (x) x ≥ 0 x T F (x) = 0 Ferris (Univ. Wisconsi) EMP Aussois, June 2010 40 / 63 Polyhedral Constraints X X and NX (·) are geometric objects Free to choose algebraic representation Partition into two components: X ≡ B ∩ C I B - simple bounds – treated specially by algorithm I C - polyhedral set Reduce complexity of C Must find X automatically Ferris (Univ. Wisconsi) EMP Aussois, June 2010 41 / 63 Polyhedral Structure Partition variables into (x, y) Identify skew symmetric structure F (x) − AT y N n (x) 0 ∈ + <+ Ax − b N n (y) <+ • Equivalent polyhedral problem (Robinson) 0 ∈ F (x) + N n (x) <+∩{x|Ax−b≥0} Implementation finds a single constraint at a time Ferris (Univ. Wisconsi) EMP Aussois, June 2010 42 / 63 Relationship 1. If (¯x, y¯) solves box constrained problem then x¯ solves the polyhedral problem 2. If x¯ solves the polyhedral problem then there exist multipliers y¯ such that (¯x, y¯) solves the box constrained problem How do we calculate the multipliers,y ¯? Ferris (Univ. Wisconsi) EMP Aussois, June 2010 43 / 63 Calculating Multipliers Given an¯x solving the polyhedral problem Choose¯y solving the following linear program T min n y (Ax¯ − b) y∈<+ T s.t. 0 ∈ F (¯x) − A y + N n (¯x) <+ If x¯ solves the polyhedral problem then 1. The linear program is solvable 2. Given any y¯ in the solution set, (¯x, y¯) solves the box constrained problem Ferris (Univ. Wisconsi) EMP Aussois, June 2010 44 / 63 Separable Structure Partition variables into (x, y) Identify separable structure F (x) N n (x) 0 ∈ + <+ G(x, y) N n (y) <+ Reductions possible if either 1 n 0 ∈ F (x) + N<+ (x) has a unique solution 2 n 0 ∈ G(x, y) + N<+ (y) has solution for all x Theory provides appropriate conditions Solve F and G sequentially Ferris (Univ. Wisconsi) EMP Aussois, June 2010 45 / 63 Presolve 1 Identify a constraint with skew symmetric property 2 Convert problem into polyhedral form 3 Modify representation of polyhedral set I Singleton and doubleton rows I Forcing constraints I Duplicate rows 4 Recover box constrained problem with reduced size I Multipliers fixed and function modified I Additional polyhedral constraints uncovered 5. Repeat 1–4 until no changes 6. Identify separable structure Ferris (Univ. Wisconsi) EMP Aussois, June 2010 46 / 63 Example Original problem x2−y − 1 N (x) 0 ∈ + <+ x − 1 N<+ (y) Polyhedral problem 2 0 ∈ x − 1 + N<+∩{x|x−1≥0}(x) Equivalent problem 2 0 ∈ x − 1 + N[1,∞)(x) Ferris (Univ. Wisconsi) EMP Aussois, June 2010 47 / 63 Example (continued) 2 0 ∈ x − 1 + N[1,∞)(x) has one solution¯x = 1 Solve optimization problem T miny∈<+ y (x¯ − 1) 2 s.t. 0 ∈ x¯ − y − 1 + N<+ (¯x) Equivalent model miny∈<+ 0 s.t. y = 0 Obtain¯y = 0 Solution is (1, 0) Ferris (Univ. Wisconsi) EMP Aussois, June 2010 48 / 63 Availability of Preprocessor PATH 4.3 for GAMS and AMPL I Finds polyhedral structure I Exploits separable structure Capability exists in other environments I User needs to provide information I Listing of linear/nonlinear elements in Jacobian I Optional - interval evaluation routines Ferris (Univ. Wisconsi) EMP Aussois, June 2010 49 / 63 Results: Linear Programs Formulated first order conditions of NETLIB problems Polyhedral structure not supplied to LCP preprocessor 30% Reduction 20% 10% 0% CPLEX LCP CPLEX (Without Aggregation) Ferris (Univ. Wisconsi) EMP Aussois, June 2010 50 / 63 Results: Quadratic Programs Solve optimality conditions Synthetic models 1 2 I NETLIB problems with 2 kxk added to objective I Selected 8 models I 17.6% reduction in size I 29.2% reduction in time World Dairy Market Model I Failed on original model (4.5 hours) I 70.4% reduction in size I Solved preprocessed model 23 minutes I 91.5% reduction in time! Ferris (Univ. Wisconsi) EMP Aussois, June 2010 51 / 63 Results: Nonlinear Complementarity Problems Models from GAMSLIB and MCPLIB Selected 6 models 9.7% reduction in size 15.3% reduction in time Ferris (Univ. Wisconsi) EMP Aussois, June 2010 52 / 63 World Dairy Market Model Statistics Quadratic program I 31,772 variables I 14,118 constraints Linear complementarity problem I 45,890 variables and constraints I 131,831 nonzeros Preprocessed problem I 22,159 variables and constraints I 70,475 nonzeros Ferris (Univ. Wisconsi) EMP Aussois, June 2010 53 / 63 World Dairy Market Model Results Note: want to analyze large number of scenarios Gauss-Seidel method I Solves 96 quadratic programs F Uses MINOS with nonstandard options I Approximates equilibrium in 42 minutes Complementarity formulation I Solves a single complementarity problem I Computes equilibrium in F 117 minutes without preprocessing F 21 minutes with preprocessing F 11 minutes with nonstandard options I Obtain more accurate result in less time! Ferris (Univ. Wisconsi) EMP Aussois, June 2010 54 / 63 If −MπC (x)−q = x −πC (x) then −MπC (x) − q ∈ NC (πC (x)) so z = πC (x) solves h−Mz − q, y − zi ≤ 0, ∀y ∈ C Find x, a zero of the normal map: 0 = MπC (x) + q + x − πC (x) Normal map for polyhedral C projection: πC (x) x − πC (x) ∈ NC (πC (x)) Normal Map Ferris (Univ. Wisconsi) EMP Aussois, June 2010 55 / 63 Find x, a zero of the normal map: 0 = MπC (x) + q + x − πC (x) Normal map for polyhedral C projection: πC (x) x − πC (x) ∈ NC (πC (x)) If −MπC (x)−q = x −πC (x) then −MNormalπC (x) − q ∈ N CMap(πC (x)) so z = πC (x) solves h−Mz − q, y − zi ≤ 0, ∀y ∈ C Ferris (Univ. Wisconsi) EMP Aussois, June 2010 55 / 63 Normal map for polyhedral C projection: πC (x) x − πC (x) ∈ NC (πC (x)) If −MπC (x)−q = x −πC (x) then −MNormalπC (x) − q ∈ N CMap(πC (x)) so z = πC (x) solves h−Mz − q, y − zi ≤ 0, ∀y ∈ C Find x, a zero of the normal map: 0 = MπC (x) + q + x − πC (x) Ferris (Univ. Wisconsi) EMP Aussois, June 2010 55 / 63 Normal manifold = {Fi + NFi } Ferris (Univ. Wisconsi) EMP Aussois, June 2010 56 / 63 0 C = {z|Bz ≥ b}, NC (z) = {B v|v ≤ 0, vI(z) = 0} Ferris (Univ. Wisconsi) EMP Aussois, June 2010 57 / 63 0 C = {z|Bz ≥ b}, NC (z) = {B v|v ≤ 0, vI(z) = 0} Ferris (Univ. Wisconsi) EMP Aussois, June 2010 58 / 63 0 C = {z|Bz ≥ b}, NC (z) = {B v|v ≤ 0, vI(z) = 0} Ferris (Univ. Wisconsi) EMP Aussois, June 2010 59 / 63 Cao/Ferris Path (Eaves) Cao/Ferris Path (Eaves) Start in cell that has interior (face is an extreme point) • Start in cell that has Move towards a zero of interior (face is an affine map in cell extreme point) Update direction when hit • Move towards a zero of boundary (pivot) affine map in cell Solves or determines • Update direction when infeasible if M is hit boundary copositive-plus on rec(C) • Solves or determines infeasible if M is Solves 2-person bimatrix copositive-plus on rec(C) games, 3-person games too, • Nails 2-person game but these are nonlinear But algorithm has exponential complexity (von Stengel et al) Ferris (Univ. Wisconsi) EMP Aussois, June 2010 60 / 63 Extensions and Computational Results Embed AVI solver in a Newton Method - each Newton step solves an AVI Compare performance of PathAVI with PATH on equivalent LCP PATH the most widely used code for solving MCP AVIs constructed to have solution with Mn×n symmetric indefinite PathAVI PATH Size (m,n) Resid Iter Resid Iter (180, 60) 3 × 10−14 193 0.9 10176 (360, 120) 3 × 10−14 1516 2.0 10594 2 - 10x speedup in Matlab using sparse LU instead of QR 2 - 10x speedup in C using sparse LU updates Ferris (Univ. Wisconsi) EMP Aussois, June 2010 61 / 63 Conclusions Complementarity problems abound in multiple application domains The PATH solver is a large scale (black-box) implementation of a (nonsmooth) Newton method for solving complementarity problems The PATH solver is available for download at http://www.cs.wisc.edu/∼ferris/path.html Mathematically rigourous extensions to Variational Inequalities and specific structures in models possible Ferris (Univ. Wisconsi) EMP Aussois, June 2010 62 / 63 References M. Cao and M. C. Ferris. A pivotal method for affine variational inequalities. Mathematics of Operations Research, 21:44–64, 1996. M. C. Ferris, C. Kanzow, and T. S. Munson. Feasible descent algorithms for mixed complementarity problems. Mathematical Programming, 86:475–497, 1999. M. C. Ferris and T. S. Munson. Preprocessing complementarity problems. In M. C. Ferris, O. L. Mangasarian, and J. S. Pang, editors, Complementarity: Applications, Algorithms and Extensions, volume 50 of Applied Optimization, pages 143–164, Dordrecht, The Netherlands, 2001. Kluwer Academic Publishers. M. C. Ferris and J. S. Pang. Engineering and economic applications of complementarity problems. SIAM Review, 39:669–713, 1997. Ferris (Univ. Wisconsi) EMP Aussois, June 2010 63 / 63