The practical revised simplex method
Julian Hall School of Mathematics University of Edinburgh
January 25th 2007
The practical revised simplex method Overview Overview
Part 1:
• The mathematics of linear programming Overview
Part 1:
• The mathematics of linear programming • The simplex method for linear programming Overview
Part 1:
• The mathematics of linear programming • The simplex method for linear programming ◦ The standard simplex method ◦ The revised simplex method Overview
Part 1:
• The mathematics of linear programming • The simplex method for linear programming ◦ The standard simplex method ◦ The revised simplex method • Sparsity Overview
Part 1:
• The mathematics of linear programming • The simplex method for linear programming ◦ The standard simplex method ◦ The revised simplex method • Sparsity ◦ Basic concepts ◦ Example from Gaussian elimination ◦ Sparsity in the standard simplex method Overview
Part 1:
• The mathematics of linear programming • The simplex method for linear programming ◦ The standard simplex method ◦ The revised simplex method • Sparsity ◦ Basic concepts ◦ Example from Gaussian elimination ◦ Sparsity in the standard simplex method
Part 2:
• Practical implementation of the revised simplex method Overview
Part 1:
• The mathematics of linear programming • The simplex method for linear programming ◦ The standard simplex method ◦ The revised simplex method • Sparsity ◦ Basic concepts ◦ Example from Gaussian elimination ◦ Sparsity in the standard simplex method
Part 2:
• Practical implementation of the revised simplex method • Parallel simplex Overview
Part 1:
• The mathematics of linear programming • The simplex method for linear programming ◦ The standard simplex method ◦ The revised simplex method • Sparsity ◦ Basic concepts ◦ Example from Gaussian elimination ◦ Sparsity in the standard simplex method
Part 2:
• Practical implementation of the revised simplex method • Parallel simplex • Research frontiers
The practical revised simplex method 1 Solving LP problems minimize f = cT x subject to Ax = b x ≥ 0 where x ∈ IRn and b ∈ IRm. Solving LP problems minimize f = cT x subject to Ax = b x ≥ 0 where x ∈ IRn and b ∈ IRm.
• The feasible region is the solution set of equations and bounds Geometrically, it is a convex polyhedron in IRn. Solving LP problems minimize f = cT x subject to Ax = b x ≥ 0 where x ∈ IRn and b ∈ IRm.
• The feasible region is the solution set of equations and bounds Geometrically, it is a convex polyhedron in IRn. • At any vertex the variables may be partitioned into index sets
◦ B of m basic variables xB ≥ 0: ◦ N of n − m nonbasic variables xN = 0. Solving LP problems minimize f = cT x subject to Ax = b x ≥ 0 where x ∈ IRn and b ∈ IRm.
• The feasible region is the solution set of equations and bounds Geometrically, it is a convex polyhedron in IRn. • At any vertex the variables may be partitioned into index sets
◦ B of m basic variables xB ≥ 0: ◦ N of n − m nonbasic variables xN = 0. • Components of c and columns of A are
◦ the basic costs cB and basis matrix B; Solving LP problems minimize f = cT x subject to Ax = b x ≥ 0 where x ∈ IRn and b ∈ IRm.
• The feasible region is the solution set of equations and bounds Geometrically, it is a convex polyhedron in IRn. • At any vertex the variables may be partitioned into index sets
◦ B of m basic variables xB ≥ 0: ◦ N of n − m nonbasic variables xN = 0. • Components of c and columns of A are
◦ the basic costs cB and basis matrix B; ◦ the non-basic costs cN and matrix N. Solving LP problems minimize f = cT x subject to Ax = b x ≥ 0 where x ∈ IRn and b ∈ IRm.
• The feasible region is the solution set of equations and bounds Geometrically, it is a convex polyhedron in IRn. • At any vertex the variables may be partitioned into index sets
◦ B of m basic variables xB ≥ 0: ◦ N of n − m nonbasic variables xN = 0. • Components of c and columns of A are
◦ the basic costs cB and basis matrix B; ◦ the non-basic costs cN and matrix N. • Results: ◦ At any vertex there is a partition {B, N} of the variables such that B is nonsingular. Solving LP problems minimize f = cT x subject to Ax = b x ≥ 0 where x ∈ IRn and b ∈ IRm.
• The feasible region is the solution set of equations and bounds Geometrically, it is a convex polyhedron in IRn. • At any vertex the variables may be partitioned into index sets
◦ B of m basic variables xB ≥ 0: ◦ N of n − m nonbasic variables xN = 0. • Components of c and columns of A are
◦ the basic costs cB and basis matrix B; ◦ the non-basic costs cN and matrix N. • Results: ◦ At any vertex there is a partition {B, N} of the variables such that B is nonsingular. ◦ There is an optimal solution of the problem at a vertex.
The practical revised simplex method 2 Conditions for optimality
At any vertex the original problem is
T T minimize f = cN xN + cBxB subject to N xN + B xB = b xN ≥ 0 xB ≥ 0. Conditions for optimality
At any vertex the original problem is
T T minimize f = cN xN + cBxB subject to N xN + B xB = b xN ≥ 0 xB ≥ 0.
Multiply through by B−1 to form the reduced equations
NˆxN + xB = bˆ Conditions for optimality
At any vertex the original problem is
T T minimize f = cN xN + cBxB subject to N xN + B xB = b xN ≥ 0 xB ≥ 0.
Multiply through by B−1 to form the reduced equations
NˆxN + xB = bˆ
where the reduced non-basic matrix is
Nˆ = B−1N Conditions for optimality
At any vertex the original problem is
T T minimize f = cN xN + cBxB subject to N xN + B xB = b xN ≥ 0 xB ≥ 0.
Multiply through by B−1 to form the reduced equations
NˆxN + xB = bˆ
where the reduced non-basic matrix is
Nˆ = B−1N
and the vector of values of the basic variables is
bˆ = B−1b.
The practical revised simplex method 3 Conditions for optimality (cont.)
Use the reduced equations to eliminate xB from the objective to give the reduced problem T ˆ minimize f = cˆN xN + f subject to Nˆ xN + I xB = bˆ xN ≥ 0 xB ≥ 0,
where cˆN is the vector of reduced costs given by T T T ˆ cˆN = cN − cBN Conditions for optimality (cont.)
Use the reduced equations to eliminate xB from the objective to give the reduced problem T ˆ minimize f = cˆN xN + f subject to Nˆ xN + I xB = bˆ xN ≥ 0 xB ≥ 0,
where cˆN is the vector of reduced costs given by T T T ˆ cˆN = cN − cBN
and the value of the objective at the vertex is
ˆ T ˆ f = cBb. Conditions for optimality (cont.)
Use the reduced equations to eliminate xB from the objective to give the reduced problem T ˆ minimize f = cˆN xN + f subject to Nˆ xN + I xB = bˆ xN ≥ 0 xB ≥ 0,
where cˆN is the vector of reduced costs given by T T T ˆ cˆN = cN − cBN
and the value of the objective at the vertex is
ˆ T ˆ f = cBb.
Necessary and sufficient conditions for optimality are
cˆN ≥ 0.
The practical revised simplex method 4 The standard simplex method (1948)
N B RHS 1 . Nˆ I bˆ m T T ˆ 0 cˆN 0 −f The standard simplex method (1948)
N B RHS 1 . Nˆ I bˆ m T T ˆ 0 cˆN 0 −f
In each iteration: The standard simplex method (1948)
N B RHS 1 . Nˆ I bˆ m T T ˆ 0 cˆN 0 −f
In each iteration:
• Select the pivotal column q0 of a nonbasic variable q ∈ N to be increased from zero. The standard simplex method (1948)
N B RHS 1 . Nˆ I bˆ m T T ˆ 0 cˆN 0 −f
In each iteration:
• Select the pivotal column q0 of a nonbasic variable q ∈ N to be increased from zero. • Find the pivotal row p of the first basic variable p0 ∈ B to be zeroed. The standard simplex method (1948)
N B RHS 1 . Nˆ I bˆ m T T ˆ 0 cˆN 0 −f
In each iteration:
• Select the pivotal column q0 of a nonbasic variable q ∈ N to be increased from zero. • Find the pivotal row p of the first basic variable p0 ∈ B to be zeroed. • Exchange indices p0 and q between sets B and N . The standard simplex method (1948)
N B RHS 1 . Nˆ I bˆ m T T ˆ 0 cˆN 0 −f
In each iteration:
• Select the pivotal column q0 of a nonbasic variable q ∈ N to be increased from zero. • Find the pivotal row p of the first basic variable p0 ∈ B to be zeroed. • Exchange indices p0 and q between sets B and N . • Update tableau corresponding to this basis change.
The practical revised simplex method 5 The standard simplex method (cont.)
Advantages:
• Easy to understand • Simple to implement The standard simplex method (cont.)
Advantages:
• Easy to understand • Simple to implement
Disadvantages:
• Expensive: the matrix Nˆ ‘usually’ treated as full. The standard simplex method (cont.)
Advantages:
• Easy to understand • Simple to implement
Disadvantages:
• Expensive: the matrix Nˆ ‘usually’ treated as full. ◦ Storage requirement: O(mn) memory locations. The standard simplex method (cont.)
Advantages:
• Easy to understand • Simple to implement
Disadvantages:
• Expensive: the matrix Nˆ ‘usually’ treated as full. ◦ Storage requirement: O(mn) memory locations. ◦ Computation requirement: O(mn) floating point operations per iteration. The standard simplex method (cont.)
Advantages:
• Easy to understand • Simple to implement
Disadvantages:
• Expensive: the matrix Nˆ ‘usually’ treated as full. ◦ Storage requirement: O(mn) memory locations. ◦ Computation requirement: O(mn) floating point operations per iteration. • Numerically unstable.
The practical revised simplex method 6 Degeneracy and termination
• A vertex is degenerate if at least one basic variable is zero • Degeneracy is very common in practice Degeneracy and termination
• A vertex is degenerate if at least one basic variable is zero • Degeneracy is very common in practice • If there are no degenerate vertices then ◦ the objective increases strictly each iteration; ◦ the simplex method cannot return to a vertex. • Since number of vertices is finite, the simplex method terminates at an optimal vertex.
The practical revised simplex method 7 Degeneracy and cycling
(3) (2)
(4) (6) (5)
(1) Degeneracy and cycling
(3) (2) • If there is a single degenerate vertex then (4) ◦ basis changes may correspond to steps of length zero; (6) ◦ a sequence of zero steps may lead to the simplex method (5) returning to a basis. • This is referred to as cycling and the simplex method fails to terminate. (1) Degeneracy and cycling
(3) (2) • If there is a single degenerate vertex then (4) ◦ basis changes may correspond to steps of length zero; (6) ◦ a sequence of zero steps may lead to the simplex method (5) returning to a basis. • This is referred to as cycling and the simplex method fails to terminate. (1) • Cycling is not an issue in practice but degeneracy is.
The practical revised simplex method 8 The revised simplex method (1953)
• Scan the reduced costs cˆN for a good candidate q to enter the basis. (CHUZC) The revised simplex method (1953)
• Scan the reduced costs cˆN for a good candidate q to enter the basis. (CHUZC) −1 • Form the pivotal column from column q of A: aˆq = B aq. (FTRAN) The revised simplex method (1953)
• Scan the reduced costs cˆN for a good candidate q to enter the basis. (CHUZC) −1 • Form the pivotal column from column q of A: aˆq = B aq. (FTRAN)
• Scan the ratios ˆbi/aˆiq for the row p of a good candidate to leave the basis. (CHUZR) The revised simplex method (1953)
• Scan the reduced costs cˆN for a good candidate q to enter the basis. (CHUZC) −1 • Form the pivotal column from column q of A: aˆq = B aq. (FTRAN)
• Scan the ratios ˆbi/aˆiq for the row p of a good candidate to leave the basis. (CHUZR)
• Update RHS bˆ and reduced cost cˆN vectors.
ˆ bp ˆ ˆ T α = , b := b + αaˆq and cˆN := cˆN − cˆqaˆp , aˆpq
T T −1 where the pivotal row aˆp = ep B N is calculated by forming
T T −1 π = ep B (BTRAN) The revised simplex method (1953)
• Scan the reduced costs cˆN for a good candidate q to enter the basis. (CHUZC) −1 • Form the pivotal column from column q of A: aˆq = B aq. (FTRAN)
• Scan the ratios ˆbi/aˆiq for the row p of a good candidate to leave the basis. (CHUZR)
• Update RHS bˆ and reduced cost cˆN vectors.
ˆ bp ˆ ˆ T α = , b := b + αaˆq and cˆN := cˆN − cˆqaˆp , aˆpq
T T −1 where the pivotal row aˆp = ep B N is calculated by forming
T T −1 π = ep B (BTRAN)
and T T aˆp = π N. (PRICE)
The practical revised simplex method 9 Computational cost
Given B−1, upper bounds on computational costs are Name Operation Cost
CHUZC Scan cˆN O(n) Computational cost
Given B−1, upper bounds on computational costs are Name Operation Cost
CHUZC Scan cˆN O(n) −1 2 FTRAN Form aˆq = B aq O(m ) Computational cost
Given B−1, upper bounds on computational costs are Name Operation Cost
CHUZC Scan cˆN O(n) −1 2 FTRAN Form aˆq = B aq O(m ) CHUZR Scan ratios ˆbi/aˆiq O(m) Computational cost
Given B−1, upper bounds on computational costs are Name Operation Cost
CHUZC Scan cˆN O(n) −1 2 FTRAN Form aˆq = B aq O(m ) CHUZR Scan ratios ˆbi/aˆiq O(m) T T −1 2 BTRAN Form π = ep B O(m ) Computational cost
Given B−1, upper bounds on computational costs are Name Operation Cost
CHUZC Scan cˆN O(n) −1 2 FTRAN Form aˆq = B aq O(m ) CHUZR Scan ratios ˆbi/aˆiq O(m) T T −1 2 BTRAN Form π = ep B O(m ) T T PRICE Form aˆp = π N O(mn) Computational cost
Given B−1, upper bounds on computational costs are Name Operation Cost
CHUZC Scan cˆN O(n) −1 2 FTRAN Form aˆq = B aq O(m ) CHUZR Scan ratios ˆbi/aˆiq O(m) T T −1 2 BTRAN Form π = ep B O(m ) T T PRICE Form aˆp = π N O(mn)
• Storage requirement: O(m2 + mn) memory locations. Computational cost
Given B−1, upper bounds on computational costs are Name Operation Cost
CHUZC Scan cˆN O(n) −1 2 FTRAN Form aˆq = B aq O(m ) CHUZR Scan ratios ˆbi/aˆiq O(m) T T −1 2 BTRAN Form π = ep B O(m ) T T PRICE Form aˆp = π N O(mn)
• Storage requirement: O(m2 + mn) memory locations. • Computation requirement: O(m2 + mn) floating point operations per iteration. Computational cost
Given B−1, upper bounds on computational costs are Name Operation Cost
CHUZC Scan cˆN O(n) −1 2 FTRAN Form aˆq = B aq O(m ) CHUZR Scan ratios ˆbi/aˆiq O(m) T T −1 2 BTRAN Form π = ep B O(m ) T T PRICE Form aˆp = π N O(mn)
• Storage requirement: O(m2 + mn) memory locations. • Computation requirement: O(m2 + mn) floating point operations per iteration. • Standard simplex method has storage/computation requirement of O(mn) per iteration. Computational cost
Given B−1, upper bounds on computational costs are Name Operation Cost
CHUZC Scan cˆN O(n) −1 2 FTRAN Form aˆq = B aq O(m ) CHUZR Scan ratios ˆbi/aˆiq O(m) T T −1 2 BTRAN Form π = ep B O(m ) T T PRICE Form aˆp = π N O(mn)
• Storage requirement: O(m2 + mn) memory locations. • Computation requirement: O(m2 + mn) floating point operations per iteration. • Standard simplex method has storage/computation requirement of O(mn) per iteration.
So what’s the advantage of the revised simplex method?
The practical revised simplex method 10 Sparsity Sparsity
LP problems are usually sparse: most of the entries of the constraint matrix A are zero Sparsity
LP problems are usually sparse: most of the entries of the constraint matrix A are zero Rows Columns Nonzeros Density (%) Nonzeros per Name (m) (n) (τ) (100τ/mn) column (τ/n) diet 3 6 18 100.0000 3.00 Sparsity
LP problems are usually sparse: most of the entries of the constraint matrix A are zero Rows Columns Nonzeros Density (%) Nonzeros per Name (m) (n) (τ) (100τ/mn) column (τ/n) diet 3 6 18 100.0000 3.00 avgas 10 8 30 37.5000 3.75 Sparsity
LP problems are usually sparse: most of the entries of the constraint matrix A are zero Rows Columns Nonzeros Density (%) Nonzeros per Name (m) (n) (τ) (100τ/mn) column (τ/n) diet 3 6 18 100.0000 3.00 avgas 10 8 30 37.5000 3.75 aqua 98 84 196 2.3810 2.33 Sparsity
LP problems are usually sparse: most of the entries of the constraint matrix A are zero Rows Columns Nonzeros Density (%) Nonzeros per Name (m) (n) (τ) (100τ/mn) column (τ/n) diet 3 6 18 100.0000 3.00 avgas 10 8 30 37.5000 3.75 aqua 98 84 196 2.3810 2.33 stair 356 467 3856 2.3219 8.26 Sparsity
LP problems are usually sparse: most of the entries of the constraint matrix A are zero Rows Columns Nonzeros Density (%) Nonzeros per Name (m) (n) (τ) (100τ/mn) column (τ/n) diet 3 6 18 100.0000 3.00 avgas 10 8 30 37.5000 3.75 aqua 98 84 196 2.3810 2.33 stair 356 467 3856 2.3219 8.26 25fv47 821 1571 10400 0.8063 6.62 Sparsity
LP problems are usually sparse: most of the entries of the constraint matrix A are zero Rows Columns Nonzeros Density (%) Nonzeros per Name (m) (n) (τ) (100τ/mn) column (τ/n) diet 3 6 18 100.0000 3.00 avgas 10 8 30 37.5000 3.75 aqua 98 84 196 2.3810 2.33 stair 356 467 3856 2.3219 8.26 25fv47 821 1571 10400 0.8063 6.62 dcp2 32388 21087 559390 0.0819 26.53 Sparsity
LP problems are usually sparse: most of the entries of the constraint matrix A are zero Rows Columns Nonzeros Density (%) Nonzeros per Name (m) (n) (τ) (100τ/mn) column (τ/n) diet 3 6 18 100.0000 3.00 avgas 10 8 30 37.5000 3.75 aqua 98 84 196 2.3810 2.33 stair 356 467 3856 2.3219 8.26 25fv47 821 1571 10400 0.8063 6.62 dcp2 32388 21087 559390 0.0819 26.53 nw04 36 87482 724148 22.9935 8.28 Sparsity
LP problems are usually sparse: most of the entries of the constraint matrix A are zero Rows Columns Nonzeros Density (%) Nonzeros per Name (m) (n) (τ) (100τ/mn) column (τ/n) diet 3 6 18 100.0000 3.00 avgas 10 8 30 37.5000 3.75 aqua 98 84 196 2.3810 2.33 stair 356 467 3856 2.3219 8.26 25fv47 821 1571 10400 0.8063 6.62 dcp2 32388 21087 559390 0.0819 26.53 nw04 36 87482 724148 22.9935 8.28 sgpf5y6 246077 308634 828070 0.0011 2.68 neos 479120 36786 1084461 0.0062 29.48 stormG2 528186 1259121 4228817 0.0006 3.36 rail4284 4284 1092610 12372358 0.2643 11.32 Sparsity
LP problems are usually sparse: most of the entries of the constraint matrix A are zero Rows Columns Nonzeros Density (%) Nonzeros per Name (m) (n) (τ) (100τ/mn) column (τ/n) diet 3 6 18 100.0000 3.00 avgas 10 8 30 37.5000 3.75 aqua 98 84 196 2.3810 2.33 stair 356 467 3856 2.3219 8.26 25fv47 821 1571 10400 0.8063 6.62 dcp2 32388 21087 559390 0.0819 26.53 nw04 36 87482 724148 22.9935 8.28 sgpf5y6 246077 308634 828070 0.0011 2.68 neos 479120 36786 1084461 0.0062 29.48 stormG2 528186 1259121 4228817 0.0006 3.36 rail4284 4284 1092610 12372358 0.2643 11.32
Structure of A has as much influence on solution time as problem size
The practical revised simplex method 11 Structure (aqua)
98 rows, 84 columns and 196 nonzeros
The practical revised simplex method 12 Structure (stair)
356 rows, 467 columns and 3856 nonzeros
The practical revised simplex method 13 Structure (dcp2)
32388 rows, 21087 columns and 559390 nonzeros
The practical revised simplex method 14 Structure (sgpf5y6)
246077 rows, 308634 columns and 828070 nonzeros
The practical revised simplex method 15 Exploiting sparsity
• Immediate aim ◦ Avoid storing zero values ◦ Avoid multiplications and additions with zero values Exploiting sparsity
• Immediate aim ◦ Avoid storing zero values ◦ Avoid multiplications and additions with zero values • Fill-in ◦ Operating on a sparse matrix may yield more nonzeros than the original matrix. ◦ These additional nonzeros are known as fill-in. ◦ Preserving sparsity by avoiding fill-in is a major challenge in computational linear algebra
The practical revised simplex method 1 Sparse data structures (vectors)
Using a full array to store a vector with relatively few nonzero entries is inefficient Sparse data structures (vectors)
Using a full array to store a vector with relatively few nonzero entries is inefficient
• The standard sparse vector data structure consists of ◦ a list of the values of the nonzeros ◦ a list of the indices of the nonzeros ◦ the number of nonzeros Sparse data structures (vectors)
Using a full array to store a vector with relatively few nonzero entries is inefficient
• The standard sparse vector data structure consists of 2 . 3 ◦ a list of the values of the nonzeros 6 −10.0 7 ← 10 6 . 7 ◦ a list of the indices of the nonzeros 6 . 7 6 7 ◦ the number of nonzeros 6 12.5 7 ← 23 6 7 6 . 7 • For example, the vector x may be stored using the 6 7 6 7 data structure x = 6 30.1 7 ← 186 6 . 7 6 . 7 6 7 Array Data 6 −14.7 7 ← 187 6 . 7 v −10.0 12.5 30.1 −14.7 −50.3 6 . 7 6 7 ix 10 23 186 187 97812 4 −50.3 5 ← 97812 n ix 5 .
The practical revised simplex method 2 Sparse data structures (matrices)
• Storing a matrix A ∈ IRm×n with τ mn nonzeros using a m × n array is very inefficient Sparse data structures (matrices)
• Storing a matrix A ∈ IRm×n with τ mn nonzeros using a m × n array is very inefficient • The standard sparse representation stores it as a set of sparse vectors • For example, the matrix 2 1.2 1.4 3 6 2.1 2.2 7 6 7 A = 6 3.3 7 6 7 4 4.1 4.3 4.4 5 5.2 5.4 may be stored using the data
Array Data a v 2.1 4.1 1.2 2.2 5.2 3.3 4.3 1.4 4.4 5.4 a ix 2 4 1 2 5 3 4 1 4 5 a sa 1 3 6 8 11
The practical revised simplex method 3 Creating fill-in: Example
• When Gaussian elimination is applied to the sparse matrix
2 1 1 1 1 1 3 6 1 2 7 6 7 A = 6 1 4 7 6 7 4 1 8 5 1 16 Creating fill-in: Example
• When Gaussian elimination is applied to the sparse matrix
2 1 1 1 1 1 3 6 1 2 7 6 7 A = 6 1 4 7 , 6 7 4 1 8 5 1 16
• eliminating the entries in the first column yields the matrix
2 1 1 1 1 1 3 6 1 −1 −1 −1 7 6 7 A = 6 −1 3 −1 −1 7 . 6 7 4 −1 −1 7 −1 5 −1 −1 −1 15
• All the zero entries have filled in.
The practical revised simplex method 4 Creating fill-in: Example (cont.)
• When completed, Gaussian elimination yields the decomposition
2 1 3 2 1 1 1 1 1 3 6 1 1 7 6 1 −1 −1 −1 7 6 7 6 7 A = LU = 6 1 −1 1 7 6 2 −2 −2 7 6 7 6 7 4 1 −1 −1 1 5 4 4 −4 5 1 −1 −1 −1 1 8
• Gaussian elimination requires 40 floating-point operations • The LU factors require 25 storage locations
The practical revised simplex method 5 Creating fill-in: Example (cont.)
• The general matrix of this form is
2 1 1 1 ... 1 3 6 1 2 7 6 7 m×m A = 6 1 4 7 ∈ IR with τ = 3m − 2 = O(m). 6 . . 7 4 . . . 5 1 2m−1
• Gaussian elimination requires O(m3) floating-point operations. • The LU factors require O(m2) storage. • Using the LU factors to solve Ax = b requires O(m2) floating-point operations.
The practical revised simplex method 6 Controlling fill-in: Example
Interchange rows and columns 1 and 5 then, after applying Gaussian elimination to 2 16 1 3 2 1 3 2 16 1 3 6 2 1 7 6 1 7 6 2 1 7 6 7 6 7 6 7 A = 6 4 1 7 ,A = LU = 6 1 7 6 4 1 7 6 7 6 7 6 7 4 8 1 5 4 1 5 4 8 1 5 1 1 1 1 1 1 1 1 1 1 16 2 4 8 1 16
• Gaussian elimination requires 8 floating-point operations. • The LU factors require no more storage than A. Controlling fill-in: Example
Interchange rows and columns 1 and 5 then, after applying Gaussian elimination to 2 16 1 3 2 1 3 2 16 1 3 6 2 1 7 6 1 7 6 2 1 7 6 7 6 7 6 7 A = 6 4 1 7 ,A = LU = 6 1 7 6 4 1 7 6 7 6 7 6 7 4 8 1 5 4 1 5 4 8 1 5 1 1 1 1 1 1 1 1 1 1 16 2 4 8 1 16
• Gaussian elimination requires 8 floating-point operations. • The LU factors require no more storage than A.
For the general matrix of this form
• After interchanging rows and columns 1 and m, Gaussian elimination requires O(m) floating- point operations. • The LU factors require O(m) storage. • Using the LU factors to solve Ax = b requires O(m) floating-point operations.
The practical revised simplex method 7 Exploiting sparsity in the standard simplex method
• Initial tableau contains model data so is sparse 1 . . . n 1 . . . m RHS 1 . A I b m
0 cT 0T 0 Exploiting sparsity in the standard simplex method
• Initial tableau contains model data so is sparse 1 . . . n 1 . . . m RHS 1 . A I b m
0 cT 0T 0
• Pivot selection is fixed by simplex method • Tableau usually fills in
The practical revised simplex method 8 Exploiting sparsity in the standard simplex method
• Updating the tableau is the major computational cost of the standard simplex method • To eliminate the entries in the pivotal column q, a multiple of the pivotal row p is added to each of the other rows of the tableau q RHS Pivotal do 20, i = 0, m µ =a ˆiq/aˆpq do 10, j = 0, n p Pivotal row column aˆij :=a ˆij − µ ∗ aˆpj 10 continue 20 continue Cost row Pivot Exploiting sparsity in the standard simplex method
• Updating the tableau is the major computational cost of the standard simplex method • To eliminate the entries in the pivotal column q, a multiple of the pivotal row p is added to each of the other rows of the tableau q RHS Pivotal do 20, i = 0, m µ =a ˆiq/aˆpq do 10, j = 0, n p Pivotal row column aˆij :=a ˆij − µ ∗ aˆpj 10 continue 20 continue Cost row Pivot
• The pivotal column may have a significant number of zero entries. Exploiting sparsity in the standard simplex method
• Updating the tableau is the major computational cost of the standard simplex method • To eliminate the entries in the pivotal column q, a multiple of the pivotal row p is added to each of the other rows of the tableau q RHS Pivotal do 20, i = 0, m µ =a ˆiq/aˆpq do 10, j = 0, n p Pivotal row column aˆij :=a ˆij − µ ∗ aˆpj 10 continue 20 continue Cost row Pivot
• The pivotal column may have a significant number of zero entries.
• If aˆiq = 0 then µ = 0 so loop 10 doesn’t change row i of the tableau. Exploiting sparsity in the standard simplex method
• Updating the tableau is the major computational cost of the standard simplex method • To eliminate the entries in the pivotal column q, a multiple of the pivotal row p is added to each of the other rows of the tableau q RHS Pivotal do 20, i = 0, m µ =a ˆiq/aˆpq do 10, j = 0, n p Pivotal row column aˆij :=a ˆij − µ ∗ aˆpj 10 continue 20 continue Cost row Pivot
• The pivotal column may have a significant number of zero entries.
• If aˆiq = 0 then µ = 0 so loop 10 doesn’t change row i of the tableau. • Skip loop 10 if µ = 0.
The practical revised simplex method 9 Exploiting sparsity in the standard simplex method (cont.)
If the pivotal column has a significant number of zero entries then so should the pivotal row. Exploiting sparsity in the standard simplex method (cont.)
If the pivotal column has a significant number of zero entries then so should the pivotal row.
q RHS Pivotal do 20, i = 0, m µ =a ˆiq/aˆpq if (µ = 0) go to 20 p Pivotal row do 10, j = 0, n column aˆij :=a ˆij − µ ∗ aˆpj 10 continue 20 continue Cost row Pivot
• If aˆpj = 0 then loop 10 doesn’t change column j of the tableau. Exploiting sparsity in the standard simplex method (cont.)
If the pivotal column has a significant number of zero entries then so should the pivotal row.
q RHS Pivotal do 20, i = 0, m µ =a ˆiq/aˆpq if (µ = 0) go to 20 p Pivotal row do 10, j = 0, n column aˆij :=a ˆij − µ ∗ aˆpj 10 continue 20 continue Cost row Pivot
• If aˆpj = 0 then loop 10 doesn’t change column j of the tableau.
• Cost of checking aˆpj = 0 may compare with the floating point operation. Exploiting sparsity in the standard simplex method (cont.)
If the pivotal column has a significant number of zero entries then so should the pivotal row.
q RHS Pivotal do 20, i = 0, m µ =a ˆiq/aˆpq if (µ = 0) go to 20 p Pivotal row do 10, j = 0, n column aˆij :=a ˆij − µ ∗ aˆpj 10 continue 20 continue Cost row Pivot
• If aˆpj = 0 then loop 10 doesn’t change column j of the tableau.
• Cost of checking aˆpj = 0 may compare with the floating point operation. • Pack the pivotal row into a sparse vector.
The practical revised simplex method 10 Exploiting sparsity in the standard simplex method (cont.)
n_ix = 0 do 5, j = 0, n if (aˆpj = 0) go to 5 n_ix = n_ix + 1 v(n_ix) = aˆpj ix(n_ix) = j 5 continue Exploiting sparsity in the standard simplex method (cont.)
n_ix = 0 do 5, j = 0, n if (aˆpj = 0) go to 5 n_ix = n_ix + 1 v(n_ix) = aˆpj ix(n_ix) = j 5 continue do 20, i = 0, m µ =a ˆiq/aˆpq if (µ = 0) go to 20 do 10, ix_n = 1, n_ix j = ix(ix_n) aˆij :=a ˆij − µ∗v(ix_n) 10 continue 20 continue Exploiting sparsity in the standard simplex method (cont.)
n_ix = 0 do 5, j = 0, n if (aˆpj = 0) go to 5 n_ix = n_ix + 1 • Adding these eight lines of code can be very effective v(n_ix) = aˆpj ix(n_ix) = j 5 continue do 20, i = 0, m µ =a ˆiq/aˆpq if (µ = 0) go to 20 do 10, ix_n = 1, n_ix j = ix(ix_n) aˆij :=a ˆij − µ∗v(ix_n) 10 continue 20 continue Exploiting sparsity in the standard simplex method (cont.)
n_ix = 0 do 5, j = 0, n if (aˆpj = 0) go to 5 n_ix = n_ix + 1 • Adding these eight lines of code can be very effective v(n_ix) = aˆpj ix(n_ix) = j • For aqua 5 continue ◦ Introducing test for µ = 0 improved do 20, i = 0, m solution time by a factor of 6.93 µ =a ˆiq/aˆpq if (µ = 0) go to 20 do 10, ix_n = 1, n_ix j = ix(ix_n) aˆij :=a ˆij − µ∗v(ix_n) 10 continue 20 continue Exploiting sparsity in the standard simplex method (cont.)
n_ix = 0 do 5, j = 0, n if (aˆpj = 0) go to 5 n_ix = n_ix + 1 • Adding these eight lines of code can be very effective v(n_ix) = aˆpj ix(n_ix) = j • For aqua 5 continue ◦ Introducing test for µ = 0 improved do 20, i = 0, m solution time by a factor of 6.93 µ =a ˆiq/aˆpq ◦ Packing pivotal row improved solution if (µ = 0) go to 20 time by a further factor of 1.27 do 10, ix_n = 1, n_ix • Overall improvement in performance is a j = ix(ix_n) factor of 8.78 aˆij :=a ˆij − µ∗v(ix_n) 10 continue 20 continue
The practical revised simplex method 11 The revised simplex method
• The revised simplex method is only of value when solving (large) sparse LP problems • The major operations each iteration are as follows
CHUZC: Scan the reduced costs cˆN for a good candidate q to enter the basis. −1 FTRAN: Form the pivotal column aˆq = B aq. CHUZR: Scan the ratios ˆbi/aˆiq for the row p of a good candidate to leave the basis. Let α = ˆbp/aˆpq. Update bˆ := bˆ − αaˆq. T T −1 BTRAN: Form π = ep B . T T PRICE: Form the pivotal row aˆp = π N. T T T Update reduced costs cˆN := cˆN − cˆqaˆp .
The practical revised simplex method 12 Exploiting sparsity in PRICE
The simplest operation in which sparsity can be exploited is PRICE which forms the pivotal row
T T aˆp = π N Exploiting sparsity in PRICE
The simplest operation in which sparsity can be exploited is PRICE which forms the pivotal row
T T aˆp = π N
• This is a matrix-vector product Exploiting sparsity in PRICE
The simplest operation in which sparsity can be exploited is PRICE which forms the pivotal row
T T aˆp = π N
• This is a matrix-vector product
• The entries in aˆp are inner products between ◦ the vector π and ◦ the columns of the constraint matrix corresponding to nonbasic variables. Exploiting sparsity in PRICE
The simplest operation in which sparsity can be exploited is PRICE which forms the pivotal row
T T aˆp = π N
• This is a matrix-vector product
• The entries in aˆp are inner products between ◦ the vector π and ◦ the columns of the constraint matrix corresponding to nonbasic variables. • This requires one operation for each nonzero in N. Exploiting sparsity in PRICE
The simplest operation in which sparsity can be exploited is PRICE which forms the pivotal row
T T aˆp = π N
• This is a matrix-vector product
• The entries in aˆp are inner products between ◦ the vector π and ◦ the columns of the constraint matrix corresponding to nonbasic variables. • This requires one operation for each nonzero in N. • The upper bound on the computational cost is thus O(τ).
The practical revised simplex method 13 Updating B−1
Discussion so far has assumed B−1 is available
• Forming B−1 explicitly requires Gaussian elimination at a computational cost of O(m3) Updating B−1
Discussion so far has assumed B−1 is available
• Forming B−1 explicitly requires Gaussian elimination at a computational cost of O(m3) • For most LP problems, m3 mn so forming B−1 each iteration makes the revised simplex method vastly more costly than the standard simplex method. Updating B−1
Discussion so far has assumed B−1 is available
• Forming B−1 explicitly requires Gaussian elimination at a computational cost of O(m3) • For most LP problems, m3 mn so forming B−1 each iteration makes the revised simplex method vastly more costly than the standard simplex method.
• However, each iteration, the change in B is the replacement of column p by aq. Updating B−1
Discussion so far has assumed B−1 is available
• Forming B−1 explicitly requires Gaussian elimination at a computational cost of O(m3) • For most LP problems, m3 mn so forming B−1 each iteration makes the revised simplex method vastly more costly than the standard simplex method.
• However, each iteration, the change in B is the replacement of column p by aq. • Algebraically this can be expressed as the rank-1 update
T B := B + (aq − ap)ep Updating B−1
Discussion so far has assumed B−1 is available
• Forming B−1 explicitly requires Gaussian elimination at a computational cost of O(m3) • For most LP problems, m3 mn so forming B−1 each iteration makes the revised simplex method vastly more costly than the standard simplex method.
• However, each iteration, the change in B is the replacement of column p by aq. • Algebraically this can be expressed as the rank-1 update
T B := B + (aq − ap)ep
• Using the Sherman-Morrison formula it can be shown that
T ! (aˆq − ep)e B−1 := I − p B−1 aˆpq Updating B−1
Discussion so far has assumed B−1 is available
• Forming B−1 explicitly requires Gaussian elimination at a computational cost of O(m3) • For most LP problems, m3 mn so forming B−1 each iteration makes the revised simplex method vastly more costly than the standard simplex method.
• However, each iteration, the change in B is the replacement of column p by aq. • Algebraically this can be expressed as the rank-1 update
T B := B + (aq − ap)ep
• Using the Sherman-Morrison formula it can be shown that
T ! (aˆq − ep)e B−1 := I − p B−1 aˆpq
• Although this is a matrix-matrix product, by exploiting its structure, its cost is O(m2).
The practical revised simplex method 14 The story so far...
Variant Cost per iteration Standard simplex method O(mn) The story so far...
Variant Cost per iteration Standard simplex method O(mn) Revised simplex method O(m2 + τ) The story so far...
Variant Cost per iteration Standard simplex method O(mn) Revised simplex method O(m2 + τ)
• Iteration costs proportional to the product of problem dimensions are prohibitively expensive. The story so far...
Variant Cost per iteration Standard simplex method O(mn) Revised simplex method O(m2 + τ)
• Iteration costs proportional to the product of problem dimensions are prohibitively expensive. • What (more) can be done to exploit sparsity? The story so far...
Variant Cost per iteration Standard simplex method O(mn) Revised simplex method O(m2 + τ)
• Iteration costs proportional to the product of problem dimensions are prohibitively expensive. • What (more) can be done to exploit sparsity? Standard simplex method: ◦ Storage costs may be reduced by representing the tableau using sparse data structures ◦ Computation costs cannot be reduced further than was achieved with the techniques above. The story so far...
Variant Cost per iteration Standard simplex method O(mn) Revised simplex method O(m2 + τ)
• Iteration costs proportional to the product of problem dimensions are prohibitively expensive. • What (more) can be done to exploit sparsity? Standard simplex method: ◦ Storage costs may be reduced by representing the tableau using sparse data structures ◦ Computation costs cannot be reduced further than was achieved with the techniques above. Revised simplex method: ◦ O(m2) storage and computation costs can be reduced significantly by exploiting sparsity when factorising B−1. The story so far...
Variant Cost per iteration Standard simplex method O(mn) Revised simplex method O(m2 + τ)
• Iteration costs proportional to the product of problem dimensions are prohibitively expensive. • What (more) can be done to exploit sparsity? Standard simplex method: ◦ Storage costs may be reduced by representing the tableau using sparse data structures ◦ Computation costs cannot be reduced further than was achieved with the techniques above. Revised simplex method: ◦ O(m2) storage and computation costs can be reduced significantly by exploiting sparsity when factorising B−1.
End of Part 1
The practical revised simplex method 15