Affine-Scaling Method, 202–204, 206, 210 Poor Performance Of

Index

affine-scaling method, 202–204, 206, 210 blocking variable, 48 poor performance of, 203 breakpoint, 163 steplength choice, 203 approximation problems, 218–227 canonical form, 8, 16, 45, 117, 120, 123, 1-norm, 221–224 129, 130, 134, 138, 139, 150, 2-norm. See least-squares problems 151, 156, 195, 197 ∞-norm. See Chebyshev approximation dual of, 109 for linear equalities, 218–223 for problems with bound constraints, with hard constraints, 223–224, 227 130 arc, 12–14, 144 centering parameter, 206, 214 central path, 205 back substitution. See triangular substitution Chebyshev approximation, 219–221, 223 basic feasible solution, 118 Cholesky factorization, 209, 212 as iterates of revised simplex method, classification, 11–12, 230–235 123 labeled points, 230 association with basis matrix, 120–121 testing set, 235 association with vertex, 121–123 training set, 235 degenerate, 120 tuning set, 235 existence, 119–120 classifier, 230 initial, 129, 134–139 linear, 230 basic solution, 118 nonlinear, 233, 234 basis, 117, 126, 152 complementarity, 101–102, 178, 237 for problems with bound constraints, almost-, 178, 179 130 existence of strictly complementary initial, 125, 134–137 primal-dual solution, 114–115 optimal, 129, 152–156 strict, 101 basis matrix, 52, 118, 120, 123, 139, 140, complementary slackness. 154 See complementarity for network linear program, 149 concave function, 161, 244 LU factorization of, 139–142 constraints, 1 big M method, 110–112 active, 14 behavior on infeasible problems, 112 bounds, 74–75 motivation, 110–111 equality, 72–87 proof of effectiveness, 111 inactive, 14 specification, 111–112 inequality, 72–87 bimatrix game, 185 convex combination, 239, 241 Bland’s rule. See smallest-subscript rule convex function, 170, 242–244, 248

261

linear underestimation, 170, 250 feasible region, 3, 46, 51, 53, 169, 171, strict, 173 182 convex hull, 56, 113, 241 feasible set. See feasible region convex set, 170, 173, 239–241 feature space, 233 cost vector, 7, 151, 154 fickle variable, 69 parametrized, 159 finite termination of simplex method, cross-validation, 235 65–72 cycling, 66–72, 104 floating-point number, 127 floating-point operations, 39 Dantzig, George B., 7, 16 forward substitution. See triangular degenerate linear program, 66, 67 substitution diet problem, 8, 96 Frank–Wolfe theorem, 172–173 divergence, 144 fundamental theorem of algebra, 113, 225 domain, 242 dual linear program Gaussian elimination, 39 of canonical form, 196 general form of linear program, 75 of standard form, 94, 102 dual of, 107–109 tableau representation, 94 global solution, 170, 171, 173, 182 dual linear system, 89 Gordan theorem, 113 dual pair of linear programs. See primal- gradient, 172, 242, 244, 247 dual pair of linear programs graphical solution of linear programs, 2–6 dual simplex method, 89, 102–106, 164, 220–223 Hessian, 171, 172, 214, 224, 242, 244, motivation for, 102 245 relationship to simplex method applied Huber estimation to the dual, 105–106 formulation as quadratic program, specification of, 103–104 229–230 duality, 89–115 motivation and definition, 227–228 for quadratic programs, 176–177 optimality conditions, 228–229 practical example, 96–98 strong, 98–99 infeasible linear program, 14, 62, 98, 225 applications of, 112–115, 167 infeasible point, 3 for quadratic programming, 176–177 integer programming, 7, 16 weak, 97–98, 101, 111, 176 interior-point methods duality gap, 197 basic properties of primal-dual duality measure, 197, 205, 213 methods, 196–197 comparison with simplex method, 212 ellipsoid method, 15 introduction, 15, 195–197 entering variable, 47 path-following. See path-following epigraph, 217, 242–243 methods equilibrium pair. See Nash equilibrium relationship to Newton’s method, 202 expected loss, 185 Jacobian, 199, 202, 249 Farkas lemma, 112–113 Jordan exchange, 17–23, 26, 27, 46–48, feasible point, 3 53, 83, 179, 221. See also pivot dual, 98, 103 block, 31

blocking of, 25, 33, 36, 38 relationship to quadratic programming, interpretation on dual tableaus, 89–91 174, 177, 180, 182 linear dependence, 23, 24, 138, 221 Karmarkar, Narendra, 15, 214 of functions, 24 Karush–Kuhn–Tucker (KKT) conditions, linear equations, 32–39 100–101, 129, 192, 195 inconsistent, 218, 227 as constrained nonlinear equations, overdetermined, 220, 221, 223, 224, 201–202 226, 227, 230 for canonical form, 195, 196 solution using Gaussian elimination, for linear programs in general form, 39–41 108 solution using Jordan exchanges, for quadratic programs, 173–175, 181, 32–39 184–185, 212, 229, 233 flop count, 39, 44 statement of, 100–101 with infinitely many solutions, 37 kernel, 234 linear independence, 17, 23–27, 51, 52, Gaussian, 234–235 118, 119 linear, 234 of functions, 24 polynomial, 234 of rows/columns in a matrix, 92 KKT (Karush–Kuhn–Tucker) conditions. local solution, 169, 170, 173, 184 See Karush–Kuhn–Tucker LU decomposition. See LU factorization conditions LU factorization, 17, 39–44, 127, 138, 197 knapsack problem, 130 complete pivoting, 43 flop count, 44 partial pivoting, 42–44 Lagrange multipliers, 173, 176, 195, 229, updating, 139–142 232 Lagrangian, 176 matching pennies game, 188–191, 193 LCP. See linear complementarity matrix problem (LCP) addition, 239 least-squares problems, 211, 224–229 basis. See basis matrix normal equations, 225 diagonal, 196, 208, 238 leaving variable, 48, 156 eigenvalues, 246 Lemke’s method, 178–185 identity, 238 application to quadratic programs, indefinite, 208, 214, 245 182–185 inverse, 27–31, 238 outline of, 178 invertible, 51, 152, 225 Phase I, 178–179, 188 loss, 185–188, 191 Phase II, 179, 188 lower triangular, 39, 209, 239 termination of, 182 unit, 40 Lemke–Howson method, 187 multiplication, 239 level set, 243 nonsingular, 27 linear combination, 240 permutation, 39–41, 139, 209, 238 linear complementarity problem (LCP), poorly conditioned, 226 169, 177–178 positive definite, 245, 246 definition, 177 positive semidefinite, 172, 174, 176, infeasible, 182 224, 245, 246

representation in MATLAB,251–252 p, 219, 247 singular, 27, 31 ∞, 219, 247 skew-symmetric, 178 dual, 231, 248 sparse, 208, 209, 238 equivalence of, 248 symmetric, 172, 244 totally unimodular, 150 objective function, 1, 154, 241 transpose, 238 contours, 4, 46, 159 upper triangular, 39 optimal face, 58. See also solution set Mehrotra predictor-corrector method. order notation See path-following methods O(·), 248 minimax problems, 217–218 o(·), 248–250 discrete, 218 minimum principle, 170, 173, 184 parametric programming, 151, 158–168 mixed strategy, 185 path-following methods mixed-integer linear programming, 150 MATLAB implementation of, 209 monotone, 247 choice of centering parameter, 207, Nash equilibrium, 186, 192 210 computation of, 186–187 choice of starting point, 211 Nash, John, 169, 186 choice of steplength, 207, 211 network linear program, 143–150 corrector direction, 210, 211 assignment problem, 149 for quadratic programming, 212–214 circulation problem, 147 linear algebra issues, 208–209, 211, general properties, 143–144 214 max-ﬂow, 146–147 normal-equations form, 208 minimum-cost, 12–14, 144–146, 149 long-step, 206–207 node-arc incidence matrix, 145 speciﬁcation of algorithm, 206–207 shortest-path, 145–146 motivation, 204–205 transportation problem, 147–149 practical enhancements, 209–211 network simplex method, 149–150 relationship to Newton’s method, Newton’s method, 197–201 205–206 for scalar nonlinear equations, 197–198 permutation, 29 for systems of nonlinear equations, Phase I of simplex method, 60–65, 179, 197–201 223 quadratic convergence of, 199 after addition of constraints, 157 with line search, 201 description of, 60–65 node, 12, 144 dual simplex as possible alternative, demand, 13, 145 102, 104 sink, 145 for canonical form, 134–138 supply, 13 for problems with parametrized con- nondegenerate linear program, 65, 66 straints, 167 nonlinear programming, 169–172 for solution of linear inequalities, 64 nonnegative orthant, 171, 251 purpose of, 53 norm, 218, 247–248 Phase II of simplex method, 53–60, 62, 1, 221, 247 71, 103, 135–137 2, 231, 247 piecewise-linear function, 161, 163, 217

Copyright ©2007 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed. Index 265 pivot, 18, 20, 46, 47. See also Jordan exchange definition, 49 block, 31 for dual simplex method, 103, 104, degenerate, 62, 64, 69 157, 165 nondegenerate, 67 in Lemke’s method, 179, 180, 188 submatrix, 31 robust implementation of, 143 pivot selection, 15. See also pricing ties in, 66, 68 Devex, 142, 143 reduced costs, 49, 68, 124, 132, 142, 152, for dual simplex, 103 153, 160 for Lemke’s algorithm, 179 regression problems. See approximation steepest-edge, 142 problems polynomial complexity, 207 residual vector, 219, 221, 227 predictor-corrector method. outliers, 227 See path-following methods resource allocation, 10–11 preprocessing, 212 revised simplex method, 52, 117, 123–143, pricing, 47, 49, 53, 68, 117 156 partial, 142 choice of initial basis, 125 primal-dual methods. See interior-point for problems with bound constraints, methods 129, 134 primal-dual pair of linear programs, 89 fundamental idea, 123–124 definition, 94 initial basic feasible solution, 134–139 for general linear programs, 108 linear algebra in, 124–126 interior-point methods and, 195 pivot selection, 142–143 optimality conditions for, 100 roundoff error issues, 127–129 weak duality for, 97 specification, 124 zero-sum games and, 192 roundoff error, 28, 127, 143 prisoners’ dilemma, 186, 191 projection, 173, 205, 244 saddle point, 184 pure strategies, 185 scalar product, 252 Scheme II for linear programs in nonstan- QR factorization, 226 dard form, 72, 80–86, 157, 220, quadratic function, 244–247 221, 223 convex, 244–247 Schur complement, 31 quadratic programming, 7, 172–177, sensitivity analysis, 4, 151 212–215, 227, 231 separating hyperplane, 11, 230, 231, 233 convex, 172, 173, 175, 212 separation lemma, 113 general form, 175 shadow prices, 97, 154, 155. See also infeasible, 182, 183 Lagrange multipliers nonconvex, 182, 184 Sherman–Morrison–Woodbury formula, unbounded, 182, 183 140 shifting, 74 rank, 91, 225 simplex method, 6, 7, 45–87 determination via Jordan exchanges, introduction, 14–15 91–92 network, 149–150 full, 225 slack variables, 8, 45, 117, 152, 196 ratio test, 48, 126, 203 definition, 45 after addition of constraints, 157 dual, 101, 103

Copyright ©2007 by the Society for Industrial and Applied Mathematics This electronic version is for personal use and may not be duplicated or distributed. 266 Index smallest-subscript rule, 67–72, 78, 104 theorem of the alternative, 113 solution set, 58–60 tolerance source, 145 pivot, 127 standard form, 7, 16, 45, 117, 151 slack, 127 standard form, transformation to, 72–76 triangular substitution, 40, 139, 141, 209 free variable method, 75 Scheme I, 72, 76–80 unbounded linear program, 14, 46, 54–56, Scheme II, 80–86 66, 98 substitution method, 74 unbounded ray, 54, 182, 183 Steinitz theorem, 26–27, 38, 92 uncertainty in formulation, 151 support vector machines linear, 230–233 variables, 2 linear programming (1) formulation, artiﬁcial, 61, 62, 135, 167, 179, 218 233 for equality constraints, 80–82, 193 nonlinear, 233–234 basic, 45, 152 nonseparable, 231–233 blocking, 133 separable, 230–231 dependent, 18, 21, 23 support vectors, 231, 232 dual, 89. See also Lagrange multipliers tableau, 18–20 free, 75, 80, 107, 134, 136–139, 167, augmented, 62 184, 222 condensed, 45, 47, 123 independent, 18, 21, 23, 26 degenerate, 65, 66, 120 nonbasic, 45, 51, 130 dual, 89 nonnegative, 107 feasible, 46, 187 slack. See slack variables initial, 47, 119 vector labels, 21 norm. See norm MATLAB representation, 21–22 notation, 237 nondegenerate, 65 representation in MATLAB, 251 optimal, 49, 53, 58, 71, 152–154, vertex, 14–15, 46, 51–53, 120, 163 156 adjacent, 15 primal, 89 degenerate, 52 unbounded, 54 Taylor’s theorem, 170, 198, 199, 249 zero-sum game, 185, 192–193